I personally suggest keeping lower-tier units of Ashigaru to defend your territories. They'll cost less in upkeep, and unless the enemy army is full of elite samurai, they can handle most kind of enemies you'll face. Particularly in castles, where their low morale will not be an issue.