Have you ever seen Material Spell Components enforced?

[MrUberGr]

IMO your argument is void, since something that has no cost goes by unmentioned. Just like you don't say "I go poop, shave and take a long shower" once you reach an inn after traveling, it's the same thing. Also, everyone is talking about materials and component pouches. No one seems to remember that casters can use foci. For example, a wizard can forget about his component pouch and just rely on his staff since,

A character can use a component pouch or a spellcasting focus (found in chapter 5) in place of the components specified for a spell.

If he decides to use a component pouch, it is more realistic that a wizard's component pouch may run out at some time, and what you could do is every X sessions he's gonna have to spend 10gp to buy a new one. That's what I used to rule as DM both for this as well as for quivers. On the other hand, Thieves Tools has a set amount of uses, and whenever a pick lock attempt failed by 10 the pick would break.
Anyway, do you think that micromanaging resources is what makes DnD fun? Using a much broader measure would be better. "You've been in the wilderness searching for the goblin HQ about a month now. You're getting low on resources and are not sure how many more of those pesky ambushes you can fall in before having to return to the town." , would be more than enough in order to spur your players and give them a sense of urgency. "10 arrows left in your quiver" would probably lead to some annoying conversations about whether or not the archer can reclaim used arrows, or try to create new arrows, right there in the middle of a forest, because, wood is all around us dammit.

As for digging through loose material, do you really think someone who dedicated the better part of his life in studying a subject would be a slob when it came to it? As Whyrocknodie said he'd probably prepare every component bundle during the daily preparation.

However, of course, if for some reason the component pouch/arcane focus is lost/taken, then the caster might have to get to work. If he's in a prison for example, he might want to actually collect guano from the pesky bats at the corner of his cell, and some sulfur from a torch or from the courtyard where it is used as pest repellent. Then he can blast that unsuspecting guard away. Using the concern of materials for a session or two would probably be very fun, but using it as a daily concern to "balance" casters, would most definitely either lead to forgetting about the rule, or some serious arguments for nothing.

"Fortunately" for me, my left leg was badly scorched by a dragon, leaving me with the excuse that my staff is my walking stick!