Another consideration:

To know a spell is being cast, you must see the verbal, somatic, or material components of the spell.

Note that a focus is a substitute. It's not the material component. It performs the same role but has its own rules, such as requiring proficiency (which material components do not) and being able to paint it on your shield if you're a cleric.

Casting a Fireball requires bat guano or a bit of sulphur, which the spell consumes. If you Subtle Spell a Fireball, and are not using a focus, you need sulphur or bat poo. That's for all Fireball spells across all classes; so perhaps the M part of seeing a spell being cast is simply the fact that someone holding bats+^t in their hand while focusing intently is enough for people to guess what's going on if they're at all clued in.

But holding a staff? Anyone can do that. There's nothing at all unusual. A cleric needs merely have their holy symbol on their shield. A bard can be idly fingering their instrument, surely something they do all the time.

TL:DR; a focus is not a material component.