A 352:

Creatures who fail their perception check against B do not perceive B, and therefore do not know he's there. There are very few situations where such a creature would even attempt to attack or otherwise affect B, so the question of disadvantage on those attacks is moot. In the rare case that they do have reason to suspect B's presence without perceiving it (for instance, they have met the party before, and knows that the people they do perceive are usually in the company of a stealthy rogue), they might attempt to guess various likely hiding places and attack them, but those are pure guesses, and would still involve disadvantage even if they did guess correctly.

The way, mechanically, for creatures to detect B is to roll Perception checks, and that's already failed. They might have more chances to do so, depending on subsequent actions and events.