What drives a poor reputation for the Rogue class?

[LudicSavant]

This definition is certainly not an exclusive definition for what "passive" means, and it definitely can include meaning the character is "being casual or passive." The only "hard rule" here is that Passive checks are checks done without a dice roll.
Going with the understanding that a Passive check would trigger when you may notice something you aren't necessarily looking for, as compared to an Active check in which you are intentionally searching/investigating/etc.

Per the definition of a passive check: 'A passive check is a special kind of ability check that doesn't involve any die rolls. Such a check can represent the average result for a task done repeatedly, such as searching for secret doors over and over again, or can be used when the DM wants to secretly determine whether the characters succeed at something without rolling dice, such as noticing a hidden monster.'

There is nothing whatsoever in the books about being casual or 'not really looking.' Indeed, the very first example of what a passive check is is searching for secret doors over and over again.

This is, incidentally, precisely why passive investigation is not an oxymoron. Per the PHB, passive checks represent the average result of a task done repeatedly, or if the DM wants to secretly determine the outcome. The player is being passive (by not rolling dice), not the character.