Advantage/disadvantage on ability checks give a +/-5 modifier on passive ability checks. (PHB chapter 7 - passive checks) "If the character has advantage on the check, add 5. For disadvantage, subtract 5. The game refers to a passive check total as a score."
Why does it need to compel the enemy? They make choices and draw conclusions based on information that they have. If a player gives their enemy misinformation the enemy will take actions based on that misinformation and make mistakes.
The simplest example of this I can think of his tricking a dog you've thrown a ball. If you know what a dog does when you throw a ball you can get it to do that without even having a ball.
If you know that hobgoblins almost always go to investigate noises, you can make them go investigate an empty bush by throwing a rock into the bush.
If you know that during a fight, a hobgoblin goes to uncover a hidden rogue from their last known location, you can trick it into going into a trap or hazard or disadvantageous position by tricking it you are somewhere else. There's no compulsion needed, and no guarantees it will work.
This should work for both rational enemies as well as emotional enemies. If you can make a rational enemy make mistakes by manipulating its knowledge you should be able to an emotional enemy make mistakes by manipulating its emotions (through the charisma skills in that case).