I'd say it isn't wrong (just creepy) if the nerd who does it simply uses his charm person (something with no side effet) merely to hang around people who, otherwise, wouldn't like him.

If he uses it to gain certains.. advantages, then I'd say it's wrong.

(I had a character, Solka Truesilver, who used Charm Person like water. But while he charmed peoples and teammates, he never abused of their magically-inspired trust. Once, the DM had given me a golden opportunity to.. ahem.. "do" a princess I had charmed, but I find sex-in-RPGs so creepy that I almost ruined the spell to avoid the whole thing.)

So, I'd say if Solka used his power to put people into danger. To make them do things they'd regret later. To abuse them, it would be wrong. If it causes no harm to other around him, that's not immoral.

(I remember, we once took 2 prisonners from an ennemy patrol. I managed to charm one of them, while the other one resisted. We ended up executing the one who resisted, while I felt responsible for the well-being of the one under my charm)