I am familiar with the story of Hermaphroditus and Salmalcis, and I agree that in the strictest sense it refers to an organism that is both male and female in terms of sex - but this is not really how it's used when you're talking about people in common usage. There absolutely are cases of, for example, true hermaphroditism in humans. However, the modern medical term is actually "ovotesticular disorder of sex development" (wow, that's a mouthful), with terms like "hermaphrodite" being phased out when referring to people.

Like many other terms that came to be used as intentional insults it began as a simple descriptive term. It did however come to be used as an insult for intersex people as well as a wide array of gender and sexual minorities. These instances were often irrational, but words mean whatever people use them to mean and the term has come to have a definite negative connotation when applied to people - outside of a historical context. So it has come to be considered an impolite or rude term. Will this persist? No idea. The word "queer" when describing a person was most definitely used as a slur and to some people it still is, but usage changed and after being reclaimed it is now proudly used as a label people identify with openly. I have no idea if something similar will happen with hermaphrodite, but for the time being the consensus seems to be that it's a rude word. I'm not telling you what words to use or not use, I just mean to point out that using this word will carry this negative connotation to a lot of people - at least when used to refer to other people.