There is nothing antagonistic about not being interested in playing certain types of games.
I am not my players' slave. Nor are they mine.
Why am I required to participate in a game that I don't enjoy?
This seems to be the most aggressive assertion of "the geek social fallacies" I have ever heard!
This is not about being charismatic, it is about refusing to participate in social scenes at all.
The fact that you feel the need to frame it as such shows to me that you either don't understand what I am saying, or you realize how ridiculous your position would sound if you came right out and said it.