If you're latching onto a fandom, you're doing it wrong. If you like this particular TV show about technicolor ponies, then don't not talk people about it because you're afraid they'll think of you as 'one of those fans.' If you aren't comfortable talking to all three at once pick one or two of them and approach them separately.
You can like ponies and not be a brony. Don't ever let fear of being associated with some of the more... exuberant... elements of the fandom stop you from saying that you like a thing.
Don't know whether it's fear or whatever. I've just never been able to form close associations with much of anything. Tried but maybe I'm going about it in the wrong way. Though I don't think I should be posting this here as it is not on topic and therefore probably wasting forum space.