I think the racism aspect more comes from the common, though increasingly subverted, portrayal of Orcs, Goblins, Trolls and the like, as a sapient species that is inherently stupid, brutish, and evil. I agree that basic differences between races or species are not racist, and that most examples in fiction are not meant allegorically- the complaint, unless I'm grossly misunderstanding the discussion, arises more from the worry that these portrayals show an attitude that is very problematic if it's kept in the real world. How strongly it translates out of fiction is the most likely point of debate, then- assuming a goblin your character runs into is an enemy is probably a very far cry to making similarly disparaging assumptions about real-life minorities, even if it does stem from the same basic mental process.