Our elves are primarily scavengers. Logically, they need pretty hefty immune system to stomach what they need to survive. Lets amp that up a bit and say they can eat fruits and berries that are lethally poisonous to others. Like certain species of insects, their bodies store the toxins.

End result? Predators let elves be because eating them kills you.

We can even extend this to partly explain why they havent been conquered. Firstly, any place with elves still has most of its wildlife intact, including beasts and monsters that would gladly eat anyone else. Secondly, elves themselves could be immune to most diseases, but that doesn't mean they can't serve as carriers for them! Just being in the presence of elves could lead to catching a deadly plague. Remember how diseases carried by european invaders wreaked havoc among native americans? It's that in reverse. The would-be conquerors kill the first bunch of elves, laughing at these naive beings. They return home with their spoils of war - and next thing you know, it is THEIR civilization that ends up decimated. Cue legends of a blood curse befalling anyone who does violence on the fair folk, and anyone who could enslave the elves is too frightened to try.

As to why these elves exists? That s easy! Even a culture of humans could survive within the guidelimes proposed. So, once upon a time there was a small group of sapients who decided they'd do harm to no creature. Their lives became harsher than anyones', but they survived. In time, they mastered their new lifestyle and slowly their numbers grew.

Add enough time and the right environment, and these sapients will evolve into our elves. There.

If you want to add magic to the mix, you could say the elves can speak with plants and animals - a mystical trait they have either learned or earned as a result of their lifestyle. This would further allow them to cooperate with animals and forge symbiotic relations with them, where elves help catle to give birth in exchange of milk, for example.