I would also not be surprised if Skyrim is not a prestigious posting by Thalmor standards, so what we're seeing is probably a mix of people who have to be there for their long term goals (like Elenwen) and people who are too low-ranking or otherwise in disgrace to get out of going (like the average foot-soldier).