Spoiler
Show
1. The Stormcloaks' goals and mentality are inherently selfish.
It's been brought up that the Stormcloaks are racist, but I think it's not that simple. They're not racist in the same way the Thalmor are, treating all other races with disdain at best or disgust at worst. The primary reasoning behind the Stormcloaks' dislike of non-Nords is more along the lines of "You didn't help us, so we're not going to help you." Their platform is "Skyrim first," and since people like the Dunmer and the Argonians, don't want to help Skyrim achieve independence, they can go hang. It's also the primary reason for why they want to secede from the Empire. The Talos issue is part of it, yes, but more importantly the Stormcloaks feel that the Empire hasn't given anything to Skyrim in exchange for everything they've taken. Because I feel the Empire has to stand together and can't afford to give one of the few provinces it has left preferential treatment over another, I can't get behind this kind of thinking.
2. The Stormcloaks are primarily a cult of personality centered around Ulfric.
While its true that the Stormcloaks talk a great game, the fact remains that they wouldn't exist as an organization without Ulfric's leadership. Had he been executed at Helgen, the war would have ended there. Most of the Stormcloaks don't seem to be a trained and disciplined army, but rather a ragtag band of peasants and minor nobles with an axe to grind both figuratively and literally. Ulfric is what holds them all together, and the way they talk about the man, you'd think he was the second coming of Talos or something. But Ulfric is just a man, and if you defeat him in the Civil War, the rebellion crumbles. This makes me question their efficacy if they DO win. Ulfric knows what the next step is, preparing for the Thalmor's guaranteed retaliation, but they've just lost the support of the Empire: the infrastructure and trade that it provided. Skyrim can definitely take care of itself militarily, but can it stand on its own in terms of logistics? While the Empire might not be perfect, they're a lot more stable, and that can only be good for Skyrim in the long run.
3. Ulfric has a near-obsession with theatricality and symbolism that makes him hard to trust.
If you look closely at Ulfric and his rival Tullius, you can see that they're near polar opposites: Tullius is blunt and his ignorance of the Nord people makes him unpopular, while Ulfric's adept use of them wins the support of his allies and fear of his enemies. Through speaking to Ulfric and reading about him, it seems that a lot of his actions are carefully planned out like a performance. For instance, when he killed Torygg, he first challenged him, to give legitimacy to his claim and so it didn't seem like random murder. Then he used the Voice to show that he had ambition and will, something he believed Torygg lacked. Then, if you believe him, he killed Torygg with a sword, to mitigate the fact that using the Voice could be considered cheating and that using it in this instance was a violation of the Way he'd learned from the Greybeards. I'd argue that the Markarth Incident was staged similarly. Reading "The Bear of Markarth" indicates that Ulfric jumped the gun and attacked before anyone had a chance to speak to the Forsworn and I think he knew they'd go back on their promise to allow Talos worship to avoid war with the Thalmor. In fact, I think he wanted it. By locking him up after he "heroically" liberated Markarth, Ulfric was able to cast himself as a victim rather than an instigator, and it earned him a lot of sympathy (especially since they let him become Jarl of Windhelm after he got out of jail). Likewise, the Battle for Whiterun is another cleverly staged message. Irileth notes if you're playing for the Imperial side that Ulfric has already proven his own personal strength with the death of Torygg, and now he needs to prove the strength of his army by capturing a city, and proving they're not loudmouthed cowards hiding in areas where they have a Jarl's support. Ulfric uses Nord traditions and symbols to give his words bite. I'd argue that he needs this theatricality to maintain his hold on power. He treads a very fine line between the love and hatred of Skyrim's people, and he has to do his best to make sure that he doesn't fall. Even when he's about to die he cares more about looking good than his own life, asking that you be the one to end his life so it'll sound more badass when the bards talk about it. This need for chicanery and using people as symbols is what really makes me suspicious of Ulfric. I'm not saying he's evil, Bethesda did a good job of making the Civil War questline morally gray. But I feel like he'd view me more as a symbol than a friend ("Look everyone! I've got the Dragonborn on my side! Nothing can stop us now!"), that he cares less for his soldiers as people than as martyrs, and that he potentially can shoot himself in the knee with the very symbols he wields so adeptly, as Elisif notes that if Ulfric had just spoken out against the Empire directly in a public forum, Torygg would have supported him, but instead Ulfric danced around the issue with vague talk that just skirted the edge of treason.
While I have no doubt that a lot of the Stormcloaks are good people who just want to help their homeland, these three problems that their movement as a whole has make me leery of them, and they're why I support the Empire. I've been loyal to them for two Elder Scrolls games, they have the needs of the many at heart, they have the infrastructure (messy as it is) to hold fast against the Thalmor, and I feel that General Tullius is more willing to be straight with me about things.