Torygg had a voice. He could have trained it to do what Ulfric could do, but he didn't. That's his lookout.
Storms and warriors - and fire, and etherealness, and domination, and trickery, and clear skies, and moving really really fast over short distances. But not anything that's not geared towards adventurers. It's an eccentric domain, that one.
Edit:
Yep, exactly. We have to bear in mind the politics going on here. Nobody's opinion or account should be taken entirely at face value: every single character in the game has their own agenda and is, whether they even realise it or not, spinning like fury to present the most favourable case they can for their own team.
It seems to me that a lot of people here apply that sort of scepticism only to one side, they take Team Torygg's view as unvarnished truth.
Further edit:
Weell... some elderly dude in a legion officer's uniform turns up for a brief cameo. Maybe he's Talos, maybe he isn't - there's no definitive statement.