This would be the thought that sends him to misery. He can not cope with the fact that Bard's are not ruleing the world since they have all the skills and chrasima.
Minor spoiler alert below:
I hope he and the big X meet and the big X sits on the thorne of his "just and noble" kingdom after bringing him down.
That is just the way the Big X role plays.