I think part of the issue (as far as I understand it) is that D&D grew out of the house-rules of a bunch of gamers whose original campaign characters (Bigby, Mordankainen et al) were magic-users, and a lot of the mechanics kind of assume that it will be the wizards who will be the world-shakers and shapers. That doesn't fit with much of other fantasy literature sources, where martial and roguish types have more ability to do the heroic deeds that change the world (often by killing the wizard overlord). If you look at core things that define D&D, I would say that amongst them are the D20 die roll vs AC for the to-hit mechanic, Fighters getting +1 to hit per level, and casters getting a new spell level every odd level, with fireball and lightning bolt at level 3 and teleport and cloudkill at 5. Even with the differential XP requirements in 2nd Ed, linear martials and exponential casters is sufficiently hard-baked into D&D that if one were to try, say, giving full casters bardic spell progression, the game would be unrecognisable as D&D.
Another issue in my mind is the advent of cRPGs and the internet. I played tabletop D&D for much of the early part of my teens, and getting a character above 5th level was a pretty rare occurrence, either because the campaign was too deadly or the DM had a better idea and wanted to start again. Between reloading after character deaths and replaying after a TPK, there's an assumption that the player will eventually complete and win a cRPG, and generally that means advancing characters to the limits of the (limited, compared to ttRPG) advancement and power available. This perhaps creates an expectation that if you create a wizard, he will be able, at some point, to cast, quickened stinking clouds and contingencies and the rest. Even early level weaknesses are avoided by the likes of abrupt jaunt. Any difficulties in squeezing maximal power out of a build can be overcome by a quick trip to the playground for advice from experts in optimisation.