There is definitely a curve between T1-T2 full casters and, well, pretty much everyone else. As you point out, though, it isn't that the casters are more powerful at all times than the martials. The difference is often, and aptly, summarized as "linear vs. quadratic advancement". In other words, the linear advancement of the fighter-types starts out at a higher point than the caster's power, but it grows much less slowly. Still, at low levels, it can be very true that fighters are the majority of the party's power. Things start to change around levels 5-6, when third level spells enter the picture, and the casters will have most definitely superseded the martials by around level 11.

At high levels, though, if you want an optimized caster you basically need to have 9th-level spells. Not just for raw power, but also for things like save DCs. Even if you like a lower level spell and don't think you'd ever cast anything higher, you still can metamagic it. Even the best martials have trouble competing next to spells beyond level ~5 and are out of their league around spell level ~7.

So, for a fully optimized level 20 character, there's essentially no reason to take martial levels, even for survivability at low levels. Even a straight wizard 20 with single-class optimization (far from the best combo around) would lose 6 high-level spell slots and 15% of successes on CL checks from taking 3 mundane levels earlier in their career, in exchange for...net +1 BAB, unusable armor proficiencies, slightly different saves, and 12 HP on average. None of that can't be replicated with a spell, and a lower-level one than 7th as well. So at level 20, in the most optimized builds, it's almost never worth it to take these levels. The exception is for a few gish builds that can certainly compete if not shine where fighter 1 is an easier choice to gain various weapon/armor proficiencies for prestige class prereqs, but it's a tax more than an advantage.

But--and here's the thing--not all of that is true at lower levels. Then, gaining extra BAB, HP, and proficiencies can mean a lot more, especially if the career won't go beyond ~10. Those boosts are a static benefit, while casting is one that increases over time as an investment. So the balance looks a lot different at lower levels. As an example, I once built for a lvl 3 one-shot a Warlock 2/Barb 1 that was dealing 1d12+1d6+5 damage per attack while raging, had a +9 in diplomacy (my lowest social skill), decent AC, and a lot more HP than I needed. It was awesome. It would have failed to keep pace with anyone else in the party, but that was OK.

So yes, in low-level campaigns, it can be fine to substitute martial for caster levels. The trick is having a good sense for how far the mundane is going to keep you alive, and for what's actually giving you your power. If you have the stats to pull it off, for example (High CHA, decent STR and CON, preferably good WIS), Paladin 5 / Sorcerer 4 is a perfectly fine combo, as is rog2/wiz7, or similar combinations. Are they going to shine over the rest of the party the way a T1 caster does at high levels? No. But the power gap is much lower at those levels, to the point where the "best" class can be hard to see. So, by all means, knock yourself out.