So a Fighter that multi-classes into a Wizard loses 2 levels of armor prof? And what about shield prof? Shields will usually be the more problematic part for casters not armor.
The real answer is just making armor prof harder to get, and increasing the Str requirements for medium and heavy armor so they require at least some attribute investment to function fully. Of course the first of those is easier said than done because it requires changing the class benefits, and a reason to start in a caster class over a class with lots of profs (assuming the profs would be beneficial).