They don't "lose" anything, they just can't cast Wizard spells while wearing Medium or Heavy - similar to Slipjig's proposal.
It's functionally not that different but I think it would be more intuitive in practice.
(Note that this is purely a suggestion on my end; I'm not the one who has a problem with wizards in plate.)