I think risky magic is a fine thing for the game to include. The Hellfire Warlock is an interesting class, and having a character where your big abilities caused backlash in various forms is potentially interesting.

But it is, in my view, very obviously not how all magic should work.

Most obviously, there are a bunch of characters who manifestly do magic without assuming the risk of going crazy or turning into demons in the source material. Harry Potter never has to worry that casting too many Summoning Charms will cause him to go insane. Gandalf's magic is limited, but not because he's worried that going all out will cause reality to break down. Even settings that do take the "magic is inherently dangerous" stance often allow protagonists to somehow avoid that danger (see: the Laundry Files). There are definitely people who are worried that if they do too much magic, demons will eat their brains. But there are also people who don't worry that, and if you hardcode "magic makes demons eat your brains" into the game, you've cut out all of those characters.

But there are also problems with most of the implementations I've seen suggested.

Sometimes the suggestion is that magic has a low risk of some terrible consequence. You make a Spellcraft check, and on a natural 1, demons eat your face. That might be balanced in the abstract, but what happens practically is that casters stomp on mundanes for a while (which isn't fun for the mundanes), and then get killed randomly (which isn't fun for the casters). I guess that's better than having casters stomp on mundanes the whole game, in that you have now spread around the suffering, but it seems like stretch to call it "good".

Sometimes the suggestion is that magic attracts demons or other monsters. That's something you can do, but again the practical effect isn't what you want. Now in addition to having powers that (potentially) obsolete mundanes, they also generate adventure hooks. In your effort to reduce the power of casters, you've put them in the spotlight more.

Sometimes the suggestion is that magic somehow makes you crazy or evil. Again, tempting, but it functionally means that casters eventually stop being PCs, which is the same as having a level limit, except it is less fun for the people playing casters.

I just don't see how "magic hurts you" is supposed to work in a way that actually achieves the nominal goal of making casters less dominant.

There are two possible solutions here. You cap character power, or you let mundanes get things that are "magic". Everything else is a distraction.