I think the DC of 5+3x spell level might be a bit high, especially since SR becomes a penalty to this check.

2SL + X would work better as a DC, since you're making a caster level check, and you get new spells every two caster levels, where X+1 is the target number you want for your biggest spells (that way, a caster at level 17 has to roll X+18 on 1d20+17 to cast a spell, and an apprentice at level 1 has to roll X+2 on 1d20+1 to cast; both have to roll X+1, before modifiers). Possibly even make it that you have to make a DC of X+Spell points to spend, so that you have to roll exactly X to spend spell points equal to your level. Even stacking intelligence on won't make this too hard.

Armor ASF penalties can be changed into a penalty on this for spells with somatic components; just divide by 5 and then you have your Armor Spellcasting Penalty (casting with 50% ASF changes to a -10 penalty on your casting roll)

Honestly I'd be in favor of a bit of caster MAD; it wouldn't utterly destroy them. I'd use INT to learn spells (making it an INT-based Spellcraft check is enough; possibly make it so that you can choose to forget spells and that the difficulty goes up for each spell you learn after a certain number (some function of level and int mod), WIS for bonus spell points (it's your willpower; how long you can keep forcing your internal energies onto the world), and CHA for DCs (it's how well you force your mind onto the world); I'd probably allow any one of them to determine the highest-level spell you can cast, though.

At the very least, keep CHA as a casting stat, somehow, even if it's just MAD between INT (knowledge) and CHA (DCs and Spell Points).

Also, I'm assuming all spells are drawn from a super-list containing the Cleric, Druid, and Sorcerer lists, as per the Generic Caster?