For the spellcasting question, one thing could be that only Magic-focus can buy full spellcasting. Spellcasting akin to 3.5's Duskblade or Beguiler, or half-casters like Ranger/Paladin, are still options for anyone, but full spellcasting is limited.
I'm not saying that's the best way.
It might be better to assume some light spellcasting (at least) will be part of each character, akin to things like Arcane Trickster or Eldritch Knight archetypes in D&D 5th edition.

---

I think your overarching idea is a good one. You might find inspiration from some 'build your race' guides, where racial pros and cons are given point-values. However, I think a system using this would require a sense of "don't break the game", much like Mutants & Masterminds has. Balance is tricky, and even someone not with a lot of system mastery can pick combos that (intentionally or not) are game-breaking in M&M. The level structure of D&D makes it less likely to accidentally make a broken class, but probably more likely to intentionally make one (especially one that is not noticeably broken at first.)