I would say start with the Bard's spells known. It's already capped at 6th level spells, and a smaller number of spells known that the Sorcerer.

Spell failure won't be a fun way to balance the class. You're randomly determining if you can't do anything that turn, with no way to improve it and no way to avoid it. Far better would be to simply increase the casting time. Full-round spellcasting for 4th and 5th spells, and up to 1 round spellcasting for 6th spells would really cut down on the spellcaster's mobility. Of course, this doesn't stop them from spamming Charm Person or Glitterdust all over the place.