5/5 gets better spellcasting, but 6/4 is more resilient.

If you're in melee range, you'll be hit. If you get hit, you'll make CON saves. If you have a bad, or even a mediocre CON save you will lose concentration and be in a bad place. Haste, if backed up by a crappy CON save, is more of a liability than anything else, and that's arguably the best spell at that level.

I would personally skip PAM and pick up warcaster. You'll have plenty of use for your bonus action anyway and once again, you really don't want to lose concentration in the middle of a fight.