It really depends what style of encounter you want.

Sure conjurations like Cloudkills, Solid Fogs and Evard's Black Tentacles could run amok, and enchantments like Incite Riot/Friend to Foe cause headaches, but for thematic purposes it might be more memorable to go with transmutations.

While the Alienist prestige class works on a different principle, you might want to tweak it to fit your mad scientist.

Keep the focus on the scientist's creations. Shock the PCs with a spell that temporarily merges two animals/monsters into one bizarre but deadly creature...another spell that gives them extra arms or a unicorn horn....make them 50% immune to crits/precision because their anatomy is so foreign...one of the creatures could appear somewhat humanoid and they realise it's a female about his age, maybe his wife??

In short, never have him directly attack the PCs but make sure the strange buffs result in minions that really stretch the PCs capabilities.