OK - maybe the Aboleth is too obvious?
But one of those could mess up the party single handed.

Insanity and Confusion etc. are probably something you want to use sparingly, since they mess up player empowerment. But combine Confusion with an otherwise easy fight and you might see some hilarity.

You could also mess with gravity? Have them walk through a Klein bottle or something. To make this a challenge instead of interesting scenery create a small maze with variable gravity.

Illusions are best when they don't attract suspicion, but on a demiplane of madness that could be tricky.