Re: I'm sending my players to a demiplane of madness inspired by Xoriat. Tips?
Without giving it any deep thought, the first thing that springs to mind would be selective illusions. Create encounters where integral parts of them are a selective illusion.
For a low level example, I would say a chasm with an illusory bridge. Half the party sees the bridge as real, the rest see it as an illusion or don't see it at all. The trick is that the ones that actually see the illusion as real can interact with it as if its real, while it behaves like an illusion for the ones that see through it.
Illusory bridges, illusory pillars holding up a ceiling, illusory walls blocking pools of acid, etc.
After a couple of encounters like that, they just might stop trying to see through illusions and treat everything as real, because it was in their favor before. Your BBEG then throws plain illusions to mess with their heads.
Just a thought.