You could look at something like Amnesia: Dark Descent and Silent Hill for quite a few good ideas in how to just screw with people.
Minimize their field of vision to keep things moving at the edges, change the dimensions of the space they're walking in as they progress, eerie sounds come from the abyss and disappear once investigated. You could personalize their madness, have voices speak to them individually, show them past memories twisted darkly, and bring back dead people they've known as if they were never gone. Mess with their perception of time have them meet themselves from an hour, ten years, thirty years, 50 years hence -- still wondering the space. Have an area where the walls all project distorted mirror images of their characters, only to have them come to life and attack them. Make the whole area into a nightmare by MC Escher, with lots of dead ends and non-euclidean architecture