The entrance is guarded, there is nothing near the single gate that you could hide behind to sneak in that way. Each woman has to show a carved figurine to the guards before being let inside.

However, there is an open window near the corner of the building you could probably climb through before the guards notice. Several towels are hanging out the window.