That's not how a space elevator would be used. The design allows for multiple climbers on the tether at any given time. Yes, each one takes days to arrive, but once the first one arrives, the next one is just a couple of hours behind it. Also, the pods will be disposable: either flung into deep space, or put in a shell and dropped back to Earth. But they won't be taking up tether time to go back down.
GW