Well when they had the underground railroad of telepaths, there were what... about 15 of them on Babylon 5? Figure when somebody gets there they were on the station for somewhere around 1-2 months while Franklin figured out their papers and destinations and you're looking at somewhere around 100 rogue telepaths being shuttled per year through Babylon 5.
That's a pretty large number and points to the overall community being in the tens or hundreds of thousands at a minimum for the rogues to be that big a community.