The doubling at 20 is fine. It gives you the power to start standing up to the super wizards before they get epic spellcasting. That really isn't the issue.

The issue is basically how does singularity handle lots of people going to the same point. Even without the doubling that's still like 30 people being pulled to a single point in space. That's what the issue is.