Not being born but having 2 rhd adds to your ecl so you need to fight monsters of a cr two higher to gain levels at the same pace.
Believe me a fighter with one level and two rhd is not nearly as strong as a cr 3 encounter(unlike a fighter with three levels which is balanced with a CR 3 encounter(50% chance of wining)).
So basically a wizard that starts adventuring as a child will have only wizard levels while a wizard that starts as an adult will have two rhd and so will be two caster levels(and one spell level) behind the child wizard quickly and the maluses to the stats of the child wizard will not be a penalty overcoming two extra levels in wizard(probably the same can be said of a fighter/barbarian/a whole of other classes).
So with your system people should start having classes and/or adventuring as fast as possible for avoiding the point of non return of becoming useless forever which happens when you become adult and have 2rhd.