The doctor has gotten older too, at times. If you look back to the transitions 5 to 6 to 7, each of them was older than the previous.

I think the original intention for Matt Smith's doctor was to pick someone older for the role. I seem to recall an interview from around the time saying that they really wanted someone older, but Smith just knocked it out of the ballpark in his audition so they took him anyway.