SpoilerThere was a nice theory floating around about Aegon, actually:
He's not a Targaryen. He's a Valyrian, but not a Targ.
Because in the same book, other Valyrians with similar white/purple colouring where first mentioned. The conspirators found themselves a baby in the correct colours and raised it as Aegon, then made up a nice true-prince-in-secret-exile story for the commoners to believe.