Typically, you'll want to pick 4-6 perk trees that you use a lot and invest most of your points in those.
I would like to reinforce this advice, and supplement it with my own: Plan your perk build, from the beginning, and keep your planned perk total to 49 perks (enough to see you to level 50). You can certainly get more perks if you raise every skill in the game to 100, but you'll find that a long and tedious journey.