I see three different combat styles for my idea of "swashbuckler"; a sword and dagger (TWF) style, a sword and shield (it's called swashbuckler for a reason), and a one-hand sword style. I think all three of those are supported fairly well, though you're right that Power Attack is still an option. What about going with 3/4 Bab? It would weaken the class a lot but it'd also let me give him two good saves as well as enforce the idea of "mobility fighter" by making full attacks less effective. Power Attack also becomes a lot weaker with less Bab, so you're more dependent on maneuvers. Extra Granted Maneuver might not even be so overpowered then. PrC access would be harder though.
On a side note, a lot of the favored enemy abilities are a bit of a stretch (literally, in the case of aberrations). If anyone can help me think up better ones, that'd be great.