Apologies. The Pathfinder duelist retains functionality in light armor. I assumed yours did as well, quite erroneously. I retract my critique of the Super Duper Parry.
Ah yeah, if you thought it kept light armor then I can understand the complaint. +10 Armor is still pretty high compared to light armor (even enchanted), or compared to the Monk's armor bonus, but on the other hand taking the -4 penalty to hit I think makes up for that. I'm open to argument if anyone feels that isn't really the case.

No, the challenge reducing movement makes sense- you are harrying your foe constantly, and in the time it takes them to get away from you, they don't have the ability to move at their full rate- in other words, your constant threat of giving them a cool new lung-piercing that all the ruffians are wearing these days, takes a great deal of their attention.

Then we go from that to "quickened dim door doesn't work for no reason".
Well, it may be from a different ability, but before you get to "quickened dimension door doesn't work", you get:

Improved Opportunist-As of 5th level Casting defensively, or using a Magic Item provokes an attack of opportunity.

So we have the precedent of using magical abilities being hard while within reach of the duelist, and it's pretty hard to run away mundanely, I felt the logical progression was to make it especially hard using magic to get away from the duelist. The entire point is the duelist doesn't have the best mobility (yes extra 5 foot steps are amazing, but he doesn't have pouce, fight, teleportation, increased move speed, or any other cool movement forms), so his method is forcing the enemy to stay next to him.

Yes, it may be a bit more targetted at high magic creatures like demons and such with su at will teleports, but on the other hand, it's a relatively high level ability, coming at minimum 4 levels after casters have started having easy access to teleportation spells, and it makes sense that the duelist would have some way to deal with it.

If you want this guy to be threatening, then probably you should have something that ups the DC of concentration checks to cast defensively, or allows him a chance to lock the opponent in place, or check out the "step-up" feat from Pathfinder- a good version of which you have in flawless footwork. Some mix of these should make an enemy caster regret standing next to you more than normally- but I'd argue that if he's dim dooring away, he already has had all the rapier he can eat.

I'd also point out that a move action followed by a cast normally will ignore this ability anyway.
Most of what you suggest is already in there. As you mention, the fancy footwork abilities gives a bit of off turn mobility, and Perfect Opportunist does increase the DC of concentration checks invoked by any attack of opportunity, and Improved Opportunist makes defensive casting still provoke, so if a caster is adjacent to you, and tries to cast, they will invoke an attack, and if you hit, that DC will be increased.

Maybe the dimensional anchor is a bit over the top on top of that, I'm just painfully aware of how easy it is to optimize something like a concentration check to point that any reasonable DC is going to be shrugged off easily. I could see replacing that with something to restrict basic movement further, but I'm still not convinced stopping teleportation effects is a bad thing. As you point out, if someone tries hard enough, they can still run away and dimension door, or whatever, but making it so they can't do it while adjacent to you makes them waste more actions doing it, and maintaining an action advantage is important, especially when you have very few ways to take advantage of Swift/Immediate actions (which is one weakness I feel is still left with this class, very limited access to the action economy, but a lot of passive benefits to what it can do)

Also as a warning- if you want a prestige class to use a certain weapon, you should flat out say that. If you grant a general ability (such as mastery of piercing weapons), understand that this applies to rapiers, but also to exotic things that are better than rapiers, as well as the spiked chain. Granted, using the spiked chain turns off some of this guys abilities, but generally in designing prestige classes you have to worry about this stuff. Of course, if you are ok with all that, then never mind. But if I were you, I'd make a list of weapons that this works with.
Honestly, if someone wants to make a Duelist with a spiked chain, that's their choice... but the class really doesn't synergize with it. Sure, there's a lot of attack of opportunity stuff here, and a lot of attack of opportunity stuff with spiked chains... the problem is the spiked chain is a two handed weapon, so opportunist, elaborate parry, penetrating strike, or precise strike won't work. That's most of the stronger class features, and the ones a chain tripper would be most interested in, do I don't think that particular example has any real problems.

The reason I don't want to make an exact list, is because I know there will be items that are added later, or I'm not aware of now, that would be fitting. I figure any piercing weapon that can be held in a single hand and be finessed is fine for a duelist, and any other piercing weapon is losing access to enough class features that it's probably fine regardless.

The rogue has some limitations that the duelist does not, including a bad hit die and a worse BAB and flanking requirements. I still agree that 2d6 is just too damned low, however. I'm not sure which is better, but they do appear to be comparable. Given that you would expect the duelist to be using a +crit weapon, it's nice if it can crit.
Well let's crunch some numbers. On a high crit weapon you're looking at threatening a crit on a 15-20 (assuming you have keen/imp crit), assuming you hit on a 8, that means the 5d6 gets you 17.5 damage 65% of the time, for 11.375 average bonus damage.

10 damage gets you 10 damage 40% of the time. 25% of the time you threaten a crit, which gets confirmed 65% of the time, so 16.25% of the time is a crit, with the other 48.75% of the time as a normal hit, and 35% miss. 4.875+2.64 = 7.515 average damage. So the 5d6 is actually about 50% better. (Worth noting, 2d6 is 7 average damage 65% of the time for 4.5 average damage per round, making the +10 flat damage is almost exactly dead between the two options)


I'm personally still inclined to lean towards the stronger end of things because you're giving up a lot of benefits of bonus strength damage and stronger weapon dice, or extra attacks per round.

Rename Riposte? Hrm, my suggestion would be something emphasizing the accurate and precise nature of the attack, without the name also implying that someone has to attack you first. Instant and accurate. Hrm. Mongoose Reflexes. Precise Strike. Relentless Opponent. Improved Attack of Opportunity :P ... Cunning Strikes. Press the Advantage. No Recourse. Unforgiving Opposition.

That's all I got for now, sorry.
I think I've got enough "strike" abilities in here, so I want to avoid those unless I make it a improvement of one of the other abilities, but none really stand out as a good place to put it. Mongoose Reflexes, Press the Advantage, and Unforgiving Opposition all seem good. Right now I'm leaning towards Mongoose Reflexes or some variation of that. Thanks for the input on that though, like I said I really am terrible with names.

As for what would make me want to use it instead... I'm not quite sure. I would obviously replace the dim anchor with something else that hinders the foe. Three five foot steps in a round gives this guy a lot of advantages over other melee as well, but I didn't really critique that because you probably have other stuff that makes that ok in your games (like you probably allow lion totem and other pouncy-things). Overall I like the duelist in light armor more than no armor as well, though both have appeal.
Yeah, the extra 5 foot steps are big, but my opinion is any melee that doesn't have maneuvers or some similar method of getting more out of a standard attack should have some form of increased mobility. That's a part of my general design philosophy, simply because without it, they can't compete. I'd actually consider the extra 5 foot steps, while big, less effective than other forms of mobility (such as pounce, and especially such as more supernatural abilities like flight and teleport), which is why there is such a high focus on control in the class.

The 3.5 duelist is of course, in line for a giant stack of buffs- I just think that Pathfinder's is pretty much on the money for what I'm looking for (it's actually rather weak for a Pathfinder class, but since I'm running a somewhat restricted 3.5 it's about right).
Honestly, Pathfinder did a terrible job with melee. They advertise themselves as fixing imbalances, but in many cases from what I've seen they became worse. Most of the problem spells still exist in the game, and they didn't do nearly enough to buff the melee classes to compensate. If you play an extremely low op campaign, where wizards do nothing but toss fireballs, and Fighters are considered very strong, the Paizo duelist may be fine.

Outside of that it suffers the same problems as the core Duelist: it lacks any real role that somebody else couldn't do better than them without all the drawbacks of the class. A prestige class should make you stronger, or at least be power neutral, not make you lose damage compared to not taking the class and using a different fighting style.