When they shouldn't. They are still at a disadvantage, since Xykon hasn't done anything since he's so scared about his phylactery being destroyed-which wouldn't actually kill him as we learned in Start of Darkness-that he didn't do anything.

Also, how exactly is fighting the Paladin one-on-one a stupid risk? A Cleric of Redcloak's level would be able to defeat a Paladin quite easily, especially that he is very powerful spellcaster and Clerics are known to be at par defensively against Paladins.