Kill 'em all. It's not like they won't attack you if you pay them. They are bandits after all.