Yeah, I'd have to agree. The party may end up protecting the bandits from the people who wanted them gone. Hopefully not in an actual fight, though. I wonder if emnity toward non-humans is so high that the locals would try to frame the bandits for anything (can't help but speculate).