I think it was a combination of "they DO trust him" with why they handed him the spear, and them wanting to show that, and a combination of "the fact that they got screwed over by Past Snart got them upset". And like, the spear calls to people, maybe it's messing with everyone's heads, putting them on edge. Literally everyone on the team has baggage they'd love to fix. Also, while Stein knows that Rory thought it was hallucinations, he's also definitely the type of guy to play armchair psychiatrist and kind of infer that sort of thing, especially if there is this underlying sense of "hmm maybe Rory isn't trustworthy" which...let's be fair, even though they've had some funny and good moments in the past, I do feel it's believable that they might not 100% fully trust him, even after all that's happened.

Also, I do feel the reason why this was "less fun" than the Lucas one was because this episode takes place during WW1 and part of the episode was reinforcing themes of suffering and great emotional pain.