Lemme draw you a comparison. Imagine a street gang, right? Doing crime things--busting up shops and people, destroying businesses, shoplifting and stealing. But they're just doing it for fun and pocket change, there's no pattern to it, and no overarching goal. Often, criminals work against each other thanks to poor coordination.
Now imagine that same gang, but with leadership. Now it's organized crime. The actions taken are done in order to destabilize the area, to make sure that the police have no power, that the gang is the one people turn to when things need to get done. They're strong, they work together, their actions are coordinated for maximum effect.
Which gang is scarier? The disjointed group of criminals, or the organized mob?
It's the same way with the Institute. Right now, their operations are disjointed, scattered, working at cross purposes. Their actions are evil, yeah, but hardly organized. They're only threatening because there's no organized competing faction until the Brotherhood arrives. They're nothing compared to what they might be with some actual organization. Instead of sporadically replacing farmers, they could seize control of the region.