Keep in mind that what we have here is not the whole wide Culture, just the group of organics that arrived on the colonization fleet. Plenty of people, sure, but they don't grow in numbers nearly as fast as the ships can and they're nowhere near the numbers of the full Culture.