According to
this guy a mother needs to produce four children who survive to adulthood between puberty and menopause. Assuming that time is betweem 14 and 40, that leaves enough time for nine births with two years of nursing between them. If half of those births result in offspring which survives to adulthood, mission accomplished.
A failed pregnancy which leaves the mother alive only takes a year out of her fertile cycle, so it would require multiple miscarriages or infant deaths to impact this average. So, math time:
Each successful pregnancy is followed by two years of nursing before another pregnancy, so 3 years per success. With a 27 year window of fertility that's 9 children.
Each failed pregnancy takes a variable amount of time, from a few months to a few years, out of that time frame. For simplicity I assume arbitrarily that that time averages a year.
Thus:
9 successful pregnancies
8 successes and up to 3 failures.
7 successes and up to 6 failures.
6 successes and up to 9 failures.
5 successes and up to 12 failures.
4 successes and up to 15 failures.
Infant mortality is far less a concern than mother mortality. A rate of 75% infant mortality would be sustainable if the mother is otherwise unaffected and remains healthy and fertile.
Since four is our target number, we won't bother beyond this. But this leaves the question of how many children survive to breeding age? The number of boys is not relevant for this because only one is needed to fertilize a tribe of females. (Gengis Khan, I'm looking at you!) So, we can use expendable boys to defend against lions and rival tribes, and only concern ourselves with the girls. If 25% of girls die childless, that then requires the survivors to have 5 children who survive to adulthood. If 50% die childless, it then requires each successful mother to have 8 children survive to adulthood.
So, if child mortality is less than 25%, and death in childbirth is less than 25%, the surviving mothers are required to produce six children each for the averages to work out to a growing society.
In other words, combined infant, child, and maternal mortality is a non-issue until around 40%. The culture would have to adapt to this reality, but humans are adaptable.
Note: this is a very simplistic analysis which ignores many factors. One could write a dissertation on the subject and barely scratch the surface. My assumptions are very broad and very imprecise. So, take it for what it is, and don't assume it serves any purpose but to broadly explore the OP's question.