I still think the number can be explained down by a rotation system.
One additional point of complexity to explain "needs two years to recover" - each crystal needs a slightly different {some mystical property} in its Myrrh. The trees are signaled to produce this by the chalices, which come from the crystals. A tree needs two years recovery to produce the same kind of Myrrh as the previous year, but can produce a different kind every year. That would enforce a minimum cycle time of three years, which means that one crystal still needs nine trees for full support, but those nine trees can support up to three crystals.
That would also explain why you can pass a caravan going in another direction (where you can't harvest anything) in some events, and why there's only one caravan per village (only one chalice, because the chalices match the crystals).
EDIT:
I made a map! Towns/dungeons to follow.
Spoiler: image