I argued that they're both cultural (social) and physical in the pony world. (It might not be very relevant, but let me mention [url=http://slatestarcodex.com/2014/09/10/society-is-fixed-biology-is-mutable/]a blog entry of Scott Alexander on problems caused by society and biologicy. [url])
As for getting up the clouds, I don't think that would be a too serious problem, but maybe. After S5 E5, I was thinking how the Mane Six got up to Rainbow Dash's floating house, and determined it was pretty easy. Twilight and Fluttershy would fly up, attach a long rope ladder to the cloud and throw it down. Applejack climbs up: we know she can climb rigid ladders quite well from S4 E20, so I think it's not much of a stretch to imagine she can climb a rope ladder as well. Then they attach a large basket, like the one on the Balloon, to a long rope, and throw it down. Rarity boards the basket, and then Appljack together with the other two are strong enough to pull her up easily. Finally, Pinkie Pie off-screen teleports up at a convenient cut of the scene, which she's allowed to do because she could get up anyway with any of the three methods shown in S1 E5.
I brought it up because it was done, at least that's how I interpret S3 E7. The Wonderbolts Academy was in a flying island that had trees growing on it. I think that ground is also solid enough for earth ponies to walk on it. Then, when the ponies fall off the hot air balloon, Rainbow Dash quickly whirls around a cloud to turn it to dense enough so that she can catch all five ponies with it. I don't think that Applejack and Rarity had cloud-walking spells still active on them during that, unless it's such an easy low level spell that Twilight could make it permanent, in which case it the spell is probably also easy enough to learn for other unicorns.
On the other hand, I'm reminded to Arthur C. Clarke's novel Childhood's End. This describes a human who visits a planet of flying alien creatures. (I think the planet also has much lower gravity than Earth, which helps the large creatures fly with their wings. This is a science fiction novel afterall, they don't just get to fly with pegasus magic.) The city on that planet has no streets, the buildings are simply built next to each other, and the creatures simply fly from one building to another. This makes it difficult for the visiting human to get around, because he can't fly up to the doors on the top of the building.
Yes, let's hope this is true then, until the show gives new evidence. We'll probably see more remote parts of Equestria in S5. Hopefully they'll be like Our Town, with all three kinds of ponies.