In the bad old days in Western societies, a land-owning elite controlled everything. There was no way for those less privileged to gain enough money to buy land. Early capitalism began to change this with the emergence of a "middle class," one that grew wealthy from trade, production, finance, etc., sometimes in "free cities," rather than through land ownership. This shift away from agriculture freed people from servitude to the land and the lords that owned it. A slow decline set in among the aristocracy.
Or did it? There is a new land-owning class that has become good at keeping itself up and preventing others from joining its ranks
When we look at modern-day New York, London, or San Francisco, we see playgrounds for the rich. Housing is so pricey in these desirable places that few "normal" people can afford to live there. It doesn't stop in those world cities, either. Housing prices throughout much of England are so ridiculously high that even families with slightly above-average incomes find it difficult to buy a home. In many of the wealthier, liberal areas of the US and the UK, at the very least, housing costs an arm and a leg.
Why?
The free-market answer is: "Because more people want to live there than there is housing for them to live in, so housing goes to those who are willing and able to pay a premium for it."
So far, so good, but it doesn't really answer the question. Lots of people are moving into Houston, Texas, but housing there is still affordable. The reason is not so much on the demand side (people wanting to move in) as it is on the supply side (there aren't enough homes). The main culprit? Zoning (UK: planning) rules. Minimum lot sizes in the countryside, maximum building heights in the cities, land usage restrictions, and historical designations all serve to restrict the space available to newcomers.
There are good reasons for many of these rules, from ensuring ample ground water and septic capacity to protecting historic buildings that are worth saving. But as Edward Glaeser, in particular, has argued (check out his book Triumph of the City), many of these rules and protections seem to be excessive. They also cause hidden harm: the rise in property prices is a nice thing for those who own property, but the mechanism by which this happens is unclear to many, meaning few protest against zoning rules because of the high housing costs they create.
The real reason for the heavy-handed implementation of well-meaning rules on who-can-build-what-where, I would argue, is that they benefit a new land-owning class. It not as conspiratorial as it might sound. Here's how it works: Once people in an area have purchased housing, they no longer have an interest in affordable housing, since they are no longer looking to buy. In fact, the more expensive housing gets, the richer they feel (and this is regardless of whether they're still paying off a mortgage). If housing prices rise elsewhere and they might eventually want to move, no matter, as long as their housing rises in price at least as fast.
One way to make sure of this is to keep a place attractive to newcomers. This, on its own, is a good thing. Another way, though, is to restrict the overall amount of housing available so that only wealthier residents can move in. The two "ways" go hand-in-hand, actually: One way to keep things nice is to keep them rural. Once wealthier people move in and land values rise, tax revenues for the town rise, meaning more money to spend on better schools, nice parks, etc., making the place more attractive and raising housing prices further. This is a "virtuous cycle" for those who already own housing. It is not so virtuous for those who don't as they become "priced out" of the market.
Things can be kept "nice" without restricting building, however. This is particularly true in cities, where the argument for keeping low densities is weaker. It's important to have public green spaces like parks. But this is actually all the more reason why built-up areas should house as many people as efficiently possible: higher density in built-up areas means fewer people elsewhere. The result? Less sprawl and better conservation of nature and rural character elsewhere. A further result is that more people can afford to live where they'd like to (like Manhattan) and closer to work, meaning less commuting, traffic congestion, and air pollution, as well as less wasted time, meaning more free time and higher productivity. All this would save workers, consumers, and employers money, which would mean increased economic output per hour worked and dollar spent, and thus a higher standard of living for all.
These are massive benefits, but they are dispersed and difficult to see. That is why, so far, the new landed classes (they have "captured" the political system in rich areas as they form the bulk of an area's voters) have succeeded in giving themselves goodies at the expense of the real estate-less. It's time for some serious thought and sensible changes.