Prop 13 contributes to making this situation worse, but the scenario you laid out is actually how development happens in the US, regardless of where you are. Municipalities at the edge of a built-up area realize there's an overspill of demand, and they can attract development because they can offer lower land values, higher lot sizes, and lower property taxes than areas that are already built out. Typically, residential demand is higher than others, and private developers persuade town officials to let farmland be upzoned for homes and subdivisions. Typically, everybody wins a little: the municipality gains significantly more revenue from property taxes on the re-assessed land, developers, barring a sudden downturn, usually turn a profit, and potential residents have more options for housing.
This works well for greenfield development. But once a town is built out, the calculus is different. For developers, the costs of buying out adjacent parcels is greater, and the marginal profits are less. This is why you predominantly see "luxury apartments" built during redevelopment, instead of, say, middle-income walk-ups, to improve margins further. These forces cause them to gravitate towards greenfield development further out (e.g. Gilroy, Tracy, Stockton, Woodland), or to focus on redeveloping areas that have the highest profit potential (ex-industrial urban revival districts).
Meanwhile, for the towns themselves, 30, 40 years after having been built out with single-family homes, the benefits of further residential development have all but evaporated: the sewers need replacing, and the = schools require more and more funding to keep at their target level of service to retain discretionary, high-income residents. Smart suburbs typically shift their focus to attracting employment, like manufacturing or office space. The Bay Area is much more fortunate than other regions of the country in this regard, as most of the employment base developed organically, rather than having to attract out-of-town businesses to start satellite facilities. Nonetheless, communities still vie with one another over employment, as new jobs are much lighter on the budget than more residents. Ever since streetcar suburbs were established, the natural state of development has been that towns in earlier stages of their existence would pick up spillover residential demand, while built-out areas focus on attracting and retaining jobs.
This works well for greenfield development. But once a town is built out, the calculus is different. For developers, the costs of buying out adjacent parcels is greater, and the marginal profits are less. This is why you predominantly see "luxury apartments" built during redevelopment, instead of, say, middle-income walk-ups, to improve margins further. These forces cause them to gravitate towards greenfield development further out (e.g. Gilroy, Tracy, Stockton, Woodland), or to focus on redeveloping areas that have the highest profit potential (ex-industrial urban revival districts).
Meanwhile, for the towns themselves, 30, 40 years after having been built out with single-family homes, the benefits of further residential development have all but evaporated: the sewers need replacing, and the = schools require more and more funding to keep at their target level of service to retain discretionary, high-income residents. Smart suburbs typically shift their focus to attracting employment, like manufacturing or office space. The Bay Area is much more fortunate than other regions of the country in this regard, as most of the employment base developed organically, rather than having to attract out-of-town businesses to start satellite facilities. Nonetheless, communities still vie with one another over employment, as new jobs are much lighter on the budget than more residents. Ever since streetcar suburbs were established, the natural state of development has been that towns in earlier stages of their existence would pick up spillover residential demand, while built-out areas focus on attracting and retaining jobs.