This is a real dynamic seen times and times. The catch is, poor people's houses are not owned by poor people. "Rich" people don't need to live there to own the land and dictate prices up to a point. They'll flee away and rent the house basically, and they can afford to keep it at the highest price the market validates.