Can't win with this one. Either you're the modern day segregationist for wanting to live in a nice neighborhood, or you're gentrifying by not doing that.

Fortunately there's also a large number of people, probably a majority, who don't give a shit about this political nonsense and just want to go on with their lives.