This is the case for some community centers, but definitely not all of them. It’s not the case for any of the 4 or 5 that are closest to my home. Many are run by private organizations.
Alternatively, it would be like kicking out the person trashing the place or screaming obscenities and making everyone uncomfortable, which is generally accepted in the real world.
Sort of, we’d only kick out the lout we disagree with but let the other lout that we agree with continue spouting obscenities and we wink at each other and congratulate ourselves on our taste.