There are consequences and there are government consequences. The latter is bad, the former falls under free speech.

A person can say <controversial opinion> and in response others can choose to no longer associate with them, do business with them, etc.

(There is an issue when the primary means of communication become privatized, but that is an area I've yet to work out.)