Getting someone fired, cancelling their appearances at universities and other public events, blacklisting them from academia -- those things are not speech.
Doesn’t seem a good idea to limit responses to speech. Speech drives action often times and it should. Someone may protest because of something said, they may not want to associate with a person etc. I’m not sure I see a huge difference between me not wanting to hire you because you are a jerk (based on my interactions) vs not wanting to hire you because I think you are a bigot based on your podcast.
What about boycotting businesses that employ them? Is that speech?
It's the employer who terminates employment, not the people asking the employer to do so. The opponents of the people asking the employer to terminate employment can exercise their free speech to argue for continued employment. That is the only action congruent with free speech absolutism. In this case, it also happens to match what the law protects.
Of course those things restrict speech for that person -- and they cause a chilling effect for anyone else who might want to say the same things.
You might as well also say that jailing people for their speech doesn't infringe on freedom of speech, because that is a consequence of speech they already made. But that's a very narrow (and ahistorical) view of what freedom of speech really means.
We can lean on the Wikipedia definition here: "Freedom of speech is a principle that supports the freedom of an individual or a community to articulate their opinions and ideas without fear of retaliation, censorship, or legal sanction."
Getting someone fired, cancelling their appearances at universities and other public events, blacklisting them from academia -- those things are not speech.