Isn't there a quote somewhere that says "Free speech is meant to protect unpopular speech. Popular speech, by definition, needs no protection"? "I disapprove of what you say, but I will defend to the death your right to say it"?
If you are not for the free speech of people you disagree with, then you are simply just not for free speech.
Nazis and the KKK, while they might not quite make calls to violence, know how to word-smith the hatred and grow it, like a verbal "I'm not touching you" kind of thing. They'll make ambiguous statements like claiming they need to deal with the $SLUR problem in the neighborhood.
While I despise Nazis, KKK and the likes from the core of my heart. And while I find the limits on free speech in German law reasonable and well proportioned given our history (for example you are not allowed to deny the holocaust having happened publicly) let me still quote something (translated with DeepL) from somebody who lived through the Third Reich as a victim:
When the Nazis came for the Communists, I kept silent; after all, I was not a Communist.
When they took the trade unionists, I kept silent; I was not a trade unionist.
When they took the Jews, I kept silent; I was not a Jew.
When they took me, there was no one left to protest.
The basic question is, how far is one willing to tolerate intolerance? This is a philosophical as well as an ideological question imho without right/wrong answers.
It is an essential question, meaning people won't budge over it but fight over it. There is no middle ground. One is either on this or on a different side.
As said, while I basically agree with your sentiment, one must accept, that even if I don't like what they say, if a Nazi states the they think all foreigners need to leave the country. And as long as they do not argue for violent action, this is an opinion they need to be allowed to have in a free society.
As also said in some other place, I agree with the German way of prohibiting the public denial of the Shoa or the usage of the Hitler salut. This is punishable by law here as the early democratic German society decided that these were red lines given our history.
But also, while I would argue against it, a Nazi is allowed to hold and voice the opinion, that these laws are repressive.
So I think there are no easy answers, but a society must decide how far they allow free speech and where it draws red lines. This must be part of a democratic process and must be discussed in a public way. And it might be that these red lines move over time.
But I would go far in what is allowed in speech or writing, as I would not want to create precedent that later can be used to censor other groups as well. But that is only my personally held opinion and others should differ on that to enable a healthy debate.
If you are not for the free speech of people you disagree with, then you are simply just not for free speech.