Do you believe that such a speech should have no consequences whatsoever?
Of course not. We should absolutely hold Trump responsible for what he says. The disagreement here is whether we should hold Trump supporters responsible for what Trump says.
(And I'm not sure if saying "all Jews should be gassed" is legal; in Canada it would probably fall under hate speech laws on the basis of inciting violence, but the USA has stronger protection of free speech rights. Per Brandenburg, it may be that "all Jews should be gassed" would be legal but "we should beat up the Jews" wouldn't, in that the latter advocates more imminent violence.)
I'm pretty sure it would be legal in US. Something like "let's go gas some Jews" would be getting into the imminence territory, but the blanket statement of desire is protected speech.
As for your main point; I think that saying that we're holding Trump supporters responsible for what he says is not really correct. We're holding them responsible for what they say by the act of supporting him and/or his platform.
And the platform is always a thing under consideration. Even someone who's supporting Trump because "Hillary is worse" is still making a relative comparison between his platform and hers, which is a political statement - and I don't see why they should get a pass on being responsible for that statement and its implications.
It all becomes a lot less ambiguous if you pick some specific thing instead of discussing it in abstract, and spell out all the consequences in full. Say, I know quite a few people who are going to vote for Trump for the sole reason that they consider him better on gun rights. But when you account for his "law & order" dogwhistling, and unpack it, what they're really saying then is, "my right to own a semi-automatic firearm is more important than this black dude's right to be treated with dignity by the police and the courts". Some people actually find it acceptable even when worded that way, but I'm not one of them.
Of course not. We should absolutely hold Trump responsible for what he says. The disagreement here is whether we should hold Trump supporters responsible for what Trump says.
(And I'm not sure if saying "all Jews should be gassed" is legal; in Canada it would probably fall under hate speech laws on the basis of inciting violence, but the USA has stronger protection of free speech rights. Per Brandenburg, it may be that "all Jews should be gassed" would be legal but "we should beat up the Jews" wouldn't, in that the latter advocates more imminent violence.)