The trouble is, you can't -- only the commenter can see their score swings, and then only if they pay attention.
Publishing upvotes and downvotes would allow others to see these effects, and would allow them to register a comment as "controversial" rather than "not worthy of notice".
There are two things I've always wanted out of a comment system that allows up and down votes:
1. I want to know who upvoted/downvoted/flagged a comment so I can identify attempts at censorship, revenge voting/flagging, etc.
2. I want the ability to exclude individuals from the upvote/downvote tally I see.
For example, if I see someone downvoting every post that is favorable to Elmer Fudd, or upvoting every post that is favorable to Bugs Bunny, I want to block that person's emotional bias from the rankings I see in order for my view of the comment section to be as objective as possible.
Well yeah, there are a lot of those "controversial" subjects that attract far-left far-right flamewars and this has normalized using downvotes/flags as a weapon. Even about innocuous things. Which is... weird and sad.