Public shaming is essential. Anyway it still happens, even more brutally, in the comments.
What they should have done is add more detail on the nature of the downvotes, like the Steam store does for negative reviews. That is, have graphs of positive and negative ratings over time to make any downvote brigading obvious. Maybe have a way to exclude "less-verified" votes, or allow the viewer to look at only e.g. YouTube Premium votes (which are more likely to be real people given the cost). And so on...
Perhaps they could do as on Stack Overflow, and let a downvote cost one point from your accrued points? Or perhaps that's too much of a re-design. It could regulate spam downvotes though. (I think the biggest problem with SO is that your point pool is visible, with gold and silver and so on, which leads to anyone with a high enough sum being treated differently, although that might actually not reflect competence or expertise.)
What they should have done is add more detail on the nature of the downvotes, like the Steam store does for negative reviews. That is, have graphs of positive and negative ratings over time to make any downvote brigading obvious. Maybe have a way to exclude "less-verified" votes, or allow the viewer to look at only e.g. YouTube Premium votes (which are more likely to be real people given the cost). And so on...