There is a thing in decision theory called the exploration/exploitation tradeoff. Exploitation is always choosing the option that is estimated to be best. Exploration is choosing other options to see if they might be even better. In this case showing new comments is exploration.
Hacker news has a nice solution to this problem. New comments appear at the top of the thread, and then slowly fall unless they get votes. I think this works very well, although I think it's still a bit too far on the exploitation tradeoff.
I've thought about how to solve this problem, and I think the best solution is votes/time since the comment was posted. New comments will immediately appear at the top, since they have infinite vote/time ratio. But they will quickly fall to their correct ranking.
There are other solutions, like treating it like a multi-armed bandit problem. And you can use something like https://en.wikipedia.org/wiki/Thompson_sampling . Create a probability distribution for the upvote/downvote ratio for every comment. Sample from that distribution, and rank the comments accordingly. This should be close to optimal, but it requires people to use the downvote button lots (because it measures comment quality by upvote/downvote ratio, which is what reddit does currently.) Without downvotes, you could use time again, and create a probability distribution for votes/time.
I would add that while it's plausible, it's by no means obvious that Reddit is leaving business value on the table. Imagine that they seek to minimize the bounce-rate on any thread such that they maximize overall time on site for each user. In this case, the important part could be that the top comment is 'not wrong' rather than 'right'. If the first thing users see is too factually incorrect, controversial, or emotionally upsetting, people could be driven people away from the site. Essentially this is the same mechanism that helps cause the Facebook news bubble effect -- people leave if you tell them what they don't want to hear, so you write an algorithm that underweights anything that might do so.
To put it into economic terms, if users have high loss aversion (from being upset, etc.) relative to their thirst for knowledge gained from great comments, you would expect exploitation to be overweighted vs exploration. The business value maximizing solution would then be that once an appropriate top comment is found, the thread should tend to stay in that equilibrium.
I really doubt there is any conscious decision by reddit to do this. For years there was a major bug in their ranking algorithm that hid posts with negative scores. People invented reasons why this was intentional and why reddit might want to do that. But eventually they fixed the bug and admitted it wasn't intentional.
The current comment sorting algorithm was just copied off the internet from some popular HN post. I don't think there was that much thought put into it.
Showing a few new comments closer to the top is not going to hurt anyone. I think if anything the massive improvement in comment quality would make users more loyal to the site. At the moment you can only see the good comments posted by the first 20 people or so, imagine if it was the best comments out of 1,000?
There are vastly more voters than commenters. If only one out of 5 people sees a new comment and votes on it, that would be more than enough to improve the comment quality 3 fold.
"I've thought about how to solve this problem, and I think the best solution is votes/time since the comment was posted. New comments will immediately appear at the top, since they have infinite vote/time ratio. But they will quickly fall to their correct ranking.
"
The traditional solution to this problem is wilson score intervals.
The real problem there is that most sites that use them only use one part of the bound, instead of using both.
Ideally, you use the lower bound to scale how good something is, and the upper bound to determine with what priority you show it order to get more votes on it to determine better bounds to be more confident of your lower bound.
This is what Google Moderator did, and pretty much never had these problems as a result.
(Is it perfect? No. You need a good place/way to show things for voting. But assuming you get that, it works very well in practice).
If you only use the lower bound, you get systems where you only ever become more confident in the scores of the things that show up first (like is happening here).
That said, it's still biased. Years ago, HN admins/mods marked my account as "low-quality content" so now my comments always quickly fall down to the bottom of the parent, no matter how many points they have. They admitted it in an email recently, and refused to take it off my account, despite the quality of my comments no longer being trollish. When you have human intervention like this, there's danger if it heavily biasing the entire forum for the worse.
Well moderators are necessary to remove trolls. If your comments really were so trollish they felt the need to do that, I wonder why they didn't just ban you outright.
Yep, banning would have been an alternative solution. But with the choice they made, it requires more human intervention, both on the shorter term (i.e. policing comments to find trolls and mark their accounts) and the longer term (i.e. policing marked accounts and unmarking them when their comments become consistently high quality), and that's a lot of work, and too subject to flawed bias, which is why it's a flawed choice.
This search for a better way to rank comments still, in my opinion, makes a large incorrect assumption; namely that it is desirable (and possible) to display comments in order from "best" to "worst". If you think of comments only from the prespective a viewer then yes ordering comments from best to worst makes sense, the viewer wants to get as much information as quickly as possible. But if you are looking at establishing a robust meaningful dicussion, part of that is that all comments should have a chance of getting responded to that is proportional to their value, not their rank. Realistically the top three comments in a large thread are about equally good, but the top comment is going the vast majority of views and replies. One system the facilities dicussion might effectivelly shard it's users into separate threads of managble size so they can have a real dicussion, though I'm fuzzy on the details.
Given all the worry we have about about how our online discussions work I think their is real societal (if not extrable monetary) value in discovering a better solution.
Yes-- would be nice if top level comments got collapsed once the thread reaches a certain size. Then you could scan the top level comments quickly and figure out which ones you want to drill into.
Back when I was a regular on LessWrong and it had an active community, it didn't have that problem at all, despite using a variant of the Reddit codebase. Their solution was pretty simple, just show the five most recent comments in the right hand column. People would interact with them, check out their parent comments, and threads would grow organically regardless of age. In fact people would often come back to very old threads and revive them. It might be tricky to scale that to larger communities though.
Yes, this non-linear idea seems very interesting. I have read several discussions on various boards with much hand-wringing about the best way of sorting comments - by time, by votes, by user rep, by some sort of weighted average of the above? Suddenly things get rather complicated as people try to decide how the weights should be balanced. But this idea of showing multiple different feeds side-by-side is very refreshing. Two (or even three?) feeds with different weightings and (and relative sizes!) could provide a better balance than what we are used to.
(As an aside, it would be entertaining to take it to an extreme - seeing the top voted comments also shows you the bottom voted, most controversial also shows least, etc. A sort of anti-echo-chamber measure.)
I think Lesswrong's solution only works because they have a small community, and they have a lot of comment junkies that want to read every new comment. Reddit and HN also have new comment feeds (https://www.reddit.com/r/all/comments/https://news.ycombinator.com/newcomments), but I don't think anyone but bots read them. Granted they could be made more prominent like on Lesswrong.
Thanks for posting that; I'd never seen it before. As my sister comment said, for smaller subs it could be quite useful for finding active discussions - like on /r/truefilm or something.
People always say "sort Reddit threads by comment age if you want the 4chan experience" and that has some merit - but that only applies once you have chosen a thread to open. A subreddit-wide comment feed gives an idea of which threads are active without relying on the blackbox thread ranking algorithm Reddit provides.
I like the simplicity of this solution, but it might be a good idea to combine it with other relatively simple measures that dampen the "first comment effect," such as randomizing which comments show up at the top, as others have suggested in other threads here.
Can you comment on how comments could benefit from NLP related advances? E.g. detecting entities and using those as entry points into discussion, detecting similar topics discussed in different threads, detecting comments which are low effort and don't add much value. I hate the fact that first/top comment sets the tone for discussion overall and lot of people end up expending energy on that topic alone. People end up discussing something tangential just because the first commenter had "interesting" observation. Also I hate the pun threads which plagues reddit.
Makes sense, due to a self-reinforcing dynamic: early comments that are good enough to gather upvotes will stay near the top of the page, gathering more upvotes.
I suspect HN is subject to the same "first comment effect," often to the detriment of later comments that might be more deserving of the top spot.
Even more so when it's a major topic with hundreds of upvotes and comments. Often the top comment of big big big stories on HN tends to be responsible for around 25% of total thread comments, if not more.
Niche stories with less attention tend to have a more rounded discussion with a larger number of parent comments relative to replies.
Can some kind of summary/NLP analysis help to avoid this? Some kind of color coding of the comment threads. Like which comment threads are sticking to single topic. Which are changing/adding new topics to discussion. A visual tool that can help navigate the jungle of different discussions.
Automatic karma deductions for 3rd tier replies?
Detachment + quoting after the 2nd tier?
Positioning based on the cumulative karma of the whole tree?
Automatic collapse of higher tiers (a la reddit...)
Super-powered collapse buttons, for when people get tired mid-tree
I wish something were done, but I bet it will be an addon, not HN, first.
Take the natural logarithm of the comment karma, and then use as the weight in a weighted shuffle algorithm?
Higher-voted comments will then be more likely to move up, but not guaranteed.
(You might want to consider putting a thumb on the scale to distinguish between comments which are low voted because they're uninteresting, and comments which contain flame-bait words etc.)
Maybe some per-user jitter in the ordering so that different people are seeing very slightly different things at the upper end, giving those posts the chance to be seen?
I like the comment collapse idea but i think it would be worth having a sorter so you can see the top comment thread but also newest first and most up-and-coming threads.
Have three categories: New, Best (up:down ratio is high), Controversial (up:down ratio of 1:1).
Pick one from each category and then randomly make them the top three. For larger threads you can salt them into the top 10 comments.
New comments that are truly good have a chance of being picked up to Best or Hot, and Controversial comments spur discussion and moderate the self-reinforcing circle-jerk that threads descend into.
I think that would have a stronger tendency to create an echo chamber.
If anything, I'd suggest the reverse: it should count for more to have more votes from a less tightly coupled portion of the graph. As in, cluster the set of users based on various types of interactions, and include a multiplicative factor in scoring based on the distinctness of the users upvoting something. That way, you can get a modest score by appealing strongly to a small subset of users, but a massive score only by appealing to many distinct sets of users.
In the aggregate, karma is a measure of volume, not of quality. Members with lots of karma are often the ones who spend all of their free time writing comments and submitting links, focusing on quick, easy wins. There's no reason to think they're the ones with the most discerning taste.
Using average karma per comment would be an improvement, but it still favors people who are on the site constantly (so they can be the first to make the obvious comment that everybody upvotes) and frequent the most popular (and lowest effort) subreddits.
Another problem: the last thing reddit needs is more incentive for users to improve their personal karma score. Karma is useful for readers, but that benefit is balanced against the incentives it provides for writers to optimize their karma score above all other considerations. I think HN achieves this balance better than reddit does by deemphasizing personal karma scores in various ways, and this is reflected in the user experience. It's bad enough when karma's just a driver of personal gamification. If it actually provided users with social clout and ability to frame the conversation, well, then it would be something with real world value. You wouldn't need click farms to manufacture viral content, just a few well-placed accounts with high scores. High-karma upvotes as a service. As a side-gig for a careful power user, it would be nigh-undetectable.
Think of the difference between HN and reddit, then multiply it 10-fold. I think that's what you would end up with.
I'd like to have the comments delivered with metadata and have a small JavaScript to sort them. A few defaults (hot, new, etc.) and chance to try popular sort algos. Then chance to weight by popularity of the user, upvotes/downvotes, or location, or whatever is possible with the metadata.
Well, the trend is so smooth that it would be very simple to apply a modifier to comments to counter it. Effectively, votes would slowly expire. In order to remain relevant, a comment would have to be constantly upvoted.
I actually think this would be an interesting experiment to try. Who's hacker friendly and has a well-used comment system that we know?
Yeah, seems good to me, in that it strikes a reasonable balance between keeping good stuff around long enough for you to see it, and keeping it around so long that you've read everything and get bored.
Thinking about this some more, though, I don't think I'd want to see the exact same algo on comments, though; comments don't have the same infinite-firehose problem, and you don't want to encourage spammy necrocommenting on old-but-still-popular threads just to get "new comment" karma.
Maybe something more like making vote fuzzing (which Reddit already does to some extent) inversely proportional to age, and actually sorting on the fuzzed value rather than the real one.
Basically, ant-algorithm. To stay relevant, you have constantly gather new votes. But it has one problem. Comments with higher votes already have momentum, hence they will be shown to new users most of the time, which gives it more chance to gather new votes, thus continuing the momentum.
I don't know if this would change the ranking but I've always wondered if it would be good if threads were ranked by the waited sum of all replies to it. In otherwords the original comment might not have a high vote but if a reply did, even a deep one, the thread might appear at the top. Or does it already work that way?
I am guessing that HN will have a lower percentage than reddit. New comments get a bit of time at the top and have a chance to gain votes if they are worthy.
Sorry, I am not familiar with Reddit, I just took the author's word for it:
Anecdotally, that doesn’t be the case with
Reddit’s modern algorithm; comments made
late in the thread appear at the bottom,
where they likely will not receive any upvotes
In my experience with Hacker News, late comments are given some space right at the very top, even if they are a day late. I thought this was a distinction worth commenting on, but maybe I misunderstood the quoted passage.
So this is kind of bizarre wording on the author's part. He describes that in fact Reddit does use time decay for exactly the reason you mentioned literally two sentences before what you quoted. It seems weird to state something as a fact then dispute it based on an anecdote without addressing the obvious tension between those things...
> Reddit’s ranking algorithm attempts to rectify this by determining comment ranking using both time and community voting; comments in a thread, by default, are ordered based on the points score (upvotes - downvotes) the comment receives, subject to a rank decay based on the age of the comment.
> In theory, this system should allow comments that posted later in the thread’s lifetime to rank much higher temporarily, then Redditors can vote on the new comment; if the new comment is good, it can now rise to the top and therefore the content which would otherwise be buried is now surfaced.
And then it continues on to the section you quoted. I'm not sure what to believe!
I commented about the HN behavior in another thread. (I did not comment about the HN behavior in the article because I have no statistical evidence of HN behavior.)
This is surprising, and it could mean that early commenters have their voice heard more than later commenters. But it could also just mean that topics raised by early commenters end up being discussed in those threads, and commenters on both sides have their voices heard.
For example, if the first comment is roughly "I read the linked article and disagree because of A, B, and C" then someone with an opposing viewpoint would probably reply to the existing first comment instead of creating a separate comment.
This doesn't mean that the second commenter's voice isn't heard—it just means it all happens under the thread created by the first commenter.
If two commenters want to say the same thing, the first arrival will post it, and the second will upvote it (and maybe reply with additional information). So it's not surprising that top comments are weighted towards early arrivals. The question needs to be are there later comments that are objectively better that don't end up sorted higher when sorting by "best".
The "best" sort, by the way, does not use time as a factor, but rather generates a confidence score. Newer comments have less votes and thus less confidence (which ends up giving them a slight boost).
What if you consider it as an optimization problem for the late commenter? You have to choose where to insert your comment, most likely there are multiple options.
A root-level comment when there are already dozens is unlikely to be noticed; there is significant momentum behind the top comments, usually for good reason - whether jokes, clever insight, or a popular viewpoint.
Instead, if you reply near the top in a context where it makes some sense, you skip the root-level graveyard and are almost guaranteed good placement with the downside of "overhead" due to the upwards chain always appearing first.
Consider also that the penalty for being the 2nd, 3rd, or worse comment at root-level gradually increases due to the increasingly nested replies to each highly rated comment. If you assume root-level comments express opinions, minority opinions are also more likely to maintain good position in a reply comment such as the one you posited.
Lastly, "trickle down karma" could be a factor. A quality reply to a root-level comment could enhance the parent comment's value and result in additional karma for them. Such situations could include biases like assuming comments which merited replies are interesting due to peer validation and thus reading them, and more fundamental added value such as uncited sources being given.
There is a "hijacking the top post" meme that pops up occasionally where an important reply is made to the top-ranking comment, since it would get more exposure. (Rare, since it most cases it would be seen as spam if unrelated to the parent)
I do disagree with one point: that there would be no correlation between time posted and upvotes if the reddit algorithm were completely fair. If I post a link to r/tipofmytongue asking "What was that low-budget movie that had engineers inventing a time machine in it?" and the first comment correctly tells me I was thinking of Primer, then it'll be highly upvoted. But later commenters won't get to grab that easy karma since it's already answered. There are only so many "great" things to say in response to most posts. Also, if someone posts in a thread after it has dropped off the front page, they'll get fewer upvotes regardless of what the algorithm does.
The reason is most likely that the top and bottom of the page are easiest to find so the comments at the bottom have a little more visibility than those in the middle.
The HN algorithm changed very recently, which was one of the reasons I had the idea to take a closer look at the Reddit data. HN, within the past few months, will highlight new comments within the first 2 slots even on busy threads.
I would perform the same analysis on HN data to confirm if I could. (HN does not expose comment scores)
Well, this is one reason to consider normalizing up/down votes by views. Is there any data on the number of views each comment received (number of times each was scrolled into view, for example)?
You'd also have to factor in the likelihood of each person up/down voting (if it's viewed by many people who weren't going to vote anyway, then no votes are nothing out of the ordinary).
The same thing happens with search engines as well, where items at the top of the search results tend to stay at the top because they're purchased or clicked on at a higher rate. Can anyone chime in with what they've done to fix this in a search context? Has anyone had success with a one-armed bandit or randomization approach?
Great read. I think the title "On Reddit, the earlier you comment, the louder your voice." or something to that tune would've made for a more impactful headline!
I think this analysis misses some subtlety of smaller subreddits by focusing on the huge mass subreddits. As such I don't know if the results are at all applicable for my experience when for example >80% of the threads currently on my frontpage have <30 comments total (and as such would be cut off from the analysis). And I don't think I'm such a total outlier here, the long tail of reddit is really long.
That's the problem with threaded views, as opposed to the flat-view style of other forums. Granted with the latter system, the first few comments on a new page will always get more exposure, too.
This is not a surprise - it is I guess inevitable and intuitively understood - I mean who bothers commenting on a thread here that has 500 earlier comments? There is zero chance of anyone finding reading or replying
I like the idea of less wrong using the right hand columns
Vote inflation might serve as an interesting dynamic for allowing other's voices to be heard. While a new vote will always carry the same value, acquired votes may suffer from inflationary decay, providing a rotation on viewed comments. You could probably modify this inflation based on thread velocity.
Hacker news has a nice solution to this problem. New comments appear at the top of the thread, and then slowly fall unless they get votes. I think this works very well, although I think it's still a bit too far on the exploitation tradeoff.
I've thought about how to solve this problem, and I think the best solution is votes/time since the comment was posted. New comments will immediately appear at the top, since they have infinite vote/time ratio. But they will quickly fall to their correct ranking.
There are other solutions, like treating it like a multi-armed bandit problem. And you can use something like https://en.wikipedia.org/wiki/Thompson_sampling . Create a probability distribution for the upvote/downvote ratio for every comment. Sample from that distribution, and rank the comments accordingly. This should be close to optimal, but it requires people to use the downvote button lots (because it measures comment quality by upvote/downvote ratio, which is what reddit does currently.) Without downvotes, you could use time again, and create a probability distribution for votes/time.