Nice work, very informative. This is an issue that I feel has the potential to kill HN in the long run. It is steadily growing closer and closer to being impossible to hit the front page without gaming the system.
The thing is that these submissions are coming in because there are more people, so in theory there should be a way to balance the increased submission rate with an increased voting rate as well.
The issue isn't the amount of time a post is on the new page for, it's how many people get to see it before it falls off the page.
So long as the number of people viewing the new page at any one time increases proportionally to the number of new submissions, a reduced time on the page won't be a problem.
It's exposure to potential upvotes that's important, not total exposure time.
That said, increased competition for the limited range of spots on the front page may eventually be a 'problem', but only if you think having too much excellent material as a problem and not an opportunity ;)
Reddit puts a random new link at the top of the page. To get votes, it has to be visible, to be visible it has to be on the front page. It's just a question of layout and maybe simple filtering based on karma or account age or whatever.
Could just intersperse 5% of the front page with random new links, or take a cue from reddit and put one at the top, or divide the page into two panes, or alternate lines of new and top links, in different colors (one darker, one lighter), or on every n-th login/visit, take the user to the new page instead of the front page.
The biggest problem with setting a minimum, particularly one that high is that it is extremely difficult to get a lot of karma from comments alone, especially for new users without name recognition.
That's the point: raise the bar by forcing the new users to hang around and get a feel for the culture before submitting. On top of that, acquiring 200 karma points on HN definitely doesn't fall into the "extremely difficult" category.
200 karma is not an onerous burden. I've been reading HN for years, but only started posting comments regularly in the past couple of months (maybe two or three posts per week).
Right now, I'm over 100, and I'm not that insightful. If I had something to say and paid more attention to posting in threads that are likely to be popular, I'm sure I'd have much more karma.
Even requiring 3 or 4 karma would wipe out nearly all the deads (which I have turned on, sometimes good front page stuff gets deaded..) and 50 karma would get the page down to about half.
The issue with that is when people post ASk HN stuff anonymously, which I think is valuable. Maybe figure out a way to create anonymous posts that don't have the limitation?
Agreed, I wondered about that. I was going to spend time doing that, but when I ran it past two friends they both said that by the time the understood the text, they didn't need the labels, and not having the labels made them spend time trying to understand the text.
I guess it's a "Read like Math" issue. You need to read carefully to understand, or skim and get the gist. And I've moved on to the next stage now. I may come back to it if more people feel it's important.
The chart for the weekend specifically has just a few red points, which has shifted the range. Removing them would make the range better, but that's not really worth the time.
What do you mean by this? Are you normalizing before coloring? If so, don't. You said the colors were for <100, 100-200, 200< and I'm holding you to it.
As for the interpolation, I think you're sampling at least hourly, so why didn't you just store all times at each scrape and match the >60min ones with those?
It is normalised, but the range over which it's normalising is pretty near constant, so you don't need to worry. What I wanted to say was that without the isloated red points we might get to see more structure in the data, becuase we could expand the range. I may yet do that, and make careful notes in the text and annotations to explain it. Then again, I probably won't get time.
Regarding the extrapolation, I hadn't thought of what you suggest, and it's a good idea. It would be non-trivial to implement quickly - I'll see. Thanks.
Start from http://news.ycombinator.com/newest go to the bottom of the page and click on "More" until one of the entries says "1 day ago", and there you are. As I write this there have been 282 items in the past 24 hours.
That's on a Sunday, which as the charts show, is a slow day.
Also just checked - the gap from "3 days ago" to "4 days ago" is 604 items. That's 604 items in a 24 hour period during the week.
Pretty much means unless something is instantly interesting to people it will fade into obscurity quickly. It's a tough balance, the main thing that gets missed a fair bit is ask hn posts where only a small amount of the community would be able to chime in on, if those people aren't around at the time they will never see it.
The thing is that these submissions are coming in because there are more people, so in theory there should be a way to balance the increased submission rate with an increased voting rate as well.
But how?