Hacker News new | past | comments | ask | show | jobs | submit login
Reddit was down for over two hours (statuspage.io)
232 points by bluedino on Sept 18, 2019 | hide | past | favorite | 268 comments



Y'know, the toll that all this activity takes on Reddit's servers gave me an idea for a sort of distributed Reddit. News links and stories assorted into topic-based groups, with the option for users to comment or respond to others' comments in a threaded discussion. It would be completely decentralized and belong to the users, not some central media company. We could call it Users' Network, or USENET for short.

We could implement it with blockchain.


Sign me up. I’d happily contribute to that as much as I can, whenever my parents aren’t using the phone line.


Lemmy is a federated alternative to Reddit: https://github.com/dessalines/lemmy

Edit: Check out the dev instance of Lemmy! https://dev.lemmy.ml/


Wow, this is quite possibly exactly what I've been looking for! I love Rust (have experience with Actix as well), React (so hopefully Inferno will come naturally), and have been looking for a Reddit alternative for some time!

I'm not sure if I should thank you or curse you for making me want to spend a ton of time on yet another side project ;)


Lemmy is a fun project that I'm sure you could make even better!


Developing a new fediverse is excessive, but it is unlikely a new sorting algorithm will be added to mastodon soon.


Who cares about Mastodon? Lemmy is ActivityPub compliant, just like PeerTube, write.as, Pleroma, rustodon, PixelFed and the other wonderful AP projects out there.

These platforms do not fracture the Fediverse FYI, you can follow cross-platform & cross-site like normal.


Don't forget to turn off call waiting. I always forgot to turn off call waiting....


I'm showing my youth here, but this is a legitimate question. Isn't call waiting just a thing that exists on the line? Did phones used to have a switch to toggle that? And what was the harm in keeping it on? When I had dial up, I don't recall having to consider call waiting.


From my recollection, there were 3 or 4 different * codes. Separate codes to turn it on and off, and also to turn it on or off permanently or just for the duration of the next call.

I think at one point it was a value-added service, and you had to pay extra on your phone bill to enable it, but once it was enabled you could turn it on or off at will.

When I was a teenager (in the 1990s), our house had three phone lines. One was for voice calls, and it had call waiting turned on permanently. Another was for modems (first BBSes, then dial-up Internet), and had call waiting turned off permanently. A third was paid for by my Dad's employer, he used it for work calls, work faxes, and call up his work's modem bank to send/receive email, I don't know what he did with call waiting settings.

My Dad ordered three phone lines, but only had two pairs running in to the house. So they ran another cable, which I think had 12 pairs in it, but only three were ever connected. My Dad was smiling about how future-proof they made it. Nowadays, only one of those pairs is still active, and I expect within the next 1-2 years that pair will be turned off too. Analog POTS phone system being phased out in favour of VoIP (the Internet comes into the house over coax, not copper pair.)


Dialing *67, I think, in the dial string for dialup turned off call waiting for the duration of the call. Otherwise a second call would try to ring through and disrupt your Super Z-modem download.


The incoming call tone would disrupt modem traffic so you had to dial a code prior to the number you wanted to dial to disable this temporarily for the duration of your outgoing call.


Call Waiting used to be a feature that you paid monthly for.

As was Caller ID, Three Way Calling, Busy Number Redial, etc.

It's amazing how much incumbent telcos nickled and dimed little features like this. I think it was a couple bucks a month to have that feature.


Wait, you didn't add the disable call waiting code to your dial string?


I've started working on a client for this decentralized reddit. I'm giving it the code name "rn".


I've cloned your inferior version and markedly improved it. And since I like not reading reddit, I decided to name it "Nothing New" or "nn"


Except UX is everything. 100s of billions in corporate value are based on nothing more than better UX.

If only everyone who talks about usenet, IRC, and the rest could actually be bothered to update the experience to what modern users demand.


You know, the people who talk about USENET and IRC are people who don't want to be "updated" to what modern users demand. I don't know if you've had a look around you lately, but the modern internet is a dumpster fire.

Anyways, ideally you would be able to connect to services with whichever client you want. You could use a pretty eye candy client, or then not. With the web, however, it's an impossible feat to implement a standards compliant browser.


> the people who talk about USENET and IRC are people who don't want to be "updated" to what modern users demand.

IRC is objectively terrible. If you ask a question, the only way to get the answer is if you stay logged in until someone answers you. That's like buying a phone that doesn't ring unless you're already holding it up to your ear, or an email client that deletes all your email unless you happen to be looking at your inbox at the moment a new message arrives.


I disagree with your false equivalency. IRC is akin to a cellphone without voicemail, as when your disconnected from the network (cellular or IRC) you just don't get communications.

Scrollback and voicemail can be useful, but a good chunk of the population does not want either, hence unread backscroll and unconfigured/full voicemail boxes being very common.


>but a good chunk of the population does not want either

The largest Freenode channels I'm on like #node.js and #javascript have about 10 regulars max, myself included. It's pretty much dead for any purpose other than hanging out with a handful of curmudgeons feeding off a corpse out of habit.

Doesn't seem like "a good chunk" to me. IRC lacks the only feature I really care about in a chat network: new people to talk to, users.

I wonder how many HNers who praise IRC only like the idea of it but don't even use it these days, either.

And I haven't even heard HNers bring up usenet outside of downloading free shit so I know that's especially dead wrt this thread as a decentralized Reddit.

So it always makes me chuckle to see "yeah, well, some people like it like that!" which is apparently virtually nobody. Or it's like how my two friends and I loved the failed, abandoned supermall in my hometown because the three of us could longboard down its parking garage.


"I have never experienced this thing, therefore this thing must not exist"


"I like this thing, therefore a sizeable chunk of the population must too"


I agree.

What makes IRC good is that it's a text-only chat with a wide range of highly configurable, blazing fast clients. Not supporting images, quoting or reactions could be considered a feature, since these distract from the actual conversation at hand (consider newbies clogging the conversation with screenshots, or 100 people adding reaction emojis to a message).


Young people now think it’s ok to send out a message and respond whenever they feel like it. Communication is no longer a conversation in the tradition sense, it’s a series of monologues or statements where the sender usually only cares about whether you are impressed by the statement.

IRC was all about conversation. You went there to have a conversation, like a party line voice bridge. You even waited around for someone to have a conversation with... oh I miss it...


> Young people now think it’s ok to send out a message and respond whenever they feel like it.

Boomers demand attention constantly.


No. you just connect to an always on session like TheLounge and you have an IRC with the features you want. Pretty simple.


This is like the famous comment in the "Show HN: Dropbox" thread that said you could just use rsync/ftp/git instead. Meanwhile Dropbox is now worth over $8B.


Not sure why the value of anything makes my point invalid? I answered there was a simple way for anyone using IRC to keep sessions alive and therefore not miss any message. This is an off the shelf solution, not something you have to build by yourself.


Because it's a workaround, and shows just how far apart the legacy tech is from a modern experience.


Synchronous and asynchronous communication both have their fundamental strengths and weaknesses. In a chat room you have the possibility of a real-time back and forth. That's invaluable at times.

You might have a preference for one over another, or each might be appropriate for different situations. But to claim one is objectively terrible is ridiculous and only makes you look like you haven't taken the time to understand it at its most basic level, or you're too solipsistic to be able to recognize the existence of value in something outside of your own preference.


Slack is a superset of IRC, except for the protocol not being open.

The UX in slack is not easily replicated in IRC (for example, how does/can global search work in IRC?)

Therefore, it is actually correct to say IRC is objectively worse in those measures. The only measure that IRC beats slack is the open protocol (which, if I'm being honest, not many users care,ala most existing chat services moved to private protocols and haven't lost all their users to IRC or xmpp).


> IRC is objectively terrible. If you ask a question, the only way to get the answer is if you stay logged in until someone answers you. That's like buying a phone that doesn't ring unless you're already holding it up to your ear, or an email client that deletes all your email unless you happen to be looking at your inbox at the moment a new message arrives.

Well it's a good thing Slack and Discord don't work exactly the same way but with a nicer UI and no need for a bouncer to emulate a persistent session.


> objectively terrible

These two words next to each other really bother me. That something is "terrible" is not objective, whatever that means. All we have are our subjective valuations.


What I meant was more like, when viewed impartially rather than through the lens of nostalgia. Two people can do that and still have different opinions.


Your use case is what mailing lists or usenet are for, of course IRC is an objectively terrible mailing list, it's also an objectively terrible toaster.


Usenet turned into a dumpster fire even more quickly than the Web did. It was completely overrun with spam and binaries by the late '90s.


However, the spam mostly died off now. It's possible to have great convos with boomers complaining about each other finally. A small number of people came back because of the pseudoanonymous qualities. Honestly the only thing that really sucks is Google Groups users bumping 15 year old threads.

Oh and binaries didn't go away, that's a feature and you just don't have to sync alt.bin.* if you want to avoid. I think it's great though for snatching rare/out of print stuff.


The spam died off because Usenet died off. If users ever went back to Usenet, the spam would come back again behind them. There's nothing in the design of Usenet that can stop spammers from spamming.

(There wasn't anything back in the '90s in the design of email to stop spammers from spamming, either; but enough people cared enough about email that billions were spent to add that spam-proofing. Nobody cared enough about Usenet to invest a similar amount, so it's still as unprotected as it was when Monica Lewinsky was in the news.)


Modern internet is both good and bad. Depends on the site. Generalizations don't work. However if those technologies are to ever be popular than they'll need to be redesigned to keep pace with what the vast majority expect, which is in response to the OPs comment. Otherwise you can already go post on usenet right now.

> "Anyways, ideally you would be able to connect to services with whichever client you want."

Make it happen. This is my other point. People complain that all these fantastic legacy technologies aren't used but nobody puts in the work to modernize and use them. If it was that simple, where are all the companies doing it?


Is it companies, that would or should do it? Colleges and other publicly-funded entities are who did it the first time. Companies can only do what's profitable. (Or what they can convince investors will someday be profitable, but we can ignore that aberration in the long term.)


I didn't mean it that deeply, just that if people who believe these older technologies can drive modern incarnations then I would expect them to have produced products by now.


Citing UX in a thread about Reddit, the company with an almost universally hated redesign that moved away from a mostly text-based condensed format is... interesting.


Every user engagement metric has increased since the redesign. Unfortunate for some of us, and it doesn't preclude the metrics measuring the wrong thing, but that's reality on the ground.

Same reason why Gmail is a bloated mess, because that's what the metrics show is wanted by users.


> Every user engagement metric has increased since the redesign.

Doesn't that just mean that users are spending more time on the site because it's so much more of a pain to get what they actually want out of it? Like, you could increase "user engagement" of a shopping mall by turning it into a labyrinth and rearranging the shops every day.


Alas, Costco does just that (rearranges where things are) to improve "discoverability." Drives me nuts!


Not necessarily for social media where there’s an endless river of entertainment content. I doubt most people go to Reddit for any specific purpose or task they’re trying to accomplish as quickly as possible.


Reddit is a corporation with a large amount of VC investment. Profit is a primary goal, and engagement that shows more time and thus more revenue is exactly what they want.


It would have been more honest of them to come out and just admit “yea, this new design is terrible but it juices our metrics and makes us more money so deal with it if you want Reddit to survive.”

Also, Protip: in case anyone still doesn’t know, go to old.reddit.com for the previous, better interface. Who knows how long it will be kept alive though.


> Also, Protip: in case anyone still doesn’t know, go to old.reddit.com for the previous, better interface. Who knows how long it will be kept alive though.

I'm using an extension to automatically redirect to the old one. When the old site is gone, I will probably leave Reddit, the redesign is too unusable for me.


Yep.

I like HN but I miss the diversity of conversations. /r/synthesizers /r/starwarssleaks etc.


The problem with that is assuming the new design is terrible for everyone. It very well could be that the majority do like it, and it's a vocal minority that complain.


The redesign is objectively terrible, adding megabytes of bloat to each page load for the sweet sweet ad revenue.


The whole point of the redesign is to remove full page loads. Why are you downloading the entire payload every time?


The whole point of the redesign is to juice user engagement metrics and get that sweet, sweet ad revenue.

You don't need megabytes upon megabytes of nonsense just to avoid some full page loads. Or take al ook at something like HN where it's light enough that a full page load isn't a problem.


No, you can increase user engagement of a shopping mall by turning it into a Mobius strip.

I don’t think most people know how to handle anything but an infinite scrolling page.


They could have accomplished that without the redesign. That's what reddit enhancement suite has done for ages.


Ever been to Ikea?


Ikea is a pretty much perfect shopping experience. It's the opposite of a labyrinth. There is exactly one path through the shop for people who want to just see everything and there are clearly marked shortcuts if you only want to see one area.


Where did you find these numbers? From what I've seen Reddit never released any sort of metrics other than saying one specific Audi ad campaign did like 10x better or something. And Alexa showed lower traffic, higher bounce rate, fewer pageviews per visitor after the redesign[1].

[1] https://i.imgur.com/6dMCwoJ.png


I don't disagree, but "engagement" does not necessarily reflect value to the user. Taking more time to accomplish one's goal (e.g., obtain information) than necessary may improve engagement metrics from the service's perspective, but from the user's perspective, that's extra time that might have been wasted and better spent on other things.


Maybe not value to each individual user, but in aggregate the numbers of users and content are increasing. Collective value and network effects are higher. This is also leading to more revenue which is critical for a corporation.

I'm sure they lose some users but they've gained far more than they've lost so far.


When you aim for the lowest common denominator, you lose the users at the other end of the Bell curve.


I wonder what will happen if they ever make old.reddit.com stop working?

I've been on the new design, trying to give it a chance, for over a year trying to give it a chance, and I still have to often switch to old.reddit.com to do some things.

1. The new interface still doesn't have the "other discussions" link to show you discussions of the same link in other subreddits. I think they said something like a year ago that they realized this was important and it was coming soon.

2. The new interface messes up scrolling on Edge on my Surface Pro 4 when viewing a discussion. Actually, for this one you don't have to switch to old.reddit to work around. All you have to do is hit refresh. When you refresh while on a discussion, that switches from the view that shows the discussion in an overlay on top of the thread list, to a more traditional page that just has the discussion. That page doesn't fuck with scrolling. Depending on how much I plan on reading and how long the discussions are, I may or may not get annoyed enough with this to just give up and switch to old.reddit.

(BTW, that discussion as overlay view also fucks up in-page search, because the thread list is still on the page, just hidden from your view--but not the browser's search functions--behind the discussion view, and so the search hits those hidden items).

The only thing I can find in the new interface that is actually functionally better than the old interface is the fancy pants editor. Aside from selecting tools/commands not always always working if it has to put them on the "..." pop-up, and it sometimes losing the ability to scroll the text being edited via the mouse, the new editor is a win.

If they would just put the new editor as an option on old Reddit, I think a lot of people would be happy.


user engagement increases with frustration.

the A/B test isn't telling you that users like something, only that its taking them longer and they didn't bounce.


personally I've probably spent an hour on Reddit in the last year. I used to spend a couple hours a day. After it modernized I realized it was just facebook for introverts and bailed.


Depends on how you define engagement. When you set all the videos to autoplay with sound of course users are going to be more engaged. They're either scrambling to figure out where that noise is coming from or "watching" the videos.


The customers (ad buyers) come first, and I'd suspect the redesign improved those metrics


I was going to say something just like this... but I think that the entire history of reddit UX has been terrible


I guess that is only the case for the HN bubble and a loud minority. Also, people are always complaining about UI changes. Personally, I prefer the re-design, even though it has a few flaws. The old design was awful.


And yet Reddit is regularly down, and apparently today for hours on end. What's the UX of a service that isn't working? Do users not demand that a service be up?

There have been some really good modern Usenet clients. The biggest issue I see is that current browsers don't have built-in support for Usenet any more, and that's not something I can influence in any way.


> Except UX is everything.

I see you haven't visited Reddit lately ;)


What do "modern users" demand that the various current GUI Usenet clients are missing?


Relevant content and other interesting people.

Look, I loved Usenet back in 1994, it was one of my first great Internet obsessions. But it effectively died in terms of relevance before the turn of the century. The majority simply preferred web forums.

Usenet was always a pain to use, posts would routinely get mangled or dropped entirely, and the multiple user-driven quoting conventions and wonky clients made any long-lived threaded discussion impossible.


Yes, but I think it is the community of reddit moderators that make reddit what it is today...


It’s too bad Google doesn’t take on this project.


I've specced out a decentralized reddit alternative a little bit, but have too many side projects. Someone please take this and build it. Let me know if you try, would love to spectate and advise on development.

The key is there shouldn't be a globally consistent front page. Sorting should be done on an individual basis. Upvotes boost signal signal to peers and downvotes squelch. By propagating content scores transitively through the network proportionally to trust scores, users can moderate their own feeds by voting and managing their friend list.

Users have a peer list, containing a list of server/users on it. Each peer has a user-managed 'trust weighting'. Each user has a list of "good content" (ideally hash identified for content addressability), with each item having a score based on that user's votes and votes from peers, weighted by that users trust in that peer.

Periodically, your server contacts all of your peers, and asks them for their good content list. The scores from peers are multiplied by your own trust weight for that peer, and you build a personal "good content" list that merges the lists from each of your peers together (and drops insufficient scores).

You are presented with a score-descending-sorted page of content. Whenever you upvote something, it increases your score weight for that content as well as the trust weight for each peer who sent you the recommendation, and vice versa for downvotes. Votes are transmitted to peers as a crypto signature of the content hash, but when retransmitted to peer-of-peer, they only see the intermediary's aggregated and trust weighted merged scores.

The specifics of the algorithms on how you calculate and adjust weights can be configurable by each individual user, the protocol only cares that peers are able to produce some kind of score list.

Dividing content into topics is a bit trickier, could just label content with tags. I think it may be preferable for each user to have multiple topic focused 'personalities' that are basically distinct user accounts with unique peer lists and votes. In that way, I could follow Dave-gardening without having to follow Dave-sports.

For this example I'm using 1 user per server for simplicity, though not required. All users could be on same server, which is probably best for MVP to avoid implementing p2p networking stuff until validated.


I would much rather have the feed based on trust weighting of experts or people smarter than me, than the feed trying to feed me what it thinks pleases me. Metacritic vs Netflix. Aldaily vs facebook.

You dont need a consistent front page for everyone, but communities of like people should get similar front pages based on what is "best" and best shouldn't be defined as universally popular, but instead popular with some subset of experts.


Well what cwkoss describes would work well for you too: you would choose your experts of choice and have them as trusted users. I would too expect some users to become trust nexuses like that.


Yep! Ideally there would be some getTrustedPeers endpoint that would allow you to transitively discover interesting users 2 hops away. Add them and they become 1-hop and could get their trust list, rinse and repeat to traverse the trust network.

Because users are 'liable' for content they rebroadcast (if you repeatedly propogate downvoted content, your trust score will fall), there is a strong incentive to "moderate" the content you promote. This allows not only the OP to gain karma/trust from a post, but also adjacent peers can earn trust by reliably upcoming content that is later upvoted downstream. Hopefully this would encourage "tastemaker" users who don't create new stuff, but carefully curate what they rebroadcast. Trusting the right users allows you to rebroadcast their trusted content automatically, if you chose to configure your trust algo that way.


I kind of disagree. Thats why rss failed. It requires the user to set it up before it works. And I may never leave my comfort zone, if setting up what to follow is my choice. I dont know what I dont know. If I dont know the experts to follow, guiding me to choose experts seems silly. Just do it for me. Let the experts pick the other experts.

There is a huge benefit to everybody in a community seeing the same page. Whether its hckrnews, drudge report, front page of wsj, if everybody sees the same page, there is a social water cooler effect of "hey did you see" "yeah."

I just think with the "infinitely customized to you" we are trapping people in a constant cycle of always consuming more of the same, while also fragmenting our sense of community and identity. We get stuck in the filter bubbles we choose. These two articles combined together make me feel trapped.

https://www.vox.com/2014/5/20/5730762/buzzfeeds-founder-used...

https://www.bbc.co.uk/blogs/adamcurtis/entries/78691781-c9b7...


Ex. Alice follows Bob with weight of 0.5, Dave with 0.1

Bob scores content A as 0.8, B as 0.2

Dave scored content A as 0.4, B as 0.9

Alice downloads both lists.

Alice score content A as avg(0.8 * 0.5, 0.4 * 0.1) = 0.22

Alice scores content B as avg(0.2 * 0.5, 0.9 * 0.1) = 0.095

Content A gets sorted higher than B.


do you guys use reddit's frontpage? I never. I hang out in a few subs i check very often and try to be part of the communities. frontpage is just so random


Yes but I manage my subreddit subscriptions


This was called Advogato, and while it was interesting it ultimately died: https://en.wikipedia.org/wiki/Advogato


Congrats! You have invented the Instagram feed.


Does instagram have some sort of internal 'trust score' type thing between users that used to weight the feed? Neat.


Sounds like Aether[1]. In contrast to the Usenet, it is peer-to-peer, instead of federated. It does also have moderation, voting, and a pretty UI – things that are arguably missing in the Usenet. (And it doesn't use blockchain.)

[1]: https://getaether.net/


So, who does the moderation, new group creation, and antispam? Those were always problems.


Exactly. Will this new platform provide a safe space for the brilliant minds at /r/fatpeoplehate? /s

The problem with all anti-censorship platforms is that they become hate platforms by default, because only people who get deplatformed have an incentive to move there, and it’s mostly hateful loudmouths who get deplatformed.


Personally, I think the solution is just to give users options to censor others from their perspective instead of just trying to create some utopia that is free from hatred of any kind - which is impossible.

Personally, I prefer having these disgusting people out in the open. When they do something horrendous, the paper trail can be long and damning - making it much easier to convict them. These people are usually proud of their deviancy and advertise it without shame.

The notion that if we just ban all the sick people they will just go away and never do anything evil is ludicrous. The evil is already in their hearts, hiding it won't heal it.


I think self managed censoring is equally impossible.

It is twitter’s model, and it puts an incredible burden on the most targeted or vulnerable users.

Or to get back to reddit, the model works for people who stay focused on a very limited set of subs (default filters are horrible); I think that would be close to your idea of personally tailored censoring, but it goes too far in the other direction IMO, unexpected information becomes very hard to reach people.


Twitter is just terribly done and does not really present a system, that tries to provide good tools for self-censoring.

There are no way to downvote people or provide a score that can be used (even in a very optional way) by people who want to filter out the noise.


Hearts and minds are not fixed. People can be recruited and radicalised. For every one person who'd do this stuff on their own there's a hundred who do it because they think it's popular.

> the paper trail can be long and damning - making it much easier to convict them

This usually comes out after they've carried out the mass shooting.


Unfortunately, you can't arrest someone that hasn't committed a crime.


I think self managed censoring is equally impossible.

It is twitter’s model, and it puts an incredible burden on the most targeted or vulnerable users.

Or to get back to reddit, the model works for people who stay focused on a very limited set of subs; I think that would be close to your idea of personally tailored censoring, but it goes too far in the other direction IMO, unexpected information becomes very hard to reach people.


This creates echo chambers like 4chan, 8chan, pleroma.site, gab.com & /r/The_Donald where likeminded people gather and decide bashing in the heads of LGBTQ+ people like myself is perfectly acceptable, and proceed to act out their hate in my community (Ballard, WA).

My fellow trans and gay friends have lived in Seattle without issue for decades, but now Nazis are coming here from rural Washington to attempt to murder us. This is a huge change from the past, and SPD's hate crime statistics are up significantly due to this.


chans are completely open, uncenssored forums - from either side - not even remotely similar to what I'm advocating for.

I have never once seen on The_Donald, even before their quarantine, any highly voted posts or comments advocating violence. The moderators regularly release transparency reports that show exactly all the things that they remove and their have been plenty of instances of people creating alts to post inflammatory things to get them in trouble. The vast majority of users condemn violence of any kind other than lawful self-defense.

I also see plenty of openly LGBT people on there, and they are just as welcome in the forum as anyone else.


A thousand times this.


>The problem with all anti-censorship platforms is that they become hate platforms by default

Popper's Paradox made reality. I still remember Voat blowing up because their lawyers started making them remove comments inciting violence.


"'plonk' is the sound of you hitting my kill file" is what we used to say. It worked pretty well at the time. I've yet to see a moderation system that works better.


We'll crowd-source that.


I made Aether for that, pretty much exactly what you're talking about. https://getaether.net

Been on Hacker News a few times too:

https://news.ycombinator.com/item?id=19161171

https://news.ycombinator.com/item?id=18373504

https://news.ycombinator.com/item?id=18370208

It's the need to make money that makes things go awry, so Aether doesn't need to make money. No servers mean costs are fairly negligible, except my own time spent.


Heh.

That being said, I've always thought that the right way to implement a distributed social network would be not with blockchain, but with IPFS, or something similar to IPFS. Pinning someone else's IPFS hash would be the equivalent to an upvote for that item.


Do it in Rust, so it will never crash.


I know you are being sarcastic, but the startup I work at is building something like this: https://beta.trustory.io/.

Here is a bit about the technical architecture: https://link.medium.com/yjevZ7y15Z.

Used to be a heavy Usenet user back in the day.


>We could implement it with blockchain.

Mining would have no monetary incentivization, so the people left controlling the network are the ones who are incentivized to push a message on your new distributed reddit.

Like with all things that have a vacuum of power, it will be filled by someone - most likely not who you really want to fill it either.


Voting is a killer feature. It is an effective spam fighter and makes farm-troll's jobs much harder.


Where is the new "alt."?


While the joke is not lost, think of the pictures. Of animated gifs and video clips. They are a large part if Reddit.

Sending every peer a copy can be wasteful and slow. It only there'd be a cluster of servers to make these media available via http...


For a long time reddit did not host any media content. It was just a link aggregator.


Could your system break the back of r/sanfrancisco? It has turned into tourist pics of the GGB, sunrises and double rainbows at the hands of moderators that want no dissent


Such a SPAM system will SPAM never have a SPAM problem with spam


I lazily made my own little private server a couple days ago implementing this concept by stitching a usenet server to a web login/account portal.


Exactly this already exists, but no blockchain is needed:

https://notabug.io/ (beware of content: anonymous decentralized reddit!)

@go1dfish runs NAB and all the code is at https://github.com/notabugio/notabug .

It is built with my P2P graph protocol ( https://github.com/amark/gun ), + he has some custom implementations of it.


The post was a joke mostly, especially the blockchain part


Mastadon. I was a Usenet user as well but it isn't user friendly to normies.


> an idea for a sort of distributed Reddit

Free name suggestion: Distreddit


How would upvoting work on USENET?


You mean Steemit?


Yeah this already exists and was done pretty well actually.


Except for the part of how the Steem token works, which basically means that built-in unmarked ads are needed for the economy to work.


Yeah but that's kind of nitpicking. If you're looking for a "decentralized reddit" on blockchain, it's pretty hard to argue Steemit isn't this.

There's others too. People are free to fork it and make a new platform. Most of the hard work is already done unless you want to change absolutely everything about it.


I noticed this outage coincided with elevated timeouts in one of our EC2 us-east-1 availability zones.

We had recovery by removing that zone from our Auto Scaling Groups, but I haven't been able to find many others affected online (other than Reddit).

Did anyone else experience something like this?

edit: Our errors started at 13:35 Pacific Time.


No problems from Australia in the last 2 hours (including just now.)

For example https://www.reddit.com/r/menwritingwomen/new/ shows a post from just now (verified the timestamp).


Also in Australia, but it has been both up and down for me in the past 2 hours.


It's 'up' for me but, I'm regularly having problems opening comments and posts.


i miss the fail whale.

IMO, their javascript UX is awful... impossible to see all the comments in a thread! what sort of nonsense is that?

old.reddit.com ftw



Whoa, that is orders of magnitude faster than the default mobile site. Thank you for sharing, I didn't know about it


Wow it's so fast an the image links are links to actual images, not the post page with an image embedded on it.


Wow, so quick. Thank you.


Odd example


It's just a white page for me in Japan.


They were doing another test run of their new live-streaming service "RPAN" today, I wouldn't be surprised if that's connected to the downtime.

If you can get the page to load: https://old.reddit.com/r/pan/comments/d5paue/a_midweek_treat...


I wonder what Reddit devops is like, I find it so strange that such a trafficked website can’t handle rollouts without what appears (on the outside) to be a full reboot. And also strange that the rollout takes so long. Makes me think they are doing something naive like running on naked vms and having one guy running a script to take them down, update os, redeploy new binary, etc. all at once. Then probably encountering issues and live debugging them until it works again




I believe it was on r/worldnews . They just tweeted it

Its still showing up on reddit.

https://old.reddit.com/r/worldnews/comments/6l17g/though_bla...


What do you mean? Is it false or something?


They've been having lots of crazy errors all week (like e.g. all comments posting 3-4 times, other downtimes, etc.). Something's broken on their end for sure.


And US productivity increased 15% during the outage.

Maybe social media should be turned off on Tuesdays and Thursdays, as suggested in Ready Player One.


I'd like to see how that would go.


Number of new posts every 15 minutes in the past 24 hours:

https://imgur.com/a/HqUsYGn

It's interesting to see how it fails. Clearly some users can still post. I'm sure there are things that still work (maybe the phone apps .. got me), but also very clearly, there's a sharp dip in new posts.


I dont know how, but reddit's tech actually feels WORSE now. The old site was simple, but it was fast, clean, and it worked for the one thing it needed to do: upvote downvote content.

Now the newer version is always trying to kick me into either Chrome or the App. I don't want the f'ing App. And the mobile experience is supppper slow. Its literally just a text webpage it should load near instantly, if they didnt have to redesign the whole thing in some new web framework.


At first I thought it was of me refreshing /r/soccer so many times this evening because of CL matches.


Stuff like this is great, you can always point to it when management demands an explanation why the company email server was unavailable for 5 minutes.


That explains the strange uptick in my productivity chart during that time.


reddit is going back to the reliability that they had a few years ago


Webroots website started acting up a couple his ago


The productivity of IT teams skyrocketed world wide.


I remember reading that when a judge ordered Whatsapp to be taken down in Brazil for a few hours, the GDP through that small period went up by 2%.


And nothing of value was lost.


Reddit is the only major social media/aggregator I can think of that actually has to go offline for scheduled maintenance still. I can’t imagine the kind of shitshow it must be internally.

Imagine if Facebook or Twitter had to go down regularly for “maintenance”. Heads would roll.


"We are a startup that inherited a 15 year old website" was one of the things that resonated when I worked there.

Facebook is Alexa #5 in the US and has ~40,000 employees. Reddit is #6 and has ~500 employees. Twitter is #17 and has ~4,000 employees. (Numbers taken from quick Google searches)

They have some super talented engineers that work miracles with band aids and bubble gum.

Edit: Yes I understand not all employees work on everything. I was trying to give a sense of scale. Reddit is a tiny company with a massive website and no way to make money. If you have ideas - they are hiring.


Not really a miracle. Reddit software is magnitudes simpler than Facebook.

Not to mention the quality is often sub-par. For example, many hate their mobile app with a passion and search field could as well just not exist when I tried.


Their new redesign is so completely awful they have to have an option to “just show the old reddit” in your settings. If they ever take that away I think I’ll just stop using it, it’s that bad.


I'm glad I'm not the only one who feels this way - if they ever force me to use only the new site, then I'll leave reddit for good. Or maybe just use it on my phone (Apollo for Reddit is pretty good)


I stopped criticizing redesigns because i usually get over it, but the Reddit one makes the site at least 8x slower for me.


That’s been my complaint. The old server side rendered site was really fast to load. Ever since the first redesign a few years back it’s been really slow.


Here’s half the page you requested, and some other ‘related’ crap.


Same here. I doN't share msot people's outrage when it coems to redesigns, but this one only seems to have downsides.

Imagine redesigning Reddit with simple HTML, CSS and only the most necessary sprinkles of JavaScript. In my opinion it's clearly a document-based website that doesn't need to pretend being a one-page-app.


Forget the settings, just use http://old.reddit.com always :)


If you think that's bad, try using the browser version on mobile... It's full of annoying pop-ups to try to encourage you to use their crappy mobile app. (sadface) :(


Serious question: What makes it completely awful? Do you have ideas that might make it a better experience, besides "just use the old UI"?


Everything. Literally everything. It uses a WYSIWYG input which completely breaks old.reddit formatting inputs, it constantly causes confusion as to what comment you are replying to so you get people replying to the wrong person or replying to the main post, it looks cartoony and blatantly obvious you're on a social website if you're trying to reddit on the sly, It doesn't like browser windows that aren't open full screen if you're trying to reddit on the sly, it severely restricts how much information you can have in the sidebar, etc.


I love the redesign. I used to spend way to much time on Reddit, the redesign fixed that.


Haha I love that. Maybe I should turn on the redesign.


I a lot of people use third party mobile apps because the official one is subpar. They should just acqui-hire Apollo for iOS at least.


Given what happened after they acqui-hired Alien Blue, I'd say no they shouldn't.


Good point, I'd forgotten about that.


500 employees is still alot. Especially given the technology ecosystem that exists for leverage.

The revenue piece is an interesting one because i was an early adopter and remember the revenue struggle. Gold/silver doesn't cut it because it relies on generosity.

FB/Twitter are an ads business. Is that really not too lucrative of a field for reddit?


There are ads on Reddit already. I only ever use their website with an, but I get ads on the iOS app.

I've been seeing a significant rise in the usage of gold/silver/platinum lately, used heavily for banter on r/soccer (and for rewarding quality content as well.)


>I've been seeing a significant rise in the usage of gold/silver/platinum lately, used heavily for banter on r/soccer (and for rewarding quality content as well.)

Same, but it's because they changed the way gold works. If you have a subscription, for stuff like removing ads and what not, you get coins every month which then allows me to give out several silvers every month or let them pile up to give out a week of gold here and there. I doubt there has been a significant rise in people paying and is likely largely the new coin system.


ads on iOS ? Do yourself a favor and install Apollo like every sane person


Meh... they could have rewritten the site from the ground up, including any sort of data migration, multiple times over by now. I have never worked there, but outside looking in, it seems like there is a significant lack of technical vision from them.

Other big players like Facebook, Twitter, etc, are literally inventing pivotal parts of the modern web technical stack. What has come out of reddit?


> including any sort of data migration, multiple times over by now

Someone's never done a data migration while serving 1M+ qps!


But they have had downtime, it's not like downtime for a migration would damage their reputation.


I've come close to that many, many times with zero downtime. It does take planning, more than many folks would think. But it can be done as long as you think it through.


I've never done anything close to that scale of migration, but I also can't really imagine taking an approach that feels like a huge migration.

I feel like I would try to strangle the old app (putting proxies in front and redirecting endpoints as they're rewritten) rather than swapping it out wholesale.

At that point I don't know if it really counts as a migration as much as just development.

Any thoughts on that approach? Is that what you have done, or was it a more holistic swap over?


That isn't a data migration, that's a release.


Did you miss the part where Reddit has 500 employees whereas FB has 40000? There was a point in Reddit history where there was literally 1 engineer.


What's your point? Wikipedia is Alexa rank 10 and has ~300 employees.


What's your point? Wikipedia is developing even slower than Reddit. Not that it would need to, though.


They created a mobile app and dark patterns that pushed users to use it so they could show ads.


[flagged]


Yes, oh no. Dark patterns are closer to fraud than to a legitimate business model I'd recommend to my friends and family. Nigerian princes selling you gold is a business model too.


The company you work for, how do they make money? Do they market their products?


Yes of course. What's your point? All marketing lies, and there is no spectrum, and therefore no moral ambiguity in doing shady things?


There’s always ethical ways to generate money, too. Opt in generosity is proven to be viable.. see: Wikipedia, All of Patreon, Etc


Reddit gold was that, for a long time. Presumably, it didn’t work. (Or maybe wasn’t profitable enough?)


We are past ethical ways...


>>they could have rewritten the site from the ground up

They did. They even tossed out Common Lisp and bought in Python because they thought it would fix their problems. It didn't.


That was in December 2005, less than 6 months after the site launched. Reddit was tiny and didn't even have subreddits or comments yet. This is what it looked like: https://web.archive.org/web/20051205025706/http://reddit.com...

The problems they're dealing with 14 years later as one of the top sites on the internet aren't even slightly comparable.


Sure, may be its time to move to something like Clojure or Golang now?

Python is a good scripting language, But if you are developing backend applications for this scale you probably need some serious JVM language. Or even Golang works fine.


Instagram's backend is written in python. Python is very slow, but still fast enough.


they could still backend to C or rust, no?


[flagged]


Do you have specific insight into working at Reddit?


You've managed to say a lot without really saying anything.

You make plenty of baseless assumptions about my background and ambitions... but I'm not even going to try to address my professional experience (or lack thereof) because it doesn't matter, since you're just using it as a straw man to deflect criticism.

The facts are as follows: 1. Reddit is one of the highest visited websites in the world. 2. Throughout nearly their entire history, they've struggled to put together a way to make money. 3. They have notorious uptime.

Of course they have talented engineers... it would be a surprise if one of the worlds most popular websites didn't.

That doesn't change the facts that they continue to look and function like they're straight out of 2005 (does their search even work yet?), lack a semi decent mobile experience, and have contributed essentially nothing to the technical community in their entire history.


> Reddit is one of the highest visited websites in the world > Throughout nearly their entire history, they've struggled to put together a way to make money.

True, but irrelevant. It also means they handle more traffic per dollar than probably any other site in it's category.

> They have notorious uptime.

Browsing https://reddit.statuspage.io/ for less than a minute proves this is hyperbolic bull.

> and have contributed essentially nothing to the technical community in their entire history

It's not like the reddit source code hasn't been open source forever. But sure, whatever you say. Would have taken you 10 seconds to Google before spouting off nonsense.


so, where is your website?


whataboutism is not a valid reply to the above (fairly legitimate) criticisms


Wikipedia is one of the Alexa top 10 globally (above Reddit, which is #20) and has 300 employees.


There's a reason he compared Reddit with Twitter and Facebook. You interact a lot with social media (commenting, liking, etc.) and also their content is much more dynamic and personalised.

Wikipedia is very static in comparison.


I'm not sure how you can say Wikipedia is static, even by comparison: https://en.wikipedia.org/wiki/Wikipedia:Statistics


From your link: Wikipedia develops at a rate of over 1.8 edits per second

The statistics I can find for Reddit put it at 160 submissions per second[0]. And that's not counting comments or votes. And each vote changes the ordering of submissions and comments.

[0]https://www.reddit.com/r/dataisbeautiful/comments/8aek7d/red...


>that's not counting comments or votes

Or 'hide', in every sub I'm active in I hide each new thread after I've decided I am not interested or after I've looked at it so I can quickly scan new stuff.

Then add automoderator removing things for manual approval, approval of threads etc. Gilding, messages, chat etc.


and Wikipedia also runs on a shoestring - even with some of the smartest people I know working at Wikimedia, the tech department could have 10x the people without running short of work at any point


Craigslist with #8 most viewed at 70 employees.

.. Didn't fact check the numbers though.



Facebook the company has 40,000 employees, but I'm willing to bet the total number of engineers working on facebook.com vs reddit.com isn't too different.


You'd definitely lose that bet.


I worked in a Facebook building with 2000 engineers, and that's not even our biggest office.


They also have very frequent cases where the site throws up an error page or is completely unreachable because it is under heavy load. I can't think of another major social network that has those problems on a weekly basis either.


Not recently, but Twitter's "fail whale" became famous due to problems Twitter had under heavy load.


I still regularly see Reddit failures every week.


That was also a decade ago.


I regularly get the fail whale on mobile, but I suspect it's an intentional dark pattern to push you into the mobile app rather than an engineering issue.


So what? 24/7 builds up to fragility like the NYC subway system or has huge costs. It's social media, not a hospital.


A website is not a subway system. There are no "huge costs" to having basic rolling deployments set up. In fact it's likely more expensive to have extended outages and maintenance windows, because it shows that there are humans running commands behind the scenes rather than automation.


Just wondering (sort of—I'm also slightly trying to point out that you're being a bit of an armchair Reddit ops person :)

What makes you think that the issue Reddit faced today would be fixed by rolling deploys?


I definitely am (but we are on Hacker News after all).

Without knowing anything about this particular issue, at a basic level I'd think any non-trivial site should have a system of rolling out changes to a small percentage of servers and have automated monitoring that fires alerts if it notices any disruptions in functionality. If nothing goes wrong then gradually roll out to 100%. If something does then stop the rollout and go back to the last stable version. Then engineers can look at what went wrong.


From my experience that mostly only applies to the application layer. Especially updates on distributed databases are often not as easy as "stop and rollback" as you have to achieve a stable quorum or other similar dynamic states like that which you can't easily achieve from restoring a snapshot. There your best shot often is to learn how to deal with all the failure scenarios manually and do as many tests on a staging system until you feel comfortable.


The revenue per minute of these sites is in the hundreds of thousands of dollars. While that can be budgeted for if needed, its not just inconvenient for users, it could mean missing a revenue target.


No site is that high other than Google or Facebook. Reddit is actually pretty poor in revenue generated and can easily make up for lost inventory with more ads on the page.


I don't think Reddit grosses a hundred billion USD per year. Do you? That would put it very close to Microsoft.


Right. Reddit's ad revenue is forecasted at 100M this year, so ~$190/min. I was more referring to Facebook's $65B annual revenue, which is $123,668 / min. That's like an FTE's annual salary each minute, so throwing people at uptime has lots of ROI.


A subway system serves only one timezone. A website can potentially need to serve every timezone.


It's more the fact that the engineers working there should be embarrassed, not that it's important. You would think that the engineers there would be able to improve their infrastructure after a few years but it's still as flaky now as it was a few years ago. That doesn't speak well of the caliber of engineers there.


That's unfair. Engineers don't always get to exercise their skills or best practices if their CTO gets in the way. Or if other external obstacles block the CTO. I've worked with very talented people who are totally wasted because the company just doesn't care to get the tech right when it comes down to making that investment.

Autonomy to get things right when it's the right call is sort of a rarity and a real luxury in my experience. I've built a ton of things I knew could have been built better if the rest of the company saw the value in doing so.

When given the option to do your job as asked or quit because you don't have your way, I'd rather just do my job and wait for opportunities to improve things if they present themselves.


Steam is worse - weekly "maintenance" on Tuesday evening, where everyone is disconnected for ~10-20 minutes. It's common knowledge not to play at that time if the game uses Steam networking, otherwise you'll be kicked mid-session with a "Lost connection to Steam" error message and lose all your progress.

And yes, this is happening in 2019.


well World of Warcraft and other MMOs regularly have outages each week and sometimes more frequently. this isn't just for major content pushes either.

heck blizzard has authentication services outages too


I remember that from 15 years ago, would have thought they had enough time to fix it since then.


At least it’s scheduled so you can schedule around it


PUBG regularly goes down for 4+ hour updates during prime NA server time. :/


Do we know this was scheduled maintenance and not some kind of unexpected outage?


This definitely wasn't scheduled. Reddit does still do scheduled maintenance occasionally though (not often, usually only for something like a database version upgrade), and I think they're talking about that, not this specific incident.


is it a big deal? so what if fb goes down, people will just spam twitter about it. reading social media is not some crucial function, it's just a compulsion. and , like drugs, withdrawal symptoms take some time to develop


For the world no it doesn't matter. For the company, every minute down is ads not being looked at and income being generated


they don't make much money so don't have the same redundancy luxuries as other sites


I think they have the resources. Recent example: https://www.reuters.com/article/us-reddit-funding/reddit-val...


i'm not saying they are completely broke, but they certainly are on the more thrifty side of things compared to most tech companies


They make a lot more than does the little outfit I work for and we manage it.


How much does reddit make, I wouldn't be surprised if it operates at a loss.


It definitely does, just like so many VC funded companies, but they could afford to fix their deploys. The reddit back end is notoriously bad and, judging by their new UI, it isn't getting better.


What's the point of making a site reliable if it costs you money to operate? Their strategy is totally different from a Twitter or facebook. All their cycles are busy trying to figure out how to turn a profit.

And the new ui is trash.


They’re Alexa rank #16 globally - that they’re having trouble monetizing their userbase just means they have absolutely no idea what they’re doing.


That doesn't follow at all. There are all sorts of insanely popular websites which have significant monetization issues (think tumblr, wikipedia).

Just because lots of people use something doesn't make them willing to pay for it, and it doesn't make a platform inherently monetizable.


Any attempt at monetization through the "normal" means, such as increasing the number of ads or selling user data, would completely undermine the things that make Reddit such a valuable site. We've already seen a few examples of Reddit moving toward a more legitimate, financially viable social media model, and each one of those moves has been met with significant backlash. It seems that they're in a lose-lose situation: either run out of money to keep the site running, or lose all their users.


I thought Wikipedia doesn’t have monetization issues, but maybe things have changed? https://www.theregister.co.uk/2012/12/20/cash_rich_wikipedia...


Indeed, also not everybody is trying to squeeze money out of their user base regardless of costs (to popularity, principles...).


Reddit gold, the new Reddit platinum and bronze, they already show ads, redesign shows ads much more prominently. Literally dozens of ad trackers when you load their site.

Reddit is just as much of a profit driven company as Facebook or Twitter, I’m not sure why you’d think otherwise.


> Reddit is just as much of a profit driven company as Facebook or Twitter, I’m not sure why you’d think otherwise.

Because, among other things, they don't show annoying ad formats and don't demand real names and phone numbers from their users.


Yet :)


We'll see. Meanwhile, the old design is still available and not burdened with lots of trackers.


Their old design totally loads ad trackers...


And I wholeheartedly wish Reddit will continue doing what they are doing. As a user, I would vote for scheduled maintenance downtime to many monetization attempts present on other networks.


Looks like they’re having trouble just keeping the lights on without taking VC money.

Not exactly a sustainable plan if you value using their service, and they’ve already got early investors pushing them to monetize.


The problem these sites have is that most of us don't particularly value using their service. We might appreciate the facility they offer for free, but what we truly value is the communities that are found there and the information and discussions that originate from those communities.

If Reddit died tomorrow, we'd all shrug and find somewhere else to talk, just like the countless forums we used before Reddit that are no longer with us. Such is the danger for any social media: if all you provide is a commodity platform, you're only ever as valuable as the community you can build and whatever that community is willing to contribute, and ultimately you have very little control over those things.


They don't make nearly as much as others with that traffic. Brand safety issues with subreddits and discussions, anonymous user profiles, poor mobile experience, mostly link aggregation instead of content, high adblocking, etc.

Here's an old answer of mine comparing reddit's $7M to Instagram's $700M in 2014: https://www.quora.com/Why-are-reddits-advertising-revenues-o...


I think they just don't sell as much data as something like Google, Facebook, Twitter, etc.


Each outbound link on reddit is tracked, similar to how Google tracks users clicking on SERPs. This can be tied back to a user to build profile on your interests, in addition to sub-reddits you subscribe to etc. which can be valuable if combined with other demographic data about you.

I dont know if they started selling this data directly. Last I spoke to ad people there, they had plans to build out their own tracking and analytics infrastructure, to improve things like geo-targeted ads, as their capabilities were lacking signficiantly.


You can turn off outbound link tracking in the privacy section of preferences. Uncheck “allow reddit to log my outbound clicks for personalization”


which means your logged in and your behavior and activity is captured, and available for ad-targeting anyway. There's no real opt-out.


Facebook goes down all the time


Can't they run in read-only mode during maintenance? That would be a relatively simple workaround, I suppose.


They used to. Not sure why they don’t now.


I remember the read-only mode as some presumably cached pre-rendered html, not sure if it ever existed for the app/api. I wonder if the read-only mode didn't mesh well with the single page app they have been pushing - perhaps it wasn't monetized in the same way.


Marked as resolved now, mods should probably update the title.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: