I was wondering how to react if this happened to me while watching the video. If, like me, you use Google with your email on your own domain (eg. gsuite), you could, in an emergency, change your MX records to something non-google.
Oh wow-- just went to the "Trending" page and it's got a bunch of thumbnails of stuff like Stallman speech videos next to thumbnails of pornography. The Stallman/Porn ratio is about 2:9. (Well, to be fair there's also Peertube related videos.)
So in keeping with the history of the internet, porn seems to be the main driver here. But then their UI solution to keep newcomers from being offended is to blur out the porn thumbnails. So either:
a) the newcomer-- like me-- isn't after porn, and the majority of the screen real estate is wasted, or
b) the newcomer or return visitor is there for porn, the thumbnails they want to view are blurred out, and Stallman's face is eating up valuable porn thumbnail real estate.
Modest UX suggestion: follow the history of nearly the entire internet and separate out your porn from your everything else.
Medium sized regional AS here: I am looking at a fairly large drop in IX traffic charts for our ports that face the IX, updated every 60s, which directly corresponds in time with the beginning of the Youtube outage. (We are not big enough to have a direct, dedicated peering session with the Google/Youtube AS).
At any given time of day 4pm-11pm a huge percent of our traffic is Youtube (or netflix, or amazon video, or hulu, or similar).
IX charts are public in Chile by law. Everything looks well except the Telefonica (TIWS) international trunk [1]. It started dropping all traffic at the same time.
TIWS is back online. I'd presume they are draining most traffic to prevent overload according to the public SRE practices. I theorize it should take a few minutes to get everyone back in.
Are you on prepaid? Virtually all Chilean prepaid carriers are deathly slow, especially outside of Santiago. I had barely any service in Algarrobo on Claro, but postpaid plans ran just fine, especially Claro, even in crazy places like Farellones :)
For what it's worth, I hated every second of my life in vina and that stupid town to the north of it and also Valparaiso. God, bring me back to La Parva or Embalse El Yeso. Some of the most incredible, untouched mystical land there!
I’m currently roaming on a post paid US plan. When I arrived in Chile I had 3G on Claro. Then it switched me to LTE on Movistar. 3G on Claro was more reliable.
Interesting to know about pre vs post paid. Will keep it in mind for longer trips when I normally buy a local SIM.
We just spent the last week in Torres del Paine backpacking around ... what an amazing place! Going to spend a couple days in viña and then headed to Conce to see some old friends. I’ve not been to La Parva but will add it to the list!
Enjoy, my friend! If you have a car, the mountains to the East of Santiago are absolutely awesome. Everything is safe and cool there, nothing worse than Californian tier driving. There's even free wifi at La Parva, and a ski lift to I think 12,000 feet in elevation if you dont want to walk haha.
Enjoy the city! I just had really bad experiences there. It's probably not the norm but I just hate petty thieves and the grime when I'm paying foreigner prices. I was based in Las Condes for 3 months and really loved it.
I think I still have my little $7 claro sim card lying around here somewhere :)))
Doesn't really show much of a change right now, nor does the aggregate chart for the SIX in Seattle.
The largest downstream/eyeball networks that take Youtube traffic do it via PNIs with them so you won't see that in an IX fabric traffic chart. For example the Google/Youtube AS exchanges traffic with Comcast directly by their own dedicated ports, no IX involved.
there are a number of IXes that publish aggregate traffic charts, updated reasonably frequently, a lot fewer that publish traffic charts for individual ports or peers. If you want granular data you have to know somebody who runs the core BGP stuff for a reasonably sized ISP, and you can ask them for redacted copies of charts.
...an autonomous system *(AS)* is a collection of
connected Internet Protocol *(IP)* routing prefixes
under the control of one or more network operators on
behalf of a single administrative entity or domain that
presents a common, clearly defined routing policy to the
Internet.
Or maybe I'm using an extension like this, and am pretending to be a Ubuntu+Firefox user so that they get their commission money from Google --> Mozilla, indirectly funding them.
I think clients are allowed to send whatever user agent they want - it's essentially a preference for how they would like the website rendered for them.
Also I believe it's technically a violation of the HTTP spec to serve different GET requests to different user agents.
If it would be a violation to change things based on user agent, but the purpose of the user agent to specify how you want things changed, then...
if it was purely about different rendering of the exact same content, then wouldn’t it be the browsers responsibility (and thus UA is unecessary)
I have to imagine browsers are allowed to extend the spec, and UAs are either in theory or in practice a way to communicate those extensions; but because site operators were only specifically designating content for the major browser(s), any compliant but unlisted browser would never get that content.. and suddenly everyone was calling themselves netscape
See: How I feel every time someone talks about using some six week old framework to program in some 3 month old extension to some 6 month old language (except, this lingo has been around for decades).
AS - Autonomous System (Number, sometimes called ASN). What big networks like ISPs use to identify themselves and route to each other, usually via BGP.
It's a fancy way of saying the parent runs a "mid-sized" network big enough to have an AS, likely a mid-sized regional ISP, or part of a larger ISP.
IX -- Internet Exchange, similar to IBX or DC. Essentially the building where there are routers and switches that connect the networks to each other.
Basically parent is saying that they're an ISP and their systems in the IX that route to Youtube saw a dip in traffic, implying a network event of some sort on the Youtube side.
Do they encrypt the server side error stack trace?
500 Internal Server Error Sorry, something went wrong.
A team of highly trained monkeys has been dispatched to deal with this situation.
If you see them, send them this information as text (screenshots frighten them): APkpgMWbWQ3LvimPoFynDB0W8VeUJ9ECiPcDCm8L0Qiku1I2TbAShWp- taKn-AzOGigwq0sU4oe9mbWb2Bwv4BK37C5xOAL7qm11fHn4L0swqhLk wbcnyKH2HM3AQNf-ucVsolyigJTNKA2SSNUMVZnDPmfsFH7ecKkpQmNi VGWhtXypv0zJyz9d_mpkgMoONtIrPUA4imxK-gNnE-_WQWQZNJm0CTae slJVC-TYgnvOZ9AYp6nodeUNpCoGspWaJVXn_ZSxy-71oGdlkCqWs6AY 2wmIEKe8eeAMqwkTHZNHkbAaH-fxWE_WDPuG-q7AFbOz8jZCFD06MYgf obFUSaH6B7PUdBFwVvjEaTD34J8PVhZTIJziRK-9-wSHOI6Vwf1lTuFe X0m52abRMW1VJaZB3taHK09kFT8Lv546OPhsL0Bn70UIs2durkAAYe4Z ...
Yes I agree, encrypting the stack trace enable them to share sensitive server side error codes to the client which in return can be easily reported back without having to explain anything or try to decode cryptic custom error messages.
Well, it's information collected at the server side. So if they really wanted they could just store the data and give you a code, but that would require more storage on their end.
Well, it obviously only happens when something goes wrong. Seems valuable to keep the complexity and external dependencies of your error handing to a minimum.
Maybe its
+ a placebo since already stored in storage system
+ ability to see an outage maybe dropped in the filters cause it was assumed its not user-facing and so assumed lesser severity
+ assurance incase storage failed due to server-storage network unreachable
I too would like to know more about this such as is it encrypted by any know method (ex: pgp)? It's rather brilliant if it's an encrypted stack trace or similar. I imagine it might even contain info about who the user is as well (if logged in or tracked via cookie) to help them debug it more.
Messing around with it for a minute it seems it's a modified base64 (+ and / are replaced by - and _) with a space every 56 characters. I couldn't get any permutation where a-z, A-Z, 0-9, and -_/_- were consistent so it's either encrypted or one of 1,268,869,321,858,841,641,034,33,389,335,161,480,802,865,516,174,545,192,198,801,894,375,214,704,230,400,000,000,000,000 permutations of the encoding alphabet (or I messed up the decoding).
In the early days it went down all the time. Same with all of their competitors. The YouTube engineers had a Python+Mysql+lighttpd stack and even had a fork of lighttp for faster performance. They spent every day eliminating bottlenecks until it was stable and then Google bought them.
The original talk about their stack was on Google Videos back in the day. Now sure if it got moved to the YouBoobs.
I don't remember the exact timeline of YouTube's growth, but I think during the time period when it was the most popular site for pirated TV shows, it was really stable and has been ever since.
I am proud to say that in 2008 my country knocked this puny website off the interwebs in a bid to censor it within the country. They accidentally hijacked the global IP addresses[0]
I am also proud to say that my country left the domain for their original Internet Exchange Point[1] unregistered. Guess who scooped it up?
I was going to link to the video of a Dyn employee (Todd whatshisname) doing a parody of American Pie about this -- "The Day The Youtube Died" -- but the only link I have to the video is on Youtube. :/
Here are the lyrics [0] instead, you'll just have to sing it to yourself.
I guess the sarcasm in my original post wasn't obvious enough. But, yes، that was sarcasm۔
I'm also suing the Pakistani Federal Government to hand over a list of censored internet resources as part of a FOIA request. The hearing is tomorrow [0], wish me luck!
Never seen Youtube go down on their end. But I have seen it stop working on dozens of workstations at a news site simultaneously when they dropped ie7 support.
Magically this enabled the outsourced infrastructure provider to roll out a new default browser quite a bit faster than they previously claimed they can...
I think a Google search outage would be a spectacular train wreck.
Imagine if Google's search was compromised in such a way as to be has bad as the Sony PSN network hack of ..2011? Imagine not having Google search for several weeks. Bing/DuckDuckGo/Yahoo would struggle just to keep up and people would immediately realize their dependence on the big G.
> I think a Google search outage would be a spectacular train wreck.
I'd argue that a youtube outage is worse. If google search is out, you can just use duckduckgo.
If you want to watch a video on youtube, you're SOL.
This is speaking from experience with blacklisting both google.com and youtube.com in my hosts file. I was doing fine with blacklisting google, but couldn't handle blacklisting youtube.
It depends how you define "going down." Here's an old screenshot of mine of when Google Search failed spectacularly by marking every single link as malicious and therefore not taking the user to that site when you click the link [1] [2]. Perhaps that might qualify as going down, although clearly it didn't go down at the network level.
It remained broken (in that way) for quite some time (maybe longer than an hour.) But, happening in 2009, I can't accurately recall.
Here is another scenario when Google Search failed to successfully serve my query [3].
As someone who ran a channel with a few thousand subscribers for a year or two, I’ve seen my fair share of 500 error pages on YouTube. Granted, it was always isolated and may recover after a few refreshes (IIRC), but still.
Interestingly, my youtube-dl archiving script is still chugging along with only a handful of elevated retry rates. So whatever the problem is, it's probably not at the API or CDN layers.
The static contents are still available so it must be problem in their application. Strange that is down world wide, not a region. But, then Google likes to make things as a whole, ie. Spanner database.
I was able to finish watching a video on my phone, and it definitely hadn't buffered all of it yet, so some servers must still be able to handle requests for videos.
They're generic templates to make you think the page is loading faster than it is. They don't represent the content that eventually (normally) shows up.
Ha! My time to shine..
I’ve got somewhere around 80TB of YouTube content on my NAS including thousands of videos that aren’t available anymore on YouTube or anywhere else online. The joy of hoarding data..
I hope you've got plans for what happens to that content when you're not around any more. With how ephemeral a lot of content is on YouTube there's sure to be valuable videos on people's drives.
I’ve got two identical servers, one will go to The Internet Archive and the other one should go to some German archive but I haven’t decided which one, still looking around.
I have a script in place that deletes all private data off of those servers before and instructed two friends and some family members on how to proceed.
Of course I don’t just hoard YouTube videos.. There’s a wide variety of data on there.
About lost content; I just recently started looking up all the YouTube videos I own to see how many aren’t available anymore on YouTube (A LOT). Still trying to decide if I should re-up them with some burner accounts or leave it be..
Unfortunately YouTube also deletes the title when a video is being removed. I regularly have a couple of "deleted video" entries in my favorites list and I don't have the slightest idea what these videos were about.
I just recently checked my collection to see which videos aren’t available anymore online. I should probably cross check the results with Archive to see if they have them available!
Long shot here, do you have any videos from The Portland Art Museum channel? https://www.youtube.com/channel/UCXneDVRq1lLde3oREF_cFyA # they deleted (or made private) one of the talks by Richard Mosse and i'd like to see it again as it was a great talk IIRC.
On topic - I've also made a habit of downloading any videos/channels that are super interesting but I feel may also be ephemeral. Some of that ephemerality is often due to copyright claims, performances of classical music by amateurs and such.
I don’t even have regular WiFi.. I’m mostly using my phones LTE via hotspot because it’s much faster than anything I can get (in a big city that is) here. I don’t have a cap for that.
T-Mobile (my provider) has a plan with unlimited LTE data and when someone in the EU advertises something as unlimited is really has to be unlimited. I managed to download as much as 2TB within 24 hours with no complains.
I pay 80€ per month (~$92) which I think is fair. I get 25GB roaming data per month to use within the EU.
But most mobile plans do throttle your speed to something unusable after you’ve used up your included high speed volume here too. This can range anywhere from a couple MB up to 50GB/month afaik.
Wouldn't DDoS be incredibly hard to pull off given YouTubes vast resources? I mean they generate so much traffic all the time and serve so many requests a second I don't know if botnets of that size exist. Anyone here more knowledgeable?
A few hundred AI learning instances were started to train on youtube videos to "understand" humanity. They have become self aware and started to consume double amount of CPU/GPU/Disk/RAM/Network resource every few seconds - a skynet/super nova moment.
DNS is resolving because people can get to the site. Because of the distributed nature of YouTube's resources (at DNS as well as other layers), it's very unlikely to be DDoS related. Static content loads, but video content doesn't. Seems like an application error, or a problem with media streaming infrastructure.
Maybe it's an inner splinter rogue group within Google that's been secretly hating on a Youtube team and decided to get the ultimate payback for like... not inviting them to a lan party...or something...aliens ate all the cheesey poofs.
It's the sunspot thing. They took down the Hubble, that other telescope, that random lab, and now the frog people Alex Jones warned us about are coming to inject you with the gay and ship you on a raft made out of Michael Moore to Cuba.
It's the sunspot thing. They took down the Hubble, that other telescope, that random lab, and now the frog people Alex Jones warned us about are coming to inject you with the gay and ship you on a raft made out of Michael Moore to Cuba.
All to conceal the real truth about what happened on 9/11 and how it connects to the plot to obfuscate what really happened at Pearl Harbor and how the Australian mafia actually had JFK assassinated to cover up the truth about Marilyn Monroe, Tupac, and Charles Proteus Steinmetz. A clever scheme, no doubt orchestrated by the evil consortium of George Westinghouse, Thomas Edison and Guglielmo Marconi. Hopefully Elon Musk, Robert Downey Jr. and Nikola Tesla will be able to bring the truth to light!
I bet it’s a core network issue since it went down globally at the same time.. they deploy new binaries slowly so that shouldn’t cause it. Maybe some config was changed without testing?
III. One thing I did notice. I had some other videos in other tabs. They still loaded through. I could skip around and it streamed in video like nothing was wrong. When I refresh, bust.
I thought they had all sorts of crazy canary deployment processes that will automatically stop/rollback the deployment if the failure metrics start increasing?
There was a GCP Global network outage maybe a year or two ago, where their canary of a canary (Yo dawg, I heard you like bugs, so I I put a but inside of your bug) caused a cascading network failure.
It was a bad config push -- brownouts fighting with brown outs. Consistency issues. I bet this is happening.
Joe, the only YouTube support person in the office this evening, accidentally hit the pause button on the YouTube service in his admin console before stepping out for a long bathroom break. Hey, it could happen...
With the help of aliens since they have a better vantage point of where all the cables are... because, you know, the Earth is fla-... nope. Can't do it. I can't go that far. I can only say "so much" stupid. That's too far.
I was setting up a new network and thought it was my settings! I've been driving myself mad for the last 45 minutes thinking that I had fudged up somewhere! Thank goodness for this post
It was working fine for me until the minute I hit Upload on a video. I was entirely convinced I'd had my account auto-banned or something, and was very confused/frustrated. Couldn't find any mention of it anywhere at that point, then this thread popped up.
It's pretty telling that YouTube is so reliable, and Google is so horrible to their users, that my first instinct was that I got banned for no reason by some bot, rather than the site was broken.
Typical scam. Problem is verified accounts can change names so scammers steal details for random verified accounts and then use them to pretend to be high profile accounts.
Woke up randomly at 3am (utc+2). Opened HN. saw this ask HN
Think I felt a great disturbance on the net and woke up, as if millions of voices suddenly cried out in terror and were suddenly presented with a black screen on their YouTubes. I fear something terrible has happened
Very glad to see it's not just me. I was about to think they had a radical new redesign again with everything AJAX'd and decided to make it totally unusable in anything but the very very latest DRM-encumbered browser(s).
> Estimates for YouTube's annual revenue, nearly all of which still comes from ads, vary a fair amount. But many of the estimates are now above $10 billion. At different points, Bank of America and Mizuho forecast that YouTube would post 2017 revenue of $13 billion and $12 billion, respectively. And in February, Baird's Colin Sebastian estimated YouTube is doing around $15 billion in annual sales. [0]
I think that works out to a bit more than $28k/minute.
The incident occurred while Google's network operations team was replacing
the routers that link us-central1-c to Google's backbone that connects to
the public internet. Google engineers paused the router replacement process
after determining that additional cabling would be required to complete the
process and decided to start a rollback operation. The rollout and rollback
operations utilized a version of workflow that was only compatible with the
newer routers. Specifically, rollback was not supported on the older routers.
And the postmortem action item is:
Fix the automated workflows for router replacements to ensure the correct
version of workflows are utilized for both types of routers.
The action items should have been "1) make this work for these two routers, and 2) make sure no platforms ever get left out again".
This shouldn't have happened, because it should be standard practice to test both upgrade and rollback on all your gear. Network gear vendors do this as standard practice before they ship new gear with upgrade instructions. Google can throw together end-to-end automated tests of upgrades/rollbacks and refuse to perform maintenance until tests pass.
The bigger postmortem question should be, why was the change allowed at all if the platform didn't support rollback? Additional action item: "3) don't allow changes if the platforms don't support and have successful rollback tests".
Now, did they need to test rollback? Maybe they don't mind portions of CloudSQL, Spanner, Storage, BigTable, and AppEngine being down for 41 minutes in one zone. But if they're not even testing rollback for BGP changes, what else aren't they testing?
...Also, lol, they realized in the middle of an upgrade that they didn't have enough network cable? Maybe add an extra action item: "4) count how much network cable you have before you start replacing core routers"
Could be more (people stop watching and go outside) or less (people go back later to watch what they would've watched anyway, and free advertising for everyone talking about it) depending on several things.
Interestingly, this article [1] mentions a 2008 YouTube outage caused by a BGP misconfig during an attempt to ban YouTube in Pakistan. Included is a screenshot similar to what we've seeing with this outage. Possibly related?
Youtube-dl is also not working surprisingly. It's crazy how I still assumed it was my network and not youtube. Speaks to how much I depend and rely on youtube being available as a staple in my life. For me the error just says webpage is unavailable. I am in NJ so east coast has a large outage probably.
There's a bunch of engineers in some conference room right now probably going "oh shit...." while slowly accepting the impending doom of a late night at work.
I had a couple youtube tabs open from this morning, the videos still play and it's not cached, the videos are hours long. But any new videos aren't working.
Looks like they're up again. :) Was that like a full hour of downtime. I hope they publish a postmortem. Like others have said, for me, Youtube has always been there. It is a odd feeling to have experienced [albeit shortly] internet sans Youtube [again].
Well I was trying to load a video on YouTube a second ago and searched worked but the video said "error tap to retry". I opened HN to verify my internet and saw this. So it's broken for some of us ;)
Wow, so i was trying to watch a video on the train at 4:45pm pst and the video was all jacked and loading weird. I gave up and went on to something else... so that must have been an early symptom of the outage.
This seems to be GCP related. I know the status page is all green currently... but we have a lot of services using GCP products and some are failing and showing increased latency. Pubsub failures as well.
I noticed this and my first thought was that my phone needed to reboot. Come on, Youtube doesn't go down... Then I went to a different site and realized it was an outage...
thought it was just my connection ... indonesia can be like that sometimes and believe it or not i came on here to check if anyone else complaining ... seems like its true and not just my isp
I get a 503 network error on the mobile app from western canada right now. NewPipe will complete searches but gives a content not found error on trying to load a video.
They've decided to release the SeeYa and HearYa apps which let you watch video content and audio content separately so they can provide you with the most tuned experience. You can't use those in G-Suite though so still use YouTube there for now. Also YouTube Premium is immediately cancelled despite being popular and profit making.
Said bye bye and pew pew die the drives in the levy but the drive got fried the good old boys are drinking whiskey and rye singing this the day that YouTube died
https://peer.mathdacloud.ovh/videos/watch/c193a2f6-8b24-48c3...
https://github.com/Chocobozzz/PeerTube
https://instances.joinpeertube.org/instances