This seems to be about some odd certificate issue, not the network, which caused Microsoft to take access to its service management system down.
From the article itself:
> It later added that less than 3.8 per cent of hosted services had been affected.
If this was about Google or Apple, this submit would already have been flagged, taken off the front page, and several accounts would have been hell-banned.
</endjoke, but it's really true>
It will be interesting to see if Microsoft expands on this issue and if we can learn something about it... Perhaps other datacenters are also vulnerable to cert issues in their management systems.
"We have identified the root cause of this incident. It has been traced back to a cert issue triggered on 2/29/2012 GMT," the software giant said.
So here is a quiz for you, does you software know that this is a leap year? I'll speculate that someone's software didn't recognize it as such. Personally I'm always on the lookout for this sort of 'weird' thing because in the world of testing the edge cases are often poorly tested.
In my rather limited experience as a sysadmin, I've found avoiding those calls to be one of my best motivators: My systems work like they're supposed to because I like my sleep.
Today, February 29th, is a "leap day" which happens every 4 years in a "leap year"... Otherwise, the last day of February is the 28th, and today would normally be March 1st.
I'm assuming that when the day rolled over, it caused an issue with an internal system at Microsoft, that had something to do with certificate dates and/or timestamps.
It's not just Microsoft that it's happening to, today I could not automatically renew any domains at Namecheap that expire tomorrow because the registry could not produce the correct expire-at date.
How the hell is that possible? Every platform out there has high quality date libraries. It's a cliché that date & time operations are very complicated (leap years, leap seconds, time zones, daylight saving time and so on), every even marginally competent developer knows instinctly to stay away from writing their own date & time code and trust a library instead to do the heavy lifting. In addition, every tester should know that when dates are involved, they should try Feb 29th and see if it breaks anything. How can bugs like this slip through?
The most frustrating thing for me was the complete lack of any real communication from Microsoft. For awhile, even their status dashboard was down. I only found out about it after I got a PagerDuty alert and had to search Twitter (other people complaining about it) to confirm.
We have an Azure CDN backed by a Compute Instance, and zero official notice from Microsoft about this still. I've learned more about the problem from news articles than the company that provides the service. Fortunately we haven't finished migrating the rest of the site to Azure. No emails from them, nothing. Not even a tweet on their official @WindowsAzure account. Frustrating.
I guess this sort of thing was bound to happen at some point. As the article says, all the major cloud platforms have had outages at some point.
As someone who is actively considering building products using this platform I'm keen to see how well they manage this issue. How do they communicate during the outage, how open about what went wrong are they afterwards, do they learn from it, and so on. I particularly like the efforts Amazon have gone to in the past to show they are learning from these issues (see http://aws.amazon.com/message/65648/) I'm hopeful that Microsoft will show the same level of openness.
Heroku Dyno's were down for a lot of clients for several hours on saturday. One site that I use a lot, Intercom.io, was completely off the grid because of this. I was wondering why there was no news about it here.
It's unclear how big a deal this outage was, as I can never make heads or tails of what's really behind an article in the The Register.
That said, I run an app on Heroku and noticed that outage - Pingdom sent me a bunch of emails, which I saw after a day spent outside. The first thing I do when I get a Pingdom failure notice is check status.heroku.com, and sure enough they had an issue and were working on fixing it. I wasn't particularly expecting there to be mention of it on HN because, first, it wasn't an enormous outage, second, it was on the weekend, and third Heroku always does such a great job of posting about issues.
I don't know if they did so, but I'd expect Intercom.io and other affected sites to post information letting people know that the site is down and why. I haven't tried it yet but Heroku allows you to specify a custom error page to be served [1], which you could update on the fly to let people know what's going on. For other outages I have let users know via Twitter and Facebook, though obviously there's no substitute for a nice up-to-date static error page.
It was on the front page for a while, however it was the weekend, so likely didn't garner as much attention. And was like 2 hours... not 8-9+ like Azure.
It's a bit unfortunate description of time, I think. They could at least say "2/29/2012 01:45 GMT". Otherwise it looks like there's a different date in each timezone (yeah, there are two different ones, but not different in each, the bug started closer to beginning of 2/29/2012 GMT+2).
I think this is the problem with rewriting platforms, libraries and languages from scratch instead of incrementally chiseling it into something stable like in the open unix world.
instead of incrementally chiseling it into something stable like in the open unix world.
LOL. If I had a penny for whenever the "open unix world" chose to re implement some stuff from scratch, I'd be rich. From Gnome and KDE always changing stuff (especially the KDE multimedia architecture took this to comical levels), to FreeBSD moving to Clang...
As a non-tech observer, Azure actually struck me as an honest attempt by Microsoft to add a compelling offer to the mix.
However, their status page-cum-website seem to be hosted on Azure itself, which is ridiculous in a situation like this. (Almost like Twitter having a status page on tumblr.)
Worst of all, even www.twitter.com/windowsazure offers no comment at all so far. Isn't this wiping out any credibility they might have built up with developers? Is anyone affected?
Edit: if this truly is because of 2/29, I guess anyone signing up from now on will get perfect service.
I don't think that's true. Because I remember giving somebody a lot of pain for exactly that when I worked there. Of course, it might have been gone down for other reasons.
Disclaimer: ex-Windows Azure person, no insider knowledge on this particular outage
That's typically because the 'service management service' typically kicks in when it needs to do things - allocate capacity, restart things when stuff goes down, etc. By default, it isn't touching the running apps inside VMs. There is no Windows Azure equivalent to Heroku's routing mesh to be taken down; the requests go to the VMs directly via the various networking layers.
We had problems today. We had the bad luck that one of our web roles crashed during the time the admin interface was down. That meant we couldn't restart it and neither could microsoft. We will be adding instances to the role to avoid similar problems. Otherwise, we are very happy with Azure.
Can you explain what your problem is? Azure will automatically restart instances that are dead afaik (unless you are saying that the automatic restarting was broken as well?).
That sucks, we have a bunch of media stored with them and we're looking forward to using the cloud for some .NET based intensive processing (reporting).
A bit off topic: I've been trying for days to get Azure to work with .webm or .ogv files. Maybe I'm using the wrong tool (CloudBerry). I want to be able to deliver HTML5 video from the cloud but I'm without success for FF users. Luckily, my video player features a Flash fallback which is awesome.
was the media stored in blob storage? From the article it looks like storage accounts weren't effected, just the management api and ~4% of service accounts.
Today's outage took down our website, our computing grid, everything. And this afternoon, when Microsoft said "the majority" of Azure clients were back up and running, we were still in the dark. Dammit.
Don't know why you got downvoted - AFAIK Siri isn't tied to Azure (although it could be using it for storage), but iCloud runs across both Azure and Amazon.
Is it really trolling? Honestly, who uses Azure? Like most MSDN subscribers I did some mashing in it -- that still exists -- but did absolutely nothing real in it. My sense is that very, very few did anything beyond prototyping in it.
And just to add some opinion, the reason I wouldn't even consider it is Microsoft absolutely flippant ADD when it comes to online services. I have zero faith that they won't just shut it down tomorrow.
Such as? Microsoft's case studies for Azure are absolutely dismal.
I note the other comment mentions Apple, which was a pre-release beta rumor based upon the IP that a beta iMessage lived at (since moved). It speaks volumes, I think, when such a disproven pre-release claim is still held as the example.
Exactly.. Not sure why my comment was down voted - sure, maybe pinging is disabled and that part of my comment wasn't valid. But still, isn't responding.
From the article itself:
> It later added that less than 3.8 per cent of hosted services had been affected.
If this was about Google or Apple, this submit would already have been flagged, taken off the front page, and several accounts would have been hell-banned.
</endjoke, but it's really true>
It will be interesting to see if Microsoft expands on this issue and if we can learn something about it... Perhaps other datacenters are also vulnerable to cert issues in their management systems.