The handling of this has been quite terrible. The article is the first "good" communication about an event that started 7 hours prior. And according to the communication will take another 4 days to solve completely.
Before this, the Technical Solutions Director tweeted solutions that did not work for end users, but highlighted a typical IT centric approach to problem resolution ("Works, What's the problem?") [1]
For anyone not already aware, check out Let's Encrypt. I am evaluating it for about 200 domains now in earnest after having it on my horizon for some time. At least to have it ready as a fallback. [2]
Getting 200 EV certificates in a hurry from a different CA has been costly this morning.
I've been on HN for nearly as long as pg has so feel free to give me a shout here if you have any questions. It's past midnight in the UK right now so I'll answer them tomorrow morning. If you prefer email: mike@certsimple.
Tried it out for the heck of it. Unfortunately Dun & Bradstreet has old address information for my company and their account creation system is broken (error 'CEPDEV101' when creating an account). So the "3 hour" validation will take a lot longer, through no fault of CertSimple's.
Thanks bakavic. Our UI is currently built for desktop since most of our customers are coming from desktops - there's some data entry and we use things like webcrypto and HTML5 clipboard that may not be available on mobile platforms. We're going to improve the mobile experience in future but have other higher priority items (like better DBA support!) on our roadmap first.
That is a good point. I guess the move to Let's Encrypt is a change in mindset completely. Away from Marketing snake-oil to a simple solution to a technical issue.
EV certificates do not add additional security as far as I am aware, it's simply a measure of extended validation of the customer, which in my experience is only fluff.
So going with Let's Encrypt may as well do the full cycle and sell this free solution to non-technical people who believe EV provides additional value.
> EV certificates do not add additional security as far as I am aware, it's simply a measure of extended validation of the customer,
Correct.
> ...which in my experience is only fluff.
Depends on what you want to accomplish. If you just care about getting to SOME website with a secure connection than a DV certificate is all you need.
But take my employer: Capital One. We're a bank. When people go to https://www.capitalone.com it is important that they get our website. ALSO, though, it is important that when people go to https://www.capitalοne.com that they do NOT get some malicious website. (Can't see the difference? What glyph did I use for the letter "o"?) This is just one example of how an EV certificates can be useful for certain purposes, although it may not apply to your business.
As most of you are aware, we are experiencing an internal process issue (details below) that is impacting your business. While we have identified the root-cause, we deeply apologize for the problems this is causing you and wanted to ensure you that we are actively resolving the issue.
GlobalSign manages several root certificates and for compatibility and browser ubiquity reasons provides several cross-certificates between those roots to maximize the effectiveness across a variety of platforms. As part of a planned exercise to remove some of those links, a cross-certificate linking two roots together was revoked. CRL responses had been operational for 1 week, however an unexpected consequence of providing OCSP responses became apparent this morning, in that some browsers incorrectly inferred that the cross-signed root had revoked intermediates, which was not the case.
GlobalSign has since removed the cross-certificate from the OCSP database and cleared all caches. However, the global nature of CDNs and effectiveness of caching continued to push some of those responses out as far as end users. End users cannot always easily clear their caches, either through lack of knowledge or lack of permission. New users (visitors) are not affected as they will now receive good responses.
The problem will correct itself in 4 days as the cached responses expire, which we know is not ideal. However, in the meantime, GlobalSign will be providing an alternative issuing CA for customers to use instead, issued by a different root which was not affected by the cross that was revoked, but offering the same ubiquity and does not require to reissue the certificate itself.
We are currently working on the detailed instructions to help you resolve the issue and will communicate those instruction to you shortly.
Thank you for your patience.
Lila Kee
Chief Product Officer
GMO GlobalSign
US +1 603-570-7060 | UK +44 1622 766 766 | EU +32 16 89 1900
www.globalsign.com/en
One thing that shocks me about this is the browser responses. Opera would not allow me to visit the sites affected, citing a revoked cert and possible attack. While I appreciate the warning, it was odd to see the site fully disabled.
Safari on the other hand silently failed the https auth and served the page regardless. Concerning behavior for a revoked cert.
That might depend on the specific Safari version. I am using Safari 10 on macOS Sierra, and it would not allow me to access any site using one of the affected certs: instead it would just serve an error page.
I think that was more because of the site, not the certificate - the grey failure page would appear for sites that use HSTS[0]. If the site didn't use HSTS or if you visited it for the first time, you'd get an ordinary certificate error alert.
"3.3 Intentional or accidental errors
This Warranty Policy warrants against intentional or accidental errors introduced into a Certificate by any personnel of a GlobalSign RA or a GlobalSign LRA due to failure to use reasonable care as required under the CPS."
Not a lawyer, but will push this with our legal team for sure.
This could destroy GlobalSign, so there must be a clause to exempt them from liability ... ah, here it is:
"8.1 Total amount for warranty exhausted
When the maximum limit allocated for warranty payments is exhausted, GlobalSign shall have no further obligation to refund any Beneficiary, unless required otherwise by applicable law."
They didn't introduce any errors _into_ a certificate. They affected the certificate chain, but not the certificate directly. I'm certain this is how their own lawyers view this.
This is going to get hilariously legal. A certificate from GlobalSign or anybody else is the product of the chain, if the chain isn't there and the cert isn't valid it's no more valid than a self-signed one.
Tell me again, anybody, why we trust these people?
So basically if you ask them for $1.250.000 in potential lost sales and damage to your reputation because your website is unavailable and appears to be compromised, you'll get it? Nice.
I bought new certificates, for a new set of domains, through AlphaSSL today. One hour later customers starts calling, complaining about revoked certificates. Initially I assumed they had screwed up somehow and revoked our old certs, but after reports saying that it worked with some browsers and failed on others I started googling for recent related issues, and found out about GlobalSign.
Man, do they suck at communication. We're now 14 hours into the incident. 6-7 hours ago they posted a trouble shooting guid, promising new intermediate certificates for AlphaSSL and I've just been informed by their support that it'll be another hour before they're ready.
It's now 02:00 in dk, so I can expect the new certs at 3 and be done by 4.
Fun night.
Thanks GlobalSign.
P.S. Also thanks to the guy who made their marketing department stop tweeting iot crap while this is going on.
That pissed me off.
Since this may take days to resolve completely, here's a temporary workaround for Chrome users -- launch your browser with this flag:
chrome --ignore-certificate-errors
You'll still know when sites have bad certificates due to the red line drawn through the https:// portion of the URL. But you will be able to access these sites. But be sure to stop using this flag as soon as you can. It could leak secure cookies to a MitM with a fake cert. Very slim odds of that, but still undesirable. Yet Chrome left us with no other option here.
I must say though, I'm increasingly frustrated by software vendors trying to strip away control over our own machines. There is no option at all from the standard error message, even under advanced, to indicate that you know about this problem and wish to proceed. And I'm sure it's only a matter of time before they remove this command-line option as well.
I get that novices probably need some protection, but I really wish there were a way to say that, "yes, I really do know what I'm doing, please stop treating me like a toddler." So instead, I'm forced to use a much less safe, hidden command-line option or be locked out of various sites for four whole days.
> I must say though, I'm increasingly frustrated by software vendors trying to strip away control over our own machines.
I get the sentiment, but if you look at how common malware is on Windows versus (walled-garden) platforms like iOS, it becomes pretty clear that it's the most effective way to keep users safe. Inconveniencing tech-savvy folk is a small price to pay for that, IMO.
> I get that novices probably need some protection, but I really wish there were a way to say that, "yes, I really do know what I'm doing, please stop treating me like a toddler."
Typing "badidea" when faced with the interstitial should do the trick.
But that's just it, I'd have to download the source to Chromium, find where this error is thrown, and add a UI element to allow me to bypass the error. That's ... a whole lot more than just inconveniencing me. Flipping a switch in chrome://flags would be an acceptable inconvenience.
This command-line switch works, but it requires accepting a larger risk with respect to secure cookies and MitM attacks.
This killed one of our main webapps, but since the site was hosted on AWS, I provisioned an ACM certificate (free) in about 5 minutes and manually applied it to the ELB listener. Couldn't have been easier.
Today's weirdness and communication around it has made me trust Globalsign a lot less now.
Revoking keys (and the necessary checking that it requires) will never work IMHO. The only way to solve this problem is short key signature lifetimes, automated signatures and, if compromised, just no re-signature.
Let's Encrypt is one way, although the lifetime with 3 months is a bit too long. One month or even less would be better. Additional verification and checks via DANE/DNSSEC help to shorten the impact of a compromised key. Constant checking for revocations do not. Again: IMHO.
Root CAs can't have 3 month validity. Any computer left offline for 3 months, or even 1 day (if coming up close to expiration) wouldn't be able to bootstrap an updated trust store.
> The only way to solve this problem is short key signature lifetimes, automated signatures and, if compromised, just no re-signature.
I wonder if Let's Encrypt is thinking ahead to a federated future where instead of issuing leaf certs, they also issue intermediate certs to domain owners, who can use those to mint very short-term leaf certificates (like 1-4h) in an automated fashion.
A 1 hour server certificate might work if all clients had the proper time set and all libraries were comparing the notAfter to the UTC current time.
Sadly, lots of phones and computers have incorrect time settings, off by one hour to work around incorrect DST settings is pretty common, off by several days isn't super rare either.
I have run into some SSL client libraries that check notBefore against the current time, in the current timezone, and therefore those in the western hemisphere would fail to work for several hours after a certificate was issued. I imagine they're using the same for notAfter, so those in the eastern hemisphere would stop working early.
My rule of thumb is a certificate that is less than 30 days old or has less than 30 days of validity left is going to cause problems for a subset of your users, and shouldn't be used unless you have a good reason (old certificate compromised would be a good reason).
This issue affected me as soon as I upgraded my chrome to V54 yesterday. It broke all my CDN hosted files which were using the AlphaSSL Wildcard certificate. We were experiencing low traffic and realized this may have been the issue. Got into Chat support with ssl2buy who provided me with a Comodo Wildcard certificate. It was a pain to recreate and install the certs everywhere. But we didn't want to lose any more traffic.
it looks like this is impacting sites hosted on google cloud using the load-balancer (you upload your cert, and I'm using a globalsign cert). I am getting 502 errors via mobile but via desktop it's fine.
anyone else use globalsign via google load balancer who can confirm?
I've been wanting to switch all our certificates to Let's Encrypt for almost one year. But, there was always something else which needed my attention more urgently.
So, I guess, thank you Global Sign for forcing me to finally make the switch!
Luckily I don't have many tenants yet: I replaced my wildcard AlphaSSL with a couple of Lets Encrypt certs to fill the 4 day gap. Four days of inaccessibility is just not acceptable.
Before this, the Technical Solutions Director tweeted solutions that did not work for end users, but highlighted a typical IT centric approach to problem resolution ("Works, What's the problem?") [1]
For anyone not already aware, check out Let's Encrypt. I am evaluating it for about 200 domains now in earnest after having it on my horizon for some time. At least to have it ready as a fallback. [2]
Getting 200 EV certificates in a hurry from a different CA has been costly this morning.
[1] - https://twitter.com/vanbroup/status/786548172864626690 [2] - https://letsencrypt.org/