After being bitten the first time with Linode I don't care what technical measures they are taking. I want to know what process and policy changes have been made.
Do they still store public/private keys on the same server ? How often are they doing security audits (which clearly never happened before) ? Are they still going to be dodgy and withhold key information from their users ? Are users still going to find out hackings from IRC/Reddit rather than Linode itself ?
Two factor authentication would have done NOTHING to prevent both hacking attempts.
I also find it really troubling they haven't released a "Here's what we're doing different" blog post in response to the attack. Their only blog post on the matter came a week (2 weeks?) after the intrusion, which they were of course pressured to release after everyone found out via a pastebin IRC transcript... By chance I happened to sign up for my first Linode account the day before that hit HN.
I hope their silence on the aftermath is due to an ongoing investigation with feds, or something, where they can't talk about it yet. Do they think their customers are stupid and will forget the incident?
Imagine if AWS had a security breach of that magnitude. They would release an initial 4000 word blog post in grave technical detail, and then follow up with a 25 page white paper, or whatever.
Oh, and to stay on topic, I tried Linode's 2-factor with Google Authenticator and it works well.
Do they think their customers are stupid and will forget the incident?
Yes. They have done it before and people on here still recommend them with a straight face. It honestly confuses me that people care so little about security.
"Do they still store public/private keys on the same server?"
As phrased, this is not a problem - there's never any worry to including your public key wherever you have your private key; your attacker can be assumed to have your public key anyway if it'll do them any good.
The problem was private keys (encrypting important things!) on a web-accessible server, was my understanding.
Well - it is not likely you will need you public key on the non-web-accessible server. In this type of application the public key is needed in the place that encryption happens and the private key is needed where decryption happens. If the two are on the same machine it likely means you messed up.
You're assuming a single piece of information has a single key-pair. E.g.:
1. Obtain sensitive information
2. Generate a new key-pair
3. Encrypt with public key
4. Store encrypted info
5. Delete public key
6. Use private key to decrypt when reading the data
It's also likely they they were using one key-pair to encrypt all of their data (or all of a specific type, e.g. one key-pair to encrypt all passwords). In this case, the public key would be needed to encrypt new data coming in.
Nice, but has nothing to do with the issues they experienced recently: Still runs on cold fusion, still they do not understand PKI( more tweets about how awesome the passphrase is on your private key, you know the one in adversarial hands...confidence + 10!....)
Of course the 2FA wouldn't prevent it from being hacked, but that's not the point of it. The point is that even if someone gets the password and cracks it, it's still useless as the attacker doesn't have the other factor.
I left Linode after 5 years of being a customer because I can no longer trust them. I let the first issue slide as I thought they would learn and communicate better to their customer base but the second incident has shown they learned nothing.
Security issues will happen with any provider it is all in how a provider communicates and remediates those issues. Linode has shown it will not communicate thoroughly and does not talk about any remediation so why would you trust a company like that with your data?
The last incident was extremely sad for me because I thought I was using a company that I had a good relationship with. I could care less that CC details were lost as CCs are easily replaceable and protected against fraudulent use. What they lost was my trust which is far more valuable that my credit card number.
Out of curiosity, who did you switch to? I would leave too (after being a customer for nearly 8 years) but I'm having trouble finding other providers which don't have their own set of issues.
I have been testing/working with a few different providers. The three I'm currently working with the most are Ramnode, Gigenet and DigitalOcean.
Ramnode's panel is SolusVM which isn't as good as Linode but their performance blows Linode out of the water. They have ipv4/ipv6, multiple locations (Atlanta and Seattle) and a good owner who seems very open/honest with customers. I expect we'll see feature enhancements as they grow bigger.
Gigenet Cloud has multiple locations (Chicago and Los Angeles), ipv4/ipv6, good performance, good custom panel and a company that has been around for a long time. They use a SAN for all their nodes. Overall one of the most underrated cloud providers out there. (Note: I got free credits for beta testing their cloud)
DigitalOcean has multiple locations (San Francisco, New York, Amsterdam), a decent custom panel (would like to see more statistics and it seems a good staff. They did have a security issues that they seemed very open about (https://www.digitalocean.com/blog_posts/resolved-lvm-data-is...)
Other hosts I have tested/used but did not choose:
Rackspace - Excellent panel, ho-hum support/performance. My biggest issue is they lock instance throughput and refuse to change that. If you have a 512 instance you are locked to 20 mbit which doesn't make sense as you are billed per GB. I asked to have this unlocked as my instances push more and they refused.
Amazon AWS - Great interface but the lack of ipv6 (unless you buy ELB) and poor performance had me look elsewhere.
Others tested/used:
Joyentcloud, Terremark, Zerigo (Was a long time customer but they went downhill when 8x8 bought them), Voxcloud, Cloudsigma , Azure, HP Cloud, Stormondemand (Another good cloud provider that just didn't fit with me), VPS.net, Gandi
Great post. I'm interested to know which you end up ultimately choosing. I just worry that ramnode and digital ocean won't be able to keep their current price model and still maintain quality service in the years to come.
I'd really like to know this as well. (and I'm writing this post instead of just upvoting to hopefully encourage the grandparent poster by showing him that more than one person would like to know of the alternatives out there for switching)
I find all the whining of all these armchair security experts a bit wearing. No one outside of Linode and the alleged hacker knows exactly why and how Linode was hacked so quit speculating, you are simply making things up. "Ooh, this is a bad thing, I bet Linode did this bad thing". For example people are suggesting that 2FA is useless if outsiders had free run of Linode's infrastructure, and so it would be. BUT, is there any evidence whatsoever that this was the case? If not then STFU and stop spreading lies. 2FA is useless against a nuclear strike, what exactly is the point of saying so? Anyone can fantasise disasters.
To get some positive content out of this thread. Is there a VM provider with a provably better security record than Linode?
If you are going to stay with Linode then 2FA seems like a no brainer. So, is there a simple way to get the 2FA iDevice systems (Google, Duo) to work on multiple devices, say to allow an iPad or an iPhone to be used interchangeably?
It would be nice if they let you set up SSL cert + MFA + password. I am kind of angry that modern desktop browsers continue to make SSL certs suck so much, but they're decent on mobile. I hope a future version of OSX builds in great cert management and UI/UX with local biometrics or something.
Not everybody owns or wants a smartphone. Linode needs to extend this to some non-smartphone device, like YubiKey, or offer SMS codes, like Google Authenticator. This is a step in the right direction, but is ultimately disappointing for me.
All they need to do is let users locally generate (i.e. in the token) seeds and then enter those into the web portal, vs. generating seeds internal to the portal, displaying them, and having the user enter them into the authenticator app or token (because the tokens don't allow you to enter a seed).
I have to imagine the overlap between Linode customers and smart phone owners was so large (and the cost of implementation so low) that leaving out hardware authenticators makes sense for v1.
One area where hardware authenticators work really well is where you want to split access to an account, or have some accountable/logged procedure for it. You put the physical token in an envelope and in a safe/put it in the control of a finance person. Tech people have the password, but need to request the token to do logins.
This also requires having role accounts which aren't able to reset authentication settings when logged in, though, to really be good (or else you just disable tokens on first successful login).
Also works well for paranoid people who don't trust their phone, or people who log in only from a phone/tablet and thus where MFA is really one-device-authentication.
I see what you mean about losing the phone, but unless you're saving your password locally it still satisfies the old "Something you have, and something you know" rule. If you lose your phone, the attacker won't know your password. And an attacker without your phone won't have your OTP.
These physically secure OTP techniques are interesting, but shouldn't you have accountability at the system level anyways? If everyone has a two-factor device and a password, it's pretty tough to plausibly deny that you logged into a server. Someone would have to guessed your password and stolen your device.
So because the majority of users have smartphones, the option for everyone else is "too bad"?
I've had my non-smartphone for six years now. It still works, and while I'm sure I'll upgrade to a smartphone one day, I have no urgent desire to do so.
Is it really that hard to set up an SMS system as a fallback? I'm still able to use two-factor on my Google account because they offer this solution.
I was hoping for Yubikey support. But I'll take this for now.
I'll have to see if the Google Authenticator app shows up on all of my iDevices linked to my Apple account and whether the code from any of them will work (from the setup process, I don't see why not). Does anybody know?
If the app will work from any of iDevices, it would not be secure enough for a service storing bitcoins :) because the second factor should be hard to copy (which a real hardware token is, while a software token isn't).
it would not be secure enough for a service storing bitcoins
Linode was hacked twice (once where Bitcoins were stolen) in recent times and was shown to have the worst security practices I've ever seen. They have never been secure enough for storing Bitcoins.
What you were doing is the equivalent of living your wallet in a public place unattended, and then shouting and screaming it got stolen. You are putting your bitcoin wallet on a public accessible server, you should know the risks of this by now.
Don't leave your wallet in a public place unattended, that includes your bitcoin wallet.
Let me guess, you didn't bother to encrypt your wallet either, didn't you?
Don't blame others for lack of security, if you can't even figure out your own security best practices...
It's per-device, not per account (I know the guy who developed it for Google; one of the smarter people in the industry).
It uses protected storage for the credential so it isn't backed up to iCloud, either. Sadly on Android they don't have the same security features available, due to limitations in the OS; it would be fun to talk to Samsung and make a "actually secure Google Authenticator" specific to the S3/S4 since they have a security element.
If you do want it on multiple iDevices, you need to do that at setup time, by copying the secret manually.
The Duo-Security people, who have an Android Token claim to use the secure element in NFC enabled phones. It is a TOTP token and can be used just like the Google Authenticator. You don't have to use Duo-Security's system to use it (though there system is worth looking at if you are rolling out your own authentication system).
URL? I don't see anything about their android "duo push" or "duo mobile" client supporting the secure element, but their website is designed around the kind of people who buy $3/mo authentication systems (enterprise, not saas developers).
Each copy of the software needs to be initialized with a token. Google tries to limit you to have one copy initialized at a time, but I'm not too sure how effective they are.
Is this an iDevice limitation? The android version doesn't connect to Google's servers at all, so there would be no way for them to know you've setup multiple copies.
Generally it is because sites only show you the seed once; you can't get them to give you the seed again. You can just write down the seed or enter it into multiple devices if you know this when you set it up initially, though.
The only to prevent it, is to implement the two-facotr authentication on protocol-level instead of application-level. This brings you to smartcard-authentication, or VPN solutions (using smartcard or RSA tokens)
But if they would implement that, everybody will start screaming that they have to pay for a smartcard or rsa-token...
Be honest here, how many of you would actually want to pay for that?
Ok, so they get hacked and passwords are stolen and those are cracked. Guess what? They're useless. With 2FA, the attackers still won't be able to get in.
No, if you hack the portal, you can get whatever info it mediates; the attackers don't need to then use the passwords and 2fa tokens to log in to get it, they just bypass authentication entirely.
It's totally reasonable to believe linode is enough of a clusterfuck internally, based on past performance that this kind of thing is plausible. Yes, this protects you from one kind of attack if an attacker only gets limited access to linode's systems.
The other issue is it doesn't protect you from password reuse. If a user is dumb and uses his global password for his linode password, and linode is hacked again, and the password is recovered, the attacker uses that userid/password/email/etc. to attack other accounts of that user at other services.
They admitted that the encrypted CC numbers were leaked, they didn't mention if the encryption keys were stored on the same machine. The alleged hacker said that the encryption keys were stored on the same machine, making the encryption useless.
Linode confirmed that the private encryption key was stored on the same machine. They've been parroting lines about the password on the private key being too strong to crack.
"which was not stored on the machine", like they should be commended ( Reminds me of exams where you received some credit for including your name... ).
I am sorry, them confirming this fact, and even if I recall adding a smiley in the tweet they did it, just cemented that they do not understand their business.
They clearly wish to give the impression that they are "secure". They need more lock icons...they are almost as effective as the racing stickers on my car!
The real problem here is that PCI certification is an absolute joke.
There should be several classes of certification, from "I want to sell a few pet rocks" to "I'm Apple with 150,000,000 credit cards on file". Right now there's basically two.
This isn't proof of anything, but a few days after this incident the CC I use for Linode got a fraudulent charge, the first such in years. I cancelled the card, so no big deal, but this makes me strongly suspect that the attacker ended up with actual card numbers, regardless of the passphrase.
just a warning: I just enabled it and it wasn't working with my account & google authenticator for android. I had to call customer support in order to disable the feature so I could login into my account again.
Regardless of Linode's previous screwups, I welcome this change and I hope more hosting companies to offer 2-factor authentication soon.
For anyone installing the Linode's recommended Windows App "Authenticator", WARNING, it does not work! I was locked out! I then used the Microsoft's Authenticator app to find the right token.
Do not logout without verifying it works first in an incognito mode. Better yet, save the secret key temporarily to your PC.
Doesn't work for the Linode Manager iPhone app.
After enabling 2FA on Linode, the iPhone app still uses only user/pass.....
(app last updated, jan 17th, 2011)
Amazon AWS has the best security of any virtual server provider I've seen, by miles. There might be specialty providers (e.g. FireHost) which are good dedicated server offerings, too, but I haven't evaluated them -- it usually is "AWS, is it good enough?" and then if no, directly to a cage, do not pass go, do not collect $50k.
AWS also has the best first and second derivative on everything related to product; they were essentially crippled crap in 2006, and have turned into a viable option over the past years, without slowing down. Compared to the level of innovation in colo/dedicated hosting (~zero per year) and openstack, AWS is amazing.
It's still inferior to a good on-premises or colocated environment (mainly due to technical limitations in the virtualized environment; AWS's policy is top-notch commercial standard), but that may not matter for you. AWS pricing and performance is also worse in a lot of ways than dedicated hardware, but may also not matter to you.
A lot of the big cloud/dedicated hosting companies have decent security (SoftLayer, Rackspace), but aren't as good at AWS at policy or technical security. The sketchy VPS providers are miles below the middling standard set by companies like Rackspace.
Linode is solidly in the "sketchy VPS provider" realm. A bit better for availability, and not likely to actually be attacking you themselves, but not a responsible choice for anyone who cares about security from everything I've seen.
PaaS, in practice, is also a good solution if you care about security but have no skills or budget. While Heroku has its own set of problems around price, performance, and availability, it's more secure out of the box than a badly configured/maintained AWS deployment of your own, or a badly configured on-premises/colocated cage or dedicated servers.
How about for those that just want a small instance system. I just do the smallest linode setup for some personal projects. From my understanding, AWS is expensive for that type of use case?
I second that question, and expand it a little. AWS is likely a more secure and trustworthy option given the recent controversy, but it really is comparing apples to oranges; they're both hosting solutions (fruit), but fundamentally very different.
Is there any service close to Linode that provides a similar service for a similar price? I know of DigitalOcean, but their CPUs are very underpowered in each plan compared to Linode's (especially after recent upgrades).
Their transparency hasn't been awful; it should be better. I am for now giving the benefit of the doubt that they'll be releasing further details as they become more certain of them and/or investigations conclude.
Is this likely to actually fix anything? Were the past intrusions via the manager? Or via a compromise of the login to the manager via individual user accounts?
Or is this just a show? Either way, this question itself reflects the fact that they refuse to give proper information and postmortems.
Do they still store public/private keys on the same server ? How often are they doing security audits (which clearly never happened before) ? Are they still going to be dodgy and withhold key information from their users ? Are users still going to find out hackings from IRC/Reddit rather than Linode itself ?
Two factor authentication would have done NOTHING to prevent both hacking attempts.