Obviously, this is a very disappointing situation for us -- we've always taken security very seriously since day 1, it's something that's been core to who we are from the beginning.
That said, how you respond in this situation can be just as important, and so we are making sure to be incredibly proactive in addressing the situation & transparent in how we communicate the details with our customers. Our top and immediate concern has been our users and the safety of their accounts.
A few days ago we became aware that an unauthorized party obtained email addresses/usernames, last login IP addresses and bcrypt hashed passwords for a large number of customers (anyone who signed up prior to March 1 of this year).
At this point we do not have evidence of any customer website/account being improperly accessed. It's also worth noting that we do not store any full credit card numbers on Weebly servers, so any credit card information was not part of this incident.
We immediately starting working on taking steps to notify our customers, and were able to get this out in a matter of a few days. We're initiating password resets as of this morning, and we've also made several improvements to the application including new password complexity requirements and a new dashboard that gives customers an overview of recent log-in history of their Weebly account to track account activity. We also increased our bcrypt work factor from 8 to 10, and all passwords will be automatically upgraded as of the next time a user logs in.
We've hired an incident response firm who is working with our internal team to complete a full investigation. In the meantime, we're examining our stack top to bottom and taking many steps to enhance our network and application security. This is an area we take very seriously and we'll be putting in tremendous effort to ensure this doesn't happen again.
This title is incorrect. Credentials not stolen, usernames and hashed passwords stolen. That is not the same as having everyone's password. The title implies someone can easily log in to your account.
This is also not true, as if they used bcrypt (a key derivation function) the hash is salted, so even users using common passwords are protected against rainbow (lookup table) attacks.
as for brute force, yes attackers now know usernames, so can try brute forcing the live sites, or brute forcing each user hash.
I believe he meant the will try the top 100 most common passwords on each account on the website directly, resulting on "82% of users at risk", assuming 82% of users use one of these 100 passwords.
Strong brute-force protection (eg block account for exponential times) could mitigate this attack vector.
Why are you guys talking about live site and rainbow table???
The attackers have the salts and the hashes, they can brute force the hashes offline with [ocl]hashcat as they wish.
Top 100 passwords * 43M accounts is only ~4B hashes to compute. We don't know what bcrypt parameters they used but we're probably talking a few hours here, maybe only a few minutes.
to brute force the top 100 passwords, only the usernames were really required. can easily bruteforce the top 100 passwords on a live site.
you are right that it's now very easy to use dictionary attacks now, on all the credentials offline. and those with super weak passwords will have their accounts compromised.
Just tried updating my password for my weebly account. It appears this functionality is broken. It keeps telling me that "Your current password must be correct". I logged in and out with the same password a couple times to confirm I'm not crazy.
Can you email me at david@weebly.com? We looked into this shortly after you posted this comment and can't replicate (also not receiving other complaints)... If you can email me we'll get to the bottom of it -- thanks for letting us know!
Why does it take a few days to send an email saying "we got hacked, please change your passwords promptly".
I accept there are costs to jumping the gun and passing out incomplete information, but if I screw up I tell the affected parties that day. Not after a few days of planning how to manage the message.
For one, it's pretty challenging to send out 43M similarly-looking emails within 24h without tripping a whole bunch of anti-spam filters - even when you're using tailored services (that distribute over IPs with good rep, etc) like Amazon SES.
This is silly – the entire process of "Oh, we got hacked. What did they access? Who's been compromised? Better write an email. Better find a channel to send that email, and wait for it to be sent" could very, very obviously take a couple of days. You are unfairly trivialising that.
My point was transactional emails does not hit any spam filters. And since email services tracks how many of emails were opened, i am pretty sure 'we have been hacked' will be opened quite often and will never be in spam folder.
From the point of view of a spam filter, there isn't really any difference between a "transactional" (what a horrible name, but not your fault that the industry has adopted it) email and any other email.
"Transactional" emails simply have some distinctive elements, such as the first and last name of the customer, which make them less likely to be filtered out.
First & last name in email alone doesn't guarantee to hit inbox. Huge free email services use social 'signals' as open rates, 'this is spam', 'this is not spam' as well as IP reputation i mentioned earlier to determinate if email hit inbox.
So, there is obviously a difference in those metrics between simple notification 'your account have been hacked. change password' and 'hey, we haven't seen you for ages'. As email system notice high user involvement it will never ever block such emails. Actually, i think it will increase IP reputation.
"transactional emails" is a term marketers use, but in reality the email systems do not know the difference. And yes we did get stopped by various ISPs, and yes we did have to get some on the phone.
As I understand, transactional email is often triggered by a user itself (new account creation, forgot password, etc). By definition, those are sent spread out over time.
This is - at least from a spam filter's perspective - a huge email broadcast, more akin to a news letter mailing or a spam run.
And semantics aside, a lot of those 43M users will consider the email to be unsolicited (didn't remember they signed up, don't care about computer security, etc). They will happily report such an email as spam, adding to the training set.
In a perfect world companies would recognize and react to security breaches almost as soon as they happen. But if you have ever managed the logging pipeline or incident response practice for a company, you understand that this is deeply unrealistic.
There is virtually no company which discovers that it has been breached within a short period of time - the nature of a security breach is such that it doesn't generally become apparent until some time later. This pattern continually plays itself out with just about every large breach you can think of.
In that respect, considering Weebly actually hashed their passwords with bcrypt and is reacting to the breach in the same year, they're fairly far ahead of the curve on this one.
"we've always taken security very seriously since day 1"
No you have not because then this would not have happened. The only one who should be able to query passwords from the database should be the DBA. Everyone else should only be able to validate against it. So either it's an inside job by your DBA, or you thought your users security was less important then avoiding the friction such high security standards would have introduced in your workflow.
Security is hard. It is very possible to take it seriously, do many things right (perhaps everything right, insofar as it's in your power), and still have your company end up in a headline like this.
You can parameterize your queries until you're blue in the face, but that won't help you if the right employee is phished (for example). This is an inherently imperfect and chaotic world, and it's unrealistic to assume that you're insulated from these scenarios just because you locked down database access correctly.
Personally, I believe David when he says Weebly takes security very seriously.
I'm kind of tired of the "Security is hard, every one gets hacked eventually and we are just victims" mentality. This is not true. Why don't we see peoples banking information plastered over the web every month? That if something would be a high value target. No, it's always these Web 2.0 services this happens to. Now, you could argue that a small SAAS service can not possibly afford security as rigorous as a bank, but guess what, if you are going to handle peoples information, and don't have the assets to protect it, then maybe your business is not viable enough?
I invite you to research this topic more thoroughly.
First, while there is a recent uptick in breaches, newsworthy ones do not happen every month. There does appear to be something of a clustering effect, which I think is attributable to a number of different causes. [1]
Second, banks, even very large ones like Citigroup and Chase, have been compromised in recent memory. [2] Even the IRS suffered one of the largest breaches ever, just last year. Peripherally "financial" institutions that aren't banks have also suffered breaches, such as every single credit card processor and NASDAQ.
You have a right to be upset about the increasing probability of your passwords being compromised by third parties. As a consumer, you can mitigate the damage of such breaches by 1. using a password manager, 2. using a different password for each and every account you have and 3. generating extremely secure passwords for each account. You can also use services like HaveIBeenPwned [3] to stay ahead of the damage.
However, your indictment here is unreasonable. Like basically everyone else in this thread, you don't have much information to go on yet. Weebly properly hashed and stored their passwords. As far as breaches go, this one is pretty tame. They are reacting responsibly and quickly considering the breach happened this year - normally we'd find out about this in three years. We do not yet know the root cause of the attack, and the criticism you're levying against Weebly is equally applicable to the industries you believe are more safe (they aren't). While many "web 2.0" companies may be rather lax in security, Weebly did not do anything obviously wrong or negligent here.
________________________
1. As data breaches become more of a hot topic, they will be more likely to be reported widely because it guarantees eyeballs. Similarly, it increases scrutiny, which aids in discoverability, and leads to more copycat hackers attempting these breaches for fame or fortune.
They kept the usernames in the clear. It is possible to create a service that never stores usernames but only hashes of them too. Keep an email linked to that account for a reset of the username / password combo.
Then breaches only reveal emails and a pair of hashes, so to control the account you need to control the email.
It is possible to do any number of weird things that practically nobody in the whole world, including companies with the best security teams in the world, actually do. Tokenizing email address is indeed one of those weird things.
So true. They also have government agencies on their side. If you hack a bank, you're messing with money which suddenly involves a whole raft of governmental agencies.
That's ridiculous; older businesses and government services are compromised all the time!
And let's not forget that there is a spectrum of value associated with information. On the one hand, I'd rather my bank details and payments weren't publicly released. On the other… IP address, bcrypted password and email address? Minimal relative value.
The reason this attack became known as the swift hack is because the hackers were able to send messages over the swift network to transfer money around, however this how the system is supposed to operate, it's what the swift network does.
What was hacked was the bank where the messages were sent from.
If you read the article (any of the articles) the headlines always talk about the 'swift hack' however it was the _banks_ that were hacked (and the article says so), not swift.
Seriously dude? People fuck up all the time, it doesn't mean the company doesn't take security seriously. Furthermore taking something seriously and being successful at your serious undertaking are different things.
Responsible disclosure and proper handling of passwords as well as not storing credit cards. Barring no breach at all, this is about as well as something like this can go.
I don't doubt that they're clueful. That's why I'm confident that they hired an excellent incident response team and wisely chose to have them review and edit any external communication. Anything less would be irresponsible.
Weebly's team made the conscious decision to use security protections like bcrypt hashing on passwords, and they've been doing that for a long time (years). They're well aware of security best practices, and that did not come out of work with an incident response team for discovery.
Anyone who tells you they're "encrypting" email addresses in their SaaS app is almost certainly describing a cosmetic security feature.
Every day of the week and four times and Tuesday I'd prefer the team that spends their marginal dollar on finding the next marginal reflected XSS bug than the one that wastes it on "two-way encryption of email addresses".
I don't know what Weebly does for appsec (I've never worked with them and probably never will), but if they've spent even $50 on external appsec testing, they're 1000% better than 90% of rest of the applications we all use every day.
In terms of appsec, we run quarterly black box pen tests and annual comprehensive white box pen tests with well regarded firms, and have been rotating vendors on a regular basis for diversity. We also do a lot of stuff internally, like regular scanning, and internal sprints focused on vuln detection. We've been doing this for years. That's not to say we're perfect (we clearly are not) but we do take it seriously.
Every time when this happens I ask myself only one question.
What about all those hacked servers that we don't know that are hacked yet?
There are ( and I'm pretty sure ) lots of hackers that do this on a daily basis, but don't try to do anything malicious on a large scale ( like dumping the whole db of customers, DDoS, etc. ). They probably target medium-large or small companies' servers, put a backdoor there and analyze. Either stealing some business secrets or leave it like that for one of the dark days when some political-corporate person will need their help.
Having the whole human knowledge on the palm of my hand made also our own lives public-knowledge.
Also in this instance, Weebly, they get an anonymous "hey look, I have all of your data".
So Weebly issues a statement to their customers to reset their passwords (which the hackers knew would be a byproduct) and unbeknownst to them the hackers are now skimming the new passwords off the network.
I really like Google's Recent Log in Activity and Location. So anyone logging into your account from a Different location, you are automatically notified. But one of the problem with this is that once hacked, it exposed your location as well.
2nd thing is 2FA. I hope 2FA becomes the standard for all login. Even SMS. ( I know SMS is not save in US, but I am not sure if similar can be said in EU or Japan )
I wonder if the hacker really interested in decoding credential or they just want to collect the email addresses which is really valuable for email marketing.
I have talked to a number of current and former Weebly employees trying to convince them to use things like hardware token based 2FA, hardened servers, hardened workstations, and strong end to end encrypted password management that can't be trivially decrypted from a private key stolen from memory. I had such things written off as being too paranoid when they are too easy -not- to set up.
I was not at all shocked by this headline.
I don't want to just single out Weebly here as I discuss these sorts of things with people at different companies all over the bay out of personal interest and anything harder than using something like lastpass to reach production systems is considered too much work. Honestly Google and Facebook are the only large companies I have seen deploy fairly decent security practices out of the dozens I have exposure to. I credit this to the fact the employ teams people who have the specific job of continually auditing and enforcing all available security tools on their systems and fostering a culture that security is everyone's job.
You will pay for security either way. Either up front paying teams of capable people, or in lost customer trust after the fact.
Security apathy in the valley is a cancer impacting companies of all sizes. Sure you can't make anything perfectly secure, but you can at least force your attacker to burn a 0day. Don't make it as easy as spoofing an email and getting an employee to click a malicious link.
If you have any sort if privileged access to PII data of your customers and are not even doing basics like using hardware tokens to gate your server and db access you are one keykogger or XSS away from a serious breach. If you know how to set such things up and still don't do it, you are additionally a terrible person.
At the very least the data required to readily plaintext the passwords is not public in this case which is a lot better off than companies using only simple hashing like md5. Some credit is due here for sure, but I can't help but strongly suspect the issues here and in now countless other orgs are a result of people having access to PII that don't really care about security or respect the privacy of the user data they are responsible for.
I have no idea if that was what caused this particular incident, but there was a lot of exposed surface on the table in that area making an attackers job that much easier via those vectors. Employee workstation or credential compromise is one of the most common ways internal assets are stolen. When easy to implement measures are not taken like using hardware tokens to gate access... then I am left to suspect if any of the harder things were done.
As for making an ad for my "services". My company does not provide security services and I am not looking for a job in this space. Pretty happy where I am. I gain nothing from posting this but to promote discussion I feel is important for our industry.
I do however participate in a not-for-profit community I have funded mostly out of pocket for the last 15 years for helping teach better system admin and security practices. If you can even find it, and want to call this an ad for that ... uh, sure.
Well, the breach was back in February this year, so I hope they have put better security in place since then. I've seen ads for Weebly all over the place, but never realised they had that many users. Good luck with your application... :)
accounts doesn't mean the same thing as current or active users. a company very likely doesn't delete accounts right away or at all, even if the service has been cancelled. for example, a few years ago microsoft's live ID system had well over a billion accounts. but MAU was only around 450M. and that is with culling / deleting accounts after a year if they were unused.
While I agree the two are different, I also think in weeks case accounts could translate into multiple users. I know for a fact that my wife and her sister both share a weebly account for their business and I believe they have to share a login to do this.
Perhaps I should've gone in more detail. I'm not upset about them wanting people who are proficient in git, I'm saddened that the job listing doesn't mention anything about what a Senior Android Developer does at Weebly.
The only Android related things on the job listing are proficiency with the IDE (Android Studio) and generic "frameworks".
I am an Android engineer, I clicked this job listing earlier today when it was on HN because I was interested in it. From the perspective of a listing that tries to get a good funnel of candidates coming in, it does nothing for me.
Would you apply to that job listing? I spend a lot of my day to day helping companies optimize their job listings, especially when they have high view numbers but low application click through rates, so this is a pet peeve.
Basically I like job listings that include _specific_ information about the role at the company. Maybe that means listing some of the hard challenges the other people on the team have worked on recently, maybe it just means listing the frameworks and libraries they're using, but anything to help me differentiate their Android role from someone else's Android role.
Great, thanks! The Android team is 2 people currently, so roles in such teams tend to be like roles in much smaller companies, but with a bit more infrastructure and process. I'll pass this around.
How long would you estimate it takes someone to "properly learn version control"? When I work with very junior developers they pick up git quite well in a week or two at most. Beyond that almost every team I work with uses their own special homegrown git workflow anyway.
I don't mean to put you down for you requirements but git is really really easy to learn. It would be a shame to miss out on a talented developer for something an otherwise well qualified developer could learn in a weekend.
I'd be very surprised if asking people to waste a week of vacation on only a 66% chance of getting a job (that they probably don't even need, if they're already employed) wasn't a strong filter against top performers.
> I tell people that the worst case scenario is that they use a week of vacation, but because of the extra pay they can take a nicer vacation later on.
I mean, how can they, if they've already used their vacation time?
I am not trying to start a flame war, just curious why the 2 jokes about Russian hackers being the origin of breaches are being heavily downvoted. I would assume people think they aren't clever or funny, but hopefully not because they disagree that Russia seems to be the catch-all scapegoat right now.
It doesn't matter if people think they're clever or funny, a core tenet of HN is that you downvote cute jokes so that they don't eat all the real discussion like they do on Reddit.
It is a semi-noob illusion that satire and irony-- which underscore & catalyse discussion about relevant and important issues; should be reserved for Reddit. Often, satire is used when it may be difficult to confront an issue directly/the issue itself (or discussion) could be unpopular.
That said, how you respond in this situation can be just as important, and so we are making sure to be incredibly proactive in addressing the situation & transparent in how we communicate the details with our customers. Our top and immediate concern has been our users and the safety of their accounts.
A few days ago we became aware that an unauthorized party obtained email addresses/usernames, last login IP addresses and bcrypt hashed passwords for a large number of customers (anyone who signed up prior to March 1 of this year).
At this point we do not have evidence of any customer website/account being improperly accessed. It's also worth noting that we do not store any full credit card numbers on Weebly servers, so any credit card information was not part of this incident.
We immediately starting working on taking steps to notify our customers, and were able to get this out in a matter of a few days. We're initiating password resets as of this morning, and we've also made several improvements to the application including new password complexity requirements and a new dashboard that gives customers an overview of recent log-in history of their Weebly account to track account activity. We also increased our bcrypt work factor from 8 to 10, and all passwords will be automatically upgraded as of the next time a user logs in.
We've hired an incident response firm who is working with our internal team to complete a full investigation. In the meantime, we're examining our stack top to bottom and taking many steps to enhance our network and application security. This is an area we take very seriously and we'll be putting in tremendous effort to ensure this doesn't happen again.