I've often heard HNers state that Project Zero is unwilling/uninterested in protecting the interests of the vulnerable host, but this seems like an excellent case study where Github, the host, just doesn't think this is a big deal. I think that applies to any host who doesn't patch their vulnerabilities - if they thought it was a big deal, they'd do something about it. Actions speak.
Quoting from the post:
2020-07-21 Report sent to Github and triaged by Github security team. Disclosure date is set for 2020-10-18.
2020-08-13 Project Zero requests a status update.
2020-08-21 Github say that they are are still working on a fix and a deprecation plan
2020-10-01 Github issued an advisory[1] deprecating the vulnerable commands, assigned CVE-2020-15228, and asked users to patch and update their workflows.
2020-10-12 Project Zero reach out and proactively mentions that a grace period is available if Github wants more time to disable the vulnerable commands.
2020-10-16 Github requested the additional 14 day grace period, with the hope of disabling the vulnerable commands after 2020-10-19.
2020-10-16 Project Zero grants grace period, new disclosure date is 2020-11-02.
2020-10-28 Project Zero reaches out, noting the deadline expires next week. No response is received.
2020-10-30 Due to no response and the deadline closing in, Project Zero reaches out to other informal Github contacts. The response is that the issue is considered fixed and that we are clear to go public on 2020-11-02 as planned.
2020-11-01 Github responds and mentions that they won't be disabling the vulnerable commands by 2020-11-02. They request an additional 48 hours, not to fix the issue, but to notify customers and determine a "hard date" at some point in the future.
2020-11-02 Project Zero responds that there is no option to further extend the deadline as this is day 104 (90 days + 14 day grace extension) and that the disclosure will be today.
I mean, in this case what difference does the Project Zero disclosure make? I've been getting alerts about my Actions build, GitHub has released the functionality necessary to implement the change, and GitHub has had a website explaining the vulnerability for weeks now. This isn't a case where disclosure actually disclosed anything new.
> Github responds and mentions that they won't be disabling the vulnerable commands by 2020-11-02. They request an additional 48 hours, not to fix the issue, but to notify customers and determine a "hard date" at some point in the future.
The issue isn't really with GitHub not making the fix available, but the issue is that if they turn it off now they will break legions of builds. I think the Project Zero disclosure issue for them then is more for pointy haired manager appearances ("Oh no! Github has a disclosed Project Zero vuln!") than for any real difference in security.
If your choices are "disable all logins" or "anybody can log into my bank account and make whatever transfers they want", the correct choice is the former. (Obviously I would prefer a third option, where the company actually fixed the login bug sometime during the 104-day lead-up, but that's not the point.)
Seem a bit inflexible from Project Zero, given that Github has already disclosed it anyway. Why didnt they disclose earlier even? Seems like program part could need some updates, a few things don't make sense here.
> 2020-10-30 Due to no response and the deadline closing in, Project Zero reaches out to other informal Github contacts. The response is that the issue is considered fixed and that we are clear to go public on 2020-11-02 as planned.
> 2020-11-01 Github responds and mentions that they won't be disabling the vulnerable commands by 2020-11-02. They request an additional 48 hours, not to fix the issue, but to notify customers and determine a "hard date" at some point in the future.
Suggests to me that GitHub has no clue whether it's resolved or not. Project Zero were following GitHub's lead. Asked them if the problem was resolved or not, didn't get anything, had to reach out to internal contacts, and then got a conflicting message saying "NO WAIT!"
I think it kinda doesn't matter. Project Zero had already given them an extra 14 days; GH should have made better use of it. If you're constantly making exceptions to your rules for people, the rules end up being more flimsy and less respected.
Regardless, GH had already disclosed this a month before, so their desire for 2 more days was unnecessary: the cat was already out of the bag, through their own actions.
Indeed, if that timeline is to be believed—and I have no reason to doubt it—it’s pretty damning. GitHub just kept putting it off. When they asked to delay the disclosure indefinitely at the very last second, of course they were told that would be unacceptable.
I dunno. I've worked in many places that ABSOLUTELY had the desire, willingness, money, time, and ability to do something, and were still unable to make that thing happen, for reasons that can't be spoken about.
While it is true that if an entire organization wants something done that it will happen, it is not true that inaction by an organization always signifies unwillingness or inability to act.
It's easy to sit on the sidelines and throw tomatoes at the parties involved. It is another matter entirely when you are hamstrung by internal process or a specific power-tripping and ignorant person in management.
>While it is true that if an entire organization wants something done that it will happen, it is not true that inaction by an organization always signifies unwillingness or inability. It's easy to sit on the sidelines and throw tomatoes at the parties involved. It is another matter entirely when you are hamstrung by internal process or a specific power-tripping and ignorant person in management.
The thing is, major releases and public shaming is pretty much the only thing that has a (not guaranteed) chance of moving these processes or stubborn people. Before that they learn to just delay indefinately.
If this is being held by a single person (or small number of people), then the problem is really the organization. This should not happen with security issues.
> I agree. Just be careful when you assign motive to an entire organization when it could be just a single person inside being an ass.
Preventing motivations of a single person inside an organization from tanking its products and direction is one of the main jobs for organizational management. Good companies are resistant to fiefdom building and having managers damage their reputation for personal gain.
When people ascribe a motive to an organization nobody thinks that every single person in the organization has that motive. Rather what it means is that the organization's structure and members cause the organization to act in accordance with that motive.
Indeed. And whether it's one person, a group of people, or a departmental "culture", is meaningless. The org allows it to happen, CEO downward, through all the ranks.
It is entirely the corp/org's fault, and yes, that reputation can rub off on you.
I think you're right that this is often the case, and my heartfelt sympathies go to any engineer who struggles to do the right thing in that situation.
But in what sense are these kinds of internal politics NOT "unwillingness or inability", at least at the organizational level?
Organizations where managers are allowed to play political games at the expense of users' security are not to be trusted with any sensitive data, and need to be publicly called out and shamed if there's any hope to effect change.
> But in what sense are these kinds of internal politics NOT "unwillingness or inability", at least at the organizational level?
An example from my past should demonstrate...
A single manager in my past simultaneously:
A) told higher management that something was not possible, and
B) told the org under him that management didn't want that same thing done, and
C) convinced his peers to let him manage this personal pet project alone.
The only person that knew the truth at the time was the lone person who didn't want it done. Result: it didn't happen.
This is extraordinarily common, in my experience, though only this particular person could do it this well.
The entire organization wanted something done except one guy who convinced everyone to let him have the power to stop it without anyone understanding what they were letting him do.
You may argue that this is an organization being unwilling to act via its own structure, and I would counter that this person was very skilled at this type of deception, and succeeded at this type of thing despite the organization above, below, and around him all being pretty much standard.
Upper management's job is to manage. Being "convinced" of something, means they did their job, made a judgement and management call, but got it wrong.
That's their fault too.
It's also the fault of other employees, who did not speak up. You're painting a picture of everyone being incapable of judging things on their own, because someone talked them out of what they knew to be correct.
If that were the case, I feel like Github should have responded on 10-28, when Project Zero reached out mentioning that the deadline was approaching. If they asked for another extension then rather than just not responding, maybe Project Zero would have considered being more accommodating?
Even if there exists some really well intentioned hamstrung individuals within the organization, I think it's desirable to highlight and potentially shame institution as a whole that aren't able to effectively mobilize.
an organization is not a human, despite what people say.
A single human may be absolutely willing to act, but the organization won't - i.e. they will not be allowed to act by their bosses. Everything else is nitpicking and blame shifting. If someone's willing to do the job "but they have too many other things to do" then they're not willing to do the job. If someone's boss or with authority over them is unwilling to prioritize the work then they're unwilling to do the work as well.
You're assuming the goal of Project Zero is to take an interest or willingness to protect vulnerable hosts. That is not the goal of Google Project Zero. The goal is to find vulnerabilities that impact the stock price of Alphabet competitors, usually Microsoft or Apple.
The software of their competitors is used by billions of people, thus finding and reporting vulns in that software is going to help billions of people by fixing that problem.
If that was actually the case then, just by the numbers, wouldn't they spend all their time fuzzing Android or GSuit or their own codebases as well? Or at least release the reports of them with the same viritol?
It's like Toyota buying a Ford and finding out that if you rear end it, it will explode. So Toyota publishes their finding. Then weeks later a 3rd party finds the prius accelerates by itself.
Maybe if Toyota wasn't so busy trying to screw Ford they would have known about their own glaring issues.
We can't tell whether or not Google lives in a glass house because they keep it proprietary. Therefore we need to assume it's no better than what they bully others for. Nobody handed Google a crown and made them ruler of everyone's code. They just kinda built themselves a castle and started cutting off heads.
> I've often heard HNers state that Project Zero is unwilling/uninterested in protecting the interests of the vulnerable host
I think that's usually people that just vastly prioritize the interests of the vulnerable host compared to the interests of all the users.
To my eyes it's obvious there's a trade-off to be made between protecting the company and protecting the people affected. Too little time is detrimental to one party (sometimes both to some degree), while too much time is definitely detrimental to the other. It's a fine line to walk, but the best thing you can do is be consistent. Nobody is served well if the companies/projects in question work under the assumption that more time will be given because it has to others in the past, and then it isn't, as the company may not have correctly prioritized the fix, and then the public is left vulnerable as well. But to not be consistent leads to abuses of the system where things just don't get fixed.
At this point, I don't think anyone can accuse Project Zero of being inconsistent, and 90 days is a long time to get something fixed if you put the resources towards it that it needs. I have little sympathy for a company that mismanages this process at this point. For an open source project, there's always groups and lists you can go to and ask for help if it seems overwhelming for the project you have. Presumably if it's an important enough project some person or company that cares about it will donate some time. If nobody is willing then my guess is that the project's not that important to people.
There's also the possibility that the problem is so large or so fundamental that fixing it is a herculean task. Maybe in that case people are better off moving to an alternative if they care about the problem. Sometimes things are so bad the best choice is just to jump ship. It sucks for that company or project, but they have no right to my usage, but I do have a right to choose what I want.
I’ve sent dozens of support tickets over the decade, and can’t recall ever not receiving a response, including on minor feature requests or bug reports (e.g. once there was this minor UI issue where scrolling causes certain UI elements to jump around on iOS Safari, or something like that). At the very least I would get an acknowledgement that the issue has been filed with the appropriate team.
We must live in alternate universes, or somehow my nothing special personal account (free at first, $7/mo since 2015 or so) is prioritized, or somehow your account is deprioritized.
I have plenty of times actually. I've given lots of feedback for the github android app and gotten great responses. And most of the user reports (spam users etc) get dealt with within a couple days.
Quoting from the post: