The piece the author is missing, and why zendesk likely ignored this is impact, and it's something I continually see submissions lacking. As a researcher, if you can't demonstrate impact of your vulnerability, then it looks like just another bug. A public program like zendesk is going to be swamped with reports, and they're using hackerone triagers to augment that volume. The triage system reads through a lot of reports - without clear impact, lots of vulnerabilities look like "just another bug". Notice that Zendesk took notice once mondev was able to escalate to an ATO[1]. That's impact, and that gets noticed!
Yes. But respectfully (residual frustration at zendesk might make me curt here) if their security triage team can't see how dangerous it is for an attacker to get access to an arbitrary thread on a their CLIENT's corporate email chains (in this world of email logins and SSO), then they have a big lapse in security culture, no?
Yes, the researcher could have tee'd himself up better, but this says way more about zendesk than it does about the 15-year-old researcher.
Unauthorized read access to private emails you were never legitimately CCed on already is impact. It should not be necessary to come up with a further exploit daisy chained on top of that in order to be taken seriously. (Otherwise why stop at Slack access? Why is that automatically "impact" if email access isn't?)
The dude demonstrated the ability to infiltrate a client’s Slack instance via their vulnerability. If that’s not enough to make the hairs on your neck stand on end as an engineer, go fucking do something else.
The researcher showed how they could hop onto any Zendesk support ticket thread with zero authentication, so that should have been enough given Zendesk was exposing customer data via that attack path.
Clearly Zendesk needs to change things so that the email address that is created for a ticket isn’t guessable.
Exploit or no, the bug and potential impact are the same. I personally find it a waste of time to sink evenings into an exploit when they're going to fix the bug anyway if I simply tell them about the problem. They also know the system better than I do and can probably find a bigger impact anyway
Of course, this is only a good strategy if you're just wanting to do a good deed and not counting on getting more than a thank you note, but Zendesk or Hackerone (whoever you want to blame here) didn't even accept the bug in the first place. That's the problem here, not the omission of an exploit chain
The audience of a security contact point (be that Hackerone or security@') is a technical person
We add impact demonstrations to a few findings per pentest report because our audience is broader: the nontechnical people who decide to allocate the money need to understand why this is useful and that the devs/sysadmins need to get enough time to do things right (developers and sysadmins are often sufficiently skilled, but are under delivery pressure). A sufficiently technical team, when the bug is adequately explained, doesn't need a functional exploit to see it's real/impactful or not
I previously worked for a mortgage software startup that attracted interest from big banks.
To ease concerns about our scalability and longevity, we move from a tiny office to an office with a lot of empty space.
This strategic move supposes signaled to prospective corporate clients that we were committed to sustaining our solution over the long term, rather than just a few years but in the end the company went out of business. so much for that.
Yet the same corporate will eat anything that Google or MSFT does while we all know they kill projects just like anyone else or like any smaller company going out.
I am actually seriously interested in what people there do day to day. I’m wondering this about a lot of very large companies, I would definitely watch a documentary about that.
Hour-long meetings about whether the copy should read "data center," "datacenter," "data-center," or whether it is really even correct to say any of these at all. And then negotiating with the design folks to fit in the extra character. Only to throw it all away because nobody thought about the fact that it has to support 5 different languages.
I wish I was kidding. Used to work at a place that did crap like that, pulling in developers for these time sucks because "only they really know the correct technical usage for our industry."
It's not weird to pick one and keep a consistent style, for example by looking at Google or at Wikipedia or some other source if the dictionary lists both or neither, but to have meetings about it?!
I work at similar size company. Basically they are like most companies building out the next 5 years while also keeping the lights on at four nines. There can be a lot of depth to product that you dont see. Anyone who says "why you need X people" often havn't tried a side hussle where you see 360 all the activities involved.
Building at scale without racking up big bill and hitting SLAs require a decent amount of effort.
At some point, most of your engineering time is spent on trying to understand what the previous team did. There's probably some engineer at Zendesk banging his head on the table because his boss wouldn't let him fix the sequential ticket IDs when he found them two months ago.
I work in one of the biggest companies of the world (employee and revenue wise) and it's basically a run-off reaction of well-articulated desk employees jerking each other off that, telling each other that they are so very important.
And the common management approach to anything not working immediately is "throw another 1.000 employees into the project" and the middle-managers measure their success by how many employees they are managing so it's a train without breaks. Hope it goes bankrupt soon.
Big companies are places where you get kudos for only taking two weeks to solve a problem you’ve solved elsewhere in two days. To an extent it’s Little’s Law. The latency requires more “CPUs” to handle the traffic.
This is super loud to me RN because some of these "big" companies are case studies in Mythical Man Month's "N channels of communication" as well as weird flashbacks to discussions on costs context switching and schedulers in various CS courses.
What's interesting is that Frank Slootman touts this transformation as a huge success in his book and talks at length about his conflict with Fred Luddy (who originally authored the simple ticketing incarnation of the ServiceNow monsterblob). The focus on keeping things simple is highlighted as an example of nerds' nearsighted thinking.
I'm sure it's a huge success for the few earning the profits from ServiceNow.
Like any SaaS, the more feature boxes you check, the more potential customers you can "satisfy". And the worse the UX gets for the average user (which then gets driven to purchasing more support).
Great for business (the few), terrible for users (the many). No contradiction there.
Every single click in ServiceNow takes a full 2 seconds to do anything. For a ticketing system. Insane.
What’s more insane is that it is still better than the vast majority of ticketing software. I don’t know what it is about ticketing and Helpdesk that it ALwAYs ends up like that.
> I don’t know what it is about ticketing and Helpdesk that it ALwAYs ends up like that.
The curse of B2B software is that every new big customer wants some custom feature or configuration that is the "deal breaker" for their multi-million dollar contract signing. And everyone except engineering is eager to give it to them because it's not their problem once the ink is dry. Support and renewals are the next guy's problem.
Software developers being surprised that software companies need to do a lot more than just write code is kind of like sailors being surprised that global logistics involves a lot more than handling a ship.
Never said that, but a competent engineer should be able to build like 75% of the main functionality of Zendesk over a weekend.
Now, I understand there's probably a lot more to it which is why I would expect it to be a company of around 50 engineers and 150 business/marketing/etc and that's being generous.
The hill I'd die on is that, with money not being a scarce resource and a technically feasible challenge present, a team of 200 should be able to build and sustain almost anything in the world. And that's being even generous. I think realistically a team of 50 should be able to build almost anything
Never said that, but a competent engineer should be able to build like 75% of the main functionality of Zendesk over a weekend.
Now, I understand there's probably a lot more to it which is why I would expect it to be a company of around 50 engineers and 150 business/marketing/etc and that's being generous.
The hill I'd die on is that, with money not being a scarce resource and a technically feasible challenge present, a team of 200 should be able to build and sustain almost anything in the world. And that's being even generous. I think realistically a team of 50 should be able to build almost anything
That’s a very HN take but the reality is that the tech is usually never the hard part. Selling, supporting, legal, all the certifications and enterprise contracts you have to do for a product like that are the hard part.
I’m not impressed with what I see of the three that there are, in some cases to do with how they came about and in some how they’ve been fixed.
The most recent one, https://security.snyk.io/vuln/SNYK-JS-MERMAID-2328372: I would be very concerned about trusting code that could be in any way adjacent to security written by whoever wrote (and whoever reviewed or committed) this original sanitizeUrl function <https://github.com/mermaid-js/mermaid/commit/066b7a0d0bda274...>. .replace(/javascript:/g, '') is obviously catastrophically wrong, breaking valid (though uncommon) URLs and completely failing to guard against javascript: URLs.
Yes, it’s fixed now, but the existence of the bug in the first place is highly alarming. No protection I can understand—for first-party use where you can trust the inputs it’s reasonable—but bad protection suggests someone tried but didn’t know what they were doing, and didn’t know that if you don’t know what you’re doing in security stuff you need to seek help, because there’s a surprisingly high chance that your bandaid will be worse than doing nothing (either that it actively makes things worse, or that it’s insufficient but gives an impression of safety). It’s dangerous cluelessness.
The middle one, https://security.snyk.io/vuln/SNYK-JS-MERMAID-1314738: the patch provided is utterly misguided and does not fix the alleged security vulnerability in the slightest—it barely even puts a bandaid over it, and it definitely breaks legitimate and reasonable stuff. See <https://github.com/mermaid-js/mermaid/pull/2123/commits/3d22...>: this is trivially insufficient and catastrophically wrong in its approach, so that if anything is actually depending on this code for security, it’s certainly broken. I haven’t immediately got an XSS in https://mermaid.live (something else is evidently providing the actual protection—so I think the advisory was either never valid, or it’s still unfixed), but it ruins reasonable labels like “Contrast with javascript: ahead-of-time compilation makes it faster” or “Do you strip javascript: URLs?” by removing the “javascript:” (eww!), but I can easily sneak a javascript: in there because of the sequential replacement done, with the likes of `java<iframescript:`. This is perhaps even more bogglingly incompetent than the /javascript:/g deletion. (Note that I’m using the word “incompetent” in its strict meaning, in no way as a slur. We all start out incompetent; but we need to develop enough of a feel for basically everything that we can identify situations where we’re not competent.) And even apart from all that, URL schemes are case-insensitive, as seen in data:text/html,<img%20src=x%20onerror=JavaScript:alert(1)>, so /javascript/ without the /i (case-insensitive) flag is insufficient anyway.
I’ve looked at two of them, might as well look at the third, https://security.snyk.io/vuln/SNYK-JS-MERMAID-174698. Oh wow. The first patch <https://github.com/mermaid-js/mermaid/commit/c33533082c598a0...> introduced /javascript:.*/g removal, which is both insufficient and excessive as already mostly discussed, implemented separately for flowchart and gantt (that’s a terrible idea that will consistently lead to divergent changes and missed places; this is library functionality that needs to be maintained in one place). Then the second patch <https://github.com/mermaid-js/mermaid/commit/f11d1a6fa1a5350...> switches to a real sanitiser, but leaves the terrible first approach around in a comment in one instance. And removes a bunch of console.log() calls that should never have been there. And starts escaping = as = for no reason (if you need this, something is badly wrong). And changes some conf to getConfig().flowchart for some reason. All in the one commit, with a very weak commit message that doesn’t address the why at all, and ignores most of the changes. This is not a clean code base or repository.
From what I’ve seen so far, I’m fairly confident that an audit of the code base would reveal multiple fairly severe security vulnerabilities. Also that if I started actually reviewing it I’d be crying out to drastically refactor large parts of it. I’m going to tip-toe away before I start poking this 20,000 line code base (excluding tests).
Someone can report that the second one hasn’t actually been fixed, and that the patch was actively harmful and worse than useless, if they’d like to. I don’t want to engage, lest I get sucked in. :-)
This would never have happened in the first place had they used an encrypted complex password and a simple password manager.
The whole company takes the hit with blunders like this. It's everyone's fault responsible for the infrastructure allowing this to happen in the first place. Those who pass the blame on others very quickly are equally to blame which means the CEO is just as to blame as the 'intern'.
Clearly the whole company doesn't train their interns.
The password was coded into a file that was checked in to git.
The git repository just happened to be public.
It's entirely reasonable to think that the person in question possibly didn't even stop to think that Solarwinds123 was an actual secret that needed to be kept, as it is the equivalent of common passwords that are published publicly in manufacturer documentation.
I think the point is at no point is it acceptable to be in a position to be able to do something deeply damaging to a company with something as simple as a intern leaking a password. The intern should never been put in a position where this was even possible.
I’d say in all the ways that matters this was basically everybody BUT the interns fault.
> Why’d he quit? ... he told me this: at each stage of a company’s growth, they have different needs. Those needs generally require different skills. What he enjoyed, and what he had the skills to do, was to take a tiny company and make it medium sized. Once a company was at that stage of growth, he was less interested and less good at taking them from there.
This is an aspect that gets overlooked in many businesses & careers. I've heard it phrased that companies go through 3 stages: Startup, Scale Up, Optimize. The above quote is a sub-stage of Scale Up. Some people are built for just a single stage and knowing how and where your skillset fits in is crucial to career happiness. As well as knowing when to encourage employees to move on.
Great post and I wish @steveklabnik continued success in his career!
In my current role, I've seen the company go from 6 to 120 people, and it's absolutely true that there are some people that are really amazing at a 6 person company who can be extremely counter-productive in a company of even 30 or 40.
Also, startups, since they are often run by people new at running companies, are often slow to respond to these kinds of folks. It's also rare that folks recognize this in themselves. I very nearly quit my job when going through a bunch of rough patches, because I thought, maybe the company finally scaled past me. I'm glad I didn't because it's smooth sailing again, but it's always important to reflect on yourself and ask if you are really doing the best you can. It's really hard to quit a role that is going poorly, let alone going passably well.
There's no one type, and it depends on the company. But a general archetype which seems to reoccur is that of the generalist.
Early on, people wear a lot of hats, which means they get to be involved in a lot of different parts of the company and its growth. At a certain stage, teams compartmentalize, and people who don't like being isolated to one or two teams tend to get frustrated with companies as they become more structured.
There's also often a phase where that structure is just starting to be formed, and it can be very awkward and unappealing to people as it plays out.
To borrow from the Harry Potter universe, early stage startups need Gryffindors and Ravenclaws, but then as the company grows more and more Hufflepuffs get hired, and sooner or later the company ends up on the radar of Slytherins, who are focused mostly on obtaining fame and renown, and who are more likely to have what muggles call "dark triad" personality traits.
What's most fascinating about the Hogwarts metaphor is that members of each house each bring some useful value, but the core values of each are often in conflict with the core values of the others.
At some point Mozilla went from being a heroic struggle that appealed to people who had a specific vision for the future of the internet, and turned into a status symbol like having Harvard on your résumé. This happens to any successful startup. A company that would never have appealed to a lot of workers suddenly becomes desirable (all else being equal) because of the status associated with it. Not to bash MBAs, but this is why I advise a "absolutely no MBAs" policy for startups.
MBA diplomas are simply status symbols and most people who have the degree joke about how easy it was to obtain and how much partying/networking they did while in school. They also graduate expecting to be placed in a leadership role due to the degree, even though young MBAs typically have little to no actual work experience or hard skills. I've seen overly confident MBAs nearly sink funding rounds for startups because they thought they were being clever with accounting and the investors saw right through it.
For what it's worth, I don't know if anyone here, me included, considers Mozilla a "status symbol" on the resume. It certainly is nowhere near FAANG in terms of inbound recruiter volume. People who want status symbols go to Google.
(I chose Mozilla over large company offers a decade ago because the work seemed more interesting, knowing full well it was more of a gamble in terms of my career. I've never regretted the decision.)
I obviously can't speak for anyone else on this, but I have actually given ex-Mozilla people a bit of a boost in the past when I've interviewed people because I view it as a) a semi-prestigious company, and b) a company filled with people who really like programming.
Great points. I agree with you on most counts except hufflepuffs are known to be loyal and I don't know if that's how I would characterize the middle stage employees. I'd say they are more like muggles. Quietly do what's told, no more no less.
An old thread[0] used different terms like “commandos, infantry & police”; or “pioneers, settlers & town planners” but which correspond roughly to your “startup, scale up & optimize” description, to explain the 3 stages of a company’s lifecycle.
> Getting motivated to lift 2x a week is another issue...
Was in the same boat until I started doing group classes. Used that to build accountability, motivation, and a "vocabulary" of how the gym works. Now I could spend hours at the gym by myself and love it. Find what works for you, cuz lifting is a blast!
When devs are promoted into management or team lead positions they are not given adequate training on what it means to manage / lead. Thus devs don't learn what good management is, and the stereotype of the bad manager perpetuates.
[1] https://gist.github.com/hackermondev/68ec8ed145fcee49d2f5e2b...
reply