The actual process of writing those reports is generally pretty laid back. It was hell mostly because it consumes all your time. Only about 50% of your time is spent hacking, if that. The rest is poured into that damn report.
It goes through multiple rounds of review, and every small detail is caught and corrected. Which is how it should be. It’s just the opposite of fun.
The report is literally the product. The implications of that didn’t sink in until a few months on the job.
Nobody spends 50% of their time writing reports. If anything, it's the exact opposite: people do their entire doc the last Friday of the test. It's enough of a problem that carefully scoped tests sometimes include doc days in the SOW. I used to get in arguments with the person we hired to run one of our offices for demanding that people stay late on Fridays to finish docs that customers weren't even going to look at until the following week.
This "multiple rounds of review" thing is news to me too.
To answer the original comment: the ordinary process for a project like this with a public report is, you deliver the whole project as if there wasn't a public report, with an internal report, and then the public report is based on the internal one (probably post-remediation). So actually delivering the project is usually no different than any other project.
I too worked for NCC Group. Before joining we spoke and you sent me a copy of the Web App Hackers Handbook and The Art of Software Security Assessments. I then worked for NCC the following 5 years.
Yes report writing was the worst and half the week was easily dedicated to that. PM’s and account managers pushed Friday readouts so it was a high priority so you would stay late to finish everything before you started another assessment the following Monday. Readouts for last week and kick off for a new project on the same day we’re the worst but happened every so often.
It was never find a bug, report and move on. It was dedicate the whole day to making it clear, presenting to the client and integrating feedback.
Don’t get me started on review. Every client doc had to have 2 reviews including from at least one senior and they put it off as much as possible.
Tom, I literally worked at your company. Matasano.
Let’s just say the culture changed after you left. Is that so hard to believe?
You probably didn’t spend 50% of your time writing reports, but Andy and the rest of my coworkers did.
Working at Matasano was one of the biggest disappointments of my career. It was basically the exact opposite of everything you pitched it to be. But I should’ve known better than to be deceived by marketing.
The actual hacking was sometimes fun, though. But again, that was maybe 50% of the time, if you’re lucky. Would you like me to break down an actual engagement timeline as a refresher?
An engagement lasts a week, max. Usually it’s two days. That means day one is hacking, day two is reporting.
The weeklong engagements are where the scales sometimes tip. But even by day three, you had better start writing up your findings, or else the report won’t be ready by Friday, and the client (or more specifically Wally, our manager) will not be pleased.
I literally worked at NCC Group after it had acquired Matasano.†
Your idea of the timeline does not match my experience. The normal engagement length is two weeks. (One week is a possibility, and I was on an engagement that was four weeks, but that was unusual.) In the one-week or two-week case, the final Thursday was devoted to writing the report, and the final Friday was devoted to presenting it. It would have been very unusual to spend 50% of total time on writing the report, though for something this high-profile (I was never on something similar) I wouldn't be shocked.
† Since they've come up, I would feel remiss if I didn't mention that they fired me for no stated reason‡ with zero days notice and zero severance. At the meeting where I was fired, they assured me that my benefits would not end immediately and would instead continue through the end of the month, despite the fact that it was October 31.
‡I have a strong guess about the reason. It wasn't a good reason.
Yes, it's hard to believe. You worked at Matasano/NCC for like a minute immediately after I left. What you are describing is nothing like the ordinary workload at NCC; the modal NCC engagement is 2 people, 2 weeks --- in fact, that's the modal engagement across the entire industry, not just at NCC. "A week, max". Sheesh.
I'm not interested in defending NCC; I have no interest in NCC (other than the friends of mine who still work there, I guess) and haven't since 2014. But I'm an inveterate message board nerd and I'm going to damn sure correct false things said about pentesting on HN.
In this instance, I'm going to go ahead and say I have better information about this than you do.
During my stint, I completed over 40 engagements. I never failed to find less than at least one medium severity flaw on any of those.
Of course, you wouldn’t know, since you showed up a grand total of ~two times. One was to brag about how Sam Altman called you up to offer you a slot at YC for Starfighter, another was to ask Wally for the rights to Microcorruption.
Meanwhile, I was the one in the field, doing the work, alongside Andy and the rest of the team. We spent a huge portion of our time writing. I’ll hop on a computer and describe it in more detail.
It’s funny that you declare “No one spends 50% of their time writing” like you’re the sole authority on the matter, across all the pentesting shops in the world. You didn’t even get it right at your own company.
Saying that I worked there “for like a minute” is cute, but not reflective of reality. Shall I go into more detail about my experiences? We can compare notes, and see where your experience diverged.
I've never spoken to anybody in management at NCC about why they fired you, but your coworkers have told stories about it, and I'm not sure you want the conversation you're asking to have. It is not the same story you tell.
I don't know why you're shocked at the number of times I "showed up" at the NCC Chicago office. I'll say it again: you and I never worked together. NCC hired you several months after I left. I know this because I still have the email thread of you asking me about the interview process. How often do you show up at ex-employer offices?
Wally was, at the time, your line manager at NCC. You get what a line manager is, right? Nobody was seriously asking Wally for the rights to anything, but I have no trouble believing you have trouble interpreting a joke.
You just claimed, a comment earlier, that NCC engagements were "a week, max". I stand by my previous comment. I have better information, and you have weird information. If your personal experience of NCC was a back-to-back sequence of 2-day projects, you were put in some strange back-bench situation.
I left Matasano and almost immediately started another consultancy, and I'll said it again: unless things have drastically changed since 2020, when I left for Fly.io, the modal pentest engagement is 3 person-weeks, not 2 days. People do not in fact spend 50% of their time writing reports.
We can compare notes if you like, but I don't think it's going to make you any happier to do so.
A moment later
I'm just sitting here thinking about this and your claim about documentation is even more risible than I had realized. By the time you were working at NCC, documentation was almost fully automated; we had the whole team working with Litany of Failure, our docgen system. Litany became such a big deal after I left that a year ago, at someone's going-away party, Erin and I got pins, which they'd made for everyone who worked on it. You were sure as shit using it in 2014.
By the time you were working at NCC, the experience of documenting a pentest was filing bugs in a bug tracker that autogenerated descriptions of things like XSS, and then pushing a button to have a PDF pop up.
I spoke with someone who worked contemporaneously with you, and who thinks you might just be confused --- your "projects last 2 days and typically consist of 50% documentation" claim would be consistent with you working SARs rather than pentests. What you're describing does sound a lot more like arch review than pentesting. Maybe that's the miscommunication here?
Edit
Somebody else pointed out that if you were mostly working on retests --- that's the project where you take someone else's report and look to see if they fixed all the findings --- you'd also have had a 1-3 day documentation-intensive workload.
Another person pointed out that netpen projects fit this bill too, but of course, you weren't doing back-to-back runs of network penetration tests out of that office.
All I care about is the claim upthread that people spend 50% of their time on pentests documenting things. If you engage a firm to do a software assessment and they spend 50% of their time in doc, fire them.
This person was not my employee. Even if I had been at the firm at the same time, he still wouldn't have been my employee.
It bothers me that they've continued to claim over the years that they were "fired for narcolepsy". The NCC I'm familiar with paid multiple people for years who weren't able to deliver work because of health issues, and did so without blinking. There's not a lot that I especially like about NCC management, but their handling of health and disability issues would have been at the bottom of my list of complaints.
> It bothers me that they've continued to claim over the years that they were "fired for narcolepsy". The NCC I'm familiar with paid multiple people for years who weren't able to deliver work because of health issues, and did so without blinking.
Maybe they've got special policies on health. I am not impressed with NCC Group's firing policies. I was fired because, as far as I can see, I was recruited to the bug bounty team with an offered perk (work from anywhere), I said I wanted the perk, they said no problem, my boss was promoted, and the new boss didn't want to allow the perk. He made repeated comments to the effect that he wasn't comfortable having me on his team after I'd made a request that he wasn't willing to grant immediately. He also told me that, if the worst came to the worst, I would be transferred back to active consultant status rather than being fired, which didn't happen.
I'm fine with that. I'm not writing here to make myself look better, just to correct the record. Some pathological shit happens at US security consultancies; it's just (in the main) not the stuff this person is talking about.
Again: I have never worked with this person, despite their claims and implications to the contrary. Further, I went way out of my way not to be a people manager (and still do). Nobody reports to me; that's not a thing I'm good at, as I'm probably making evident.
> your coworkers have told stories about it, and I'm not sure you want the conversation you're asking to have. It is not the same story you tell.
I hereby give you permission to lay out the full story, full-stop. Don't hold back. I know it's usually not a classy move, but you're hereby absolved of that.
> I don't know why you're shocked at the number of times I "showed up" at the NCC Chicago office.
I wasn't shocked. It was to point out that you weren't in the field anymore. We were.
> Wally was, at the time, your line manager at NCC. You get what a line manager is, right? Nobody was seriously asking Wally for the rights to anything, but I have no trouble believing you have trouble interpreting a joke.
No doubt. But that raises the question of why you met with Wally privately. That's a bit at odds with your immediate previous claim that there's no reason to show up to your ex-employer's office. But this is just a distraction.
> You just claimed, a comment earlier, that NCC engagements were "a week, max". I stand by my previous comment. I have better information, and you have weird information. If your personal experience of NCC was a back-to-back sequence of 2-day projects, you were put in some strange back-bench situation.
Pop quiz: If I worked there for a full year -- 52 weeks -- then how did I complete over 40 engagements?
> I'm just sitting here thinking about this and your claim about documentation is even more risible than I had realized. By the time you were working at NCC, documentation was almost fully automated; we had the whole team working with Litany of Failure, our docgen system. Litany became such a big deal after I left that a year ago, at someone's going-away party, Erin and I got pins, which they'd made for everyone who worked on it. You were sure as shit using it in 2014.
Yes, we used Litany exclusively for our docgen. That's the templating system that creates the PDFs. It's quite a nice system; thanks for making it.
It doesn't change the fact that you have to actually fill it out with words.
> 50% of your time. Give me a break.
We did.
So, let's hear those stories. I imagine it'll go something like this: "That Shawn was such a slacker that we couldn't figure out what to do with him. He spent most of his time fooling around on the computer. At one point he went to the bathroom for a full half hour."
My work spoke for itself. Pentesting was a necessary part of the job, but the product is the report. The report is what the client sees, and (in some cases) what they pay several hundred thousand dollars for. Is it any surprise that a huge amount of time is spent preparing this extremely-valuable product for consumption? This isn't even a particularly strange claim; Damien was the one who pointed out to me that the PDF is what clients are paying for.
I'd be interested to compare notes. What did you have in mind?
I think these claims are pretty much risible. The report you're talking about is autogenerated from a bug tracker. The client sees your bugs. If you spend 50% of your time writing them, you're cheating the client.
What I really think happens is that you misinterpret things people tell you and blow them into weird directions. Somebody told you that "the PDF is what clients are paying for". Well, no shit. The PDF is the only deliverable from the project. It's where the bugs are written down.
I wasn't there for whatever you got told, but it sounds to me like the subtext of it was probably "so it doesn't matter much what you do on a project as long as the client ends up with a report that they can use to claim they've had an assessment done". That's a cynical thing to say, but definitely a thing that gets said. It's also, strictly speaking, true.
What I hear you saying is, "the PDF is the only thing that matters, so we should spend most of our time making the best possible PDF". That's not only silly, but also actively difficult to do in a practice where the reports are autogenerated.
The actual figure of merit from a software pentest is the list of bugs, full stop. Yes, the list is delivered in PDF. Don't let that confuse you.
I don't think you've worked in this field seriously enough or long enough to use the word "we" the way you are.
Can confirm a 2 day engagement is unusual, and 50% of time writing the report is possible but very much an outlier for standard pen tests. Some interesting exceptions include:
* Some regions have a much shorter average engagement time. North America is usually pretty generous, where markets in other countries will only bear half or a third of the time.
* If you are a junior or less skilled you are perhaps more likely to get the small jobs while you are learning.
* External inf can be short on testing time and long in reporting if you find lots of issues, but automation helps the reporting in that regard.
* Some pentests are very documentation intense for specific reasons, such as M&A due diligence, or clients who want threat models and design reviews incuded. Still isn't 50% though.
And others. But in general what Thomas describes has been my experience over the years.
Disclaimer: I work for NCC, but nothing related to former Matasano and I don't know Thomas. Opinions are my own.
It's interesting to read about other philosophies for engagements. In the places I've worked it would be rare to send a junior engineer on a short engagement. The reason being that short engagements are usually 1 engineer, maybe 2. There are always tools and tests that take time and it's better to have 1 engineer for 2 days than 2 engineers for 1 day. We'd send our junior engineers on the multiweek engagements so they'd learn more. They'd get a chance to encounter all types of systems and networks, and would be able to see how the senior engineers approach problems. We could even leave them to figure out complex topics on their own in some cases (and often they'd teach us new things in the process!).
But as I said in another comment, depending on what people consider to include as "report writing" I can definitely see some engagements needing 50% time there. So maybe this person did just get unlucky.
Sub-week software pentest engagements at established firms are pretty rare. There's a logistical reason for that: engagements are overwhelmingly measured in person/weeks, and if you book out a consultant for two days, you fuck the schedule for the rest of that person's week. It's the same reason (or one of them) that you shouldn't bill hourly if you do your own consulting work: if a client books you for a couple hours in a day, they've fucked the rest of the day for you.
A 1 person-week engagement is pretty short. On a 1 p/w engagement, you'll have scoped back drastically what you can test; maybe one functional area of a smallish web app, or, every once in awhile, you'll get a big client that has the budget flexibility to do things like book "one week of just looking for SQLI and nothing else across all our internal web apps".
The typical CRUD app for a small tech company would tend to come in between 3-4 person weeks. Sometimes, those engagements would have their last 2 days explicitly reserved for doc in the SOW. I felt like (still feel like) that's rustproofing; clients are paying for testing, not writing. Usually there's a couple days of "discovery" at the beginning. The rest of it is just testing.
The typical order of a project with a public report (those are pretty infrequent) is that the public report is done after the the original test is accepted. That's in part because clients want to triage and remediate findings before they release a public report; you sort of can't drop a public report and the internal report at the same time. So public report writing shouldn't have much of an impact on the project delivery schedule, because it's not done at the same time.
For sure, a short engagement of 1-2 days would be rare. We'd occasionally do them to get a foot in the door or as a follow-up for a regular customer. We'd still not want a junior engineer on them. You want to make as good of an impression as you can so you get the bigger contracts and you don't want someone there who isn't experienced communicating with clients.
But, per the thread, there are some special-case projects that are short and do take junior delivery staff; SARs, which are "pentest the documentation" projects meant to derive a thread model and inform a later, real, test (I don't like SARs and think they're a kind of consulting rustproofing, but people smarter than me disagree strongly with this), and, of course, retests.
Thank you. (And thanks for being dispassionate; it's a nice change.)
It sounds like the most likely explanation is that Matasano was an outlier. My career was cut short before I had a window into the rest of the pentesting world, but it's good to hear that places exist that aren't so obsessive about the actual writing process. I also happened to experience most of your list, so it sounds like it was an exceptional situation in general, so it's best not to draw sweeping conclusions from it.
It's an interesting tactic to set up a strawman and then beat it up. Where am I to start when the reply will mostly be "That's not true" or "I didn't say that"? This is devolving into being boring for the audience, but you're (for some reason) attacking my reputation. You're also backpedaling; I thought you were going to tell stories? I'd like to hear them. Or did you check with someone, and they said "Um, actually, Shawn wasn't that bad"?
> I think these claims are pretty much risible. The report you're talking about is autogenerated from a bug tracker. The client sees your bugs. If you spend 50% of your time writing them, you're cheating the client.
You keep saying the report is autogenerated because Litany existed. This is a bit like claiming that scientific papers are autogenerated because LaTeX exists. Yes, it does a lot of heavy lifting. No, it doesn't change the fact that you start with a template and then rework it into the proper form. As any grad student will tell you, that work takes a lot of time. My experience at Matasano was similar. I bet Drew, Andy, Damien, Dmitri, and a few other folks would at least say that reporting occupied a significant chunk of time.
From where I'm sitting, the claim seems unremarkable. Look at how long this thread's security assessment is. Most of the words are boilerplate, but you can't simply ship boilerplate. And yeah, the reports went through multiple rounds of back-and-forth before they got shipped, during which every detail was combed over.
> What I really think happens is that you misinterpret things people tell you and blow them into weird directions. Somebody told you that "the PDF is what clients are paying for". Well, no shit. The PDF is the only deliverable from the project. It's where the bugs are written down.
It wasn't "someone." Damien was one of the most experienced pentesters at Matasano. He led several redteam projects, and taught me a lot of interesting tricks for sneaking your way into a network.
> I wasn't there for whatever you got told, but it sounds to me like the subtext of it was probably "so it doesn't matter much what you do on a project as long as the client ends up with a report that they can use to claim they've had an assessment done". That's a cynical thing to say, but definitely a thing that gets said. It's also, strictly speaking, true.
This is a strawman. What he was saying was that we need to do a good job on the report, in addition to the hacking. I don't know why you'd spin it into some cynical thing, but at least you're consistent.
> The actual figure of merit from a software pentest is the list of bugs, full stop. Yes, the list is delivered in PDF. Don't let that confuse you.
We can debate who's the one confused, but at this point it's pretty clear that our experiences were dramatically different. What possible benefit would it be to me to sit here and lie about it? Not only would you (and everyone else) call me out, but I'd have to keep making up more and more elaborate falsehoods.
Sounds exhausting. I'm just reporting what I saw, and what I did.
I don't see a productive way to continue this. Your position is that "nobody spends 50% of their time on reports." I'll concede that maybe it's closer to 40%. But it's certainly not 10%, or whatever small amount you're hinting at. And as you pointed out, my last month was filled with 100% documentation-writing.
I'm comfortable with what the thread says about how we came at this disagreement.
As you've acknowledged upthread: your claim that documentation is 50% of the time on a pentest doesn't hold up. I believe it took 50% of your time, because you say it did, but like you said upthread: it's was an exceptional case. Maybe:
1. You spent more time on doc than was actually required
2. You worked a bunch of SARs, which are short doc-focused projects (and not pentests)
3. You were given a bunch of retests, which are short doc-focused projects; this almost fits, since it's some of the first work that's given to new team members after they're done shadowing on projects, except that it would be weird for there to be so much retest work that you could do them back-to-back for a long period of time (most clients don't purchase retests)
4. You worked ENPTs (external netpens), which are short projects and have a huge doc component of "fitting tool results into a report". But (a) your office didn't do a lot of those (Matasano in general didn't do a lot of netpen work) and (b) it wasn't low-seniority work; as I recall, the people who did netpens specialized in it.
It could be some combination of all these factors.
Meanwhile: Litany is nothing at all like LaTeX (though after I left LaTeX staged a coup and replaced the commercial PDF library it had been using). Litany is a bug tracker. You enter a finding title, a URL/location, and a description --- which Litany autogenerates for common bug classes --- and when you're done with the project you push a button and get a complete PDF. This would have been the primary way all reporting was done during your tenure.
Interestingly, one of the few major disputes between Matasano and iSec Partners (the big two constituent firms in NCC US, iSec bigger by a factor of almost 2) was report generation. iSec's actually had a LaTeX template consultants would use to generate reports; they wrote them by hand. "I refuse to use Litany" (and lose control over my report formatting) was supposedly such a big thing that they had to rename Litany when they re-introduced it across the firm; it's now called Limb. Which is a tragedy, because Litany of Failure is one of the all-time great product names (credit, I believe, to Craig Brozefsky).
Some of the people you've mentioned in this thread have reached out to me. I stand by literally everything I've said. It's OK to be wrong, and you are.
Just in case this ever comes up again, I'll at least commit to not having changed my story. :)
Long story short: don't pay for software pentests that spend 50% of their time in doc. You're paying for the testing, not the report, even if the report is all you ultimately care about.
> Nobody spends 50% of their time writing reports. If anything, it's the exact opposite
At my employer we don't, but a friend said that 50% of their time at a big multinational who does pentesting and other auditing services was reporting and fighting the MS Word template. Apparently it's not that abnormal and hard enough to change that my friend would rather leave than work with the team to get it fixed.
> This "multiple rounds of review" thing is news to me too.
Depends on the definition, like do you count your teammates reading what you wrote on top of the boss checking off on the report? If yes, then we have multiple rounds of reviews for every report. If no, then we only have it for certain (not all) public reports. Again, not a weird thing.
(To get a bit meta, your writing style across any security-related comments is a bit "this is definitively how it definitely is" whereas... well, in this case it's objectively not true. Some orgs do spend (waste) disproportionate amounts of time on reporting.)
> the ordinary process for a project like this with a public report is, you deliver the whole project as if there wasn't a public report, with an internal report, and then the public report is based on the internal one (probably post-remediation)
This is also not how we do it. We typically know beforehand if it is going to be public and the report that gets sent to the customer and released to the public is literally the same file. It's not that a second report gets written "based on" the real report. That's how we maintain integrity as a third-party auditor: we don't sugar-coat or alter our reports if it's going to be public. If desired, both for public and nonpublic, we can do a retest and add remediation status. What is different is that we triple check for typos, lines running out of the page, etc., and might be more verbose so a wider audience can understand the findings (some customers are very technical, others not at all; for a public report we mostly switch to non-technical-customer mode). Interesting to know that this is not, at least not ordinarily, the case at places you work(ed).
I can't speak to the big multinational your friend works at, only NCC, and, I guess, a bunch of boutique-y firms that don't qualify as "big multinationals" the way NCC does. And, just generally: if you're contracting software pentests, you should not be OK with the prospect of your consultants spending 50% of their time in doc. That isn't a norm. You should expect most of the time to be spent looking for the bugs you're paying for them to find.
After your edit
With respect to your thing about how public reports work, I appreciate the intellectual honesty of the approach your firm takes, and I have real problems with public reports in general (something I've ranted about here before).
But at a big US pentest firm I think it'd be hard to do public reports any other way. For one thing: clients routinely dispute findings, and they're often right (they know more about their own code than the testers do). You'd have to let them see the report before you could write a version of the report for the public, so you can scrub out the findings that don't pan out.
Another obvious issue is that reports often include tentative findings, which need to be confirmed before the report is finalized.
I'm interested, just as a security consulting nerd, in how your process handles these things. But past that, the question that started this thread was whether having Google breathing down your neck would make this a hard engagement to deliver, and I think the answer to that is almost certainly "no". If anything, Google wants good findings, and the stressful part of this engagement would be coming up with a report that never gets anything more than a sev:med.
(I don't know, Google's a big company, I haven't done a software assessment in over a year, &c &c).
> I guess, a bunch of boutique-y firms that don't qualify as "big multinationals" the way NCC does
NCC has on the order of 1% of the revenue of this "boutique-y firm". I'd mention the name but then I'd want to check with $friend first in case it would somehow make them identifiable.
> you should not be OK with the prospect of your consultants spending 50% of their time in doc
Just so we're clear: I'm saying my personal experience extends only as far as a single org of NCC's size, and then a variety of boutique-y firms of ~Matasano's size. I have no experience with Deloitte or PwC. I'm not calling your friend's firm a boutique; it's clearly not.
(Sorry about the substantial edit after you apparently already read my comment. Most of the time people don't see it that fast ^^')
> at a big US pentest firm I think it'd be hard to do public reports any other way [...] You'd have to let [clients] see the report before you could write a version of the report for the public, so you can scrub out the findings that don't pan out.
We do let them see before, also because there is indeed time needed to roll out fixes, but we aren't often asked to make substantial changes (and if they ask, whether we can oblige depends).
Sometimes we make a real mistake and then it just looks silly for both parties so that would be removed, but usually things are just factually correct. For example, we can't always know about backports when we see an old version (and PoCs / detailed info is notoriously missing from CVEs, one of my pet peeves), so we recommend they check that the thing is up to date and implement an update procedure if it's not. That advice is valid no matter the actual state. We can mark it as checked and OK as part of a retest (if we got ssh to see for ourselves, or we would write something like "$customer checked and says it was up to date" as the retest description alongside a 'resolved' marker).
> Another obvious issue is that reports often include tentative findings,
Is this like an int overflow that you're not yet sure is exploitable or so? Or do you have an example of this? I'm not familiar with the concept of a tentative finding unless it's like a little note "this search function at /search.php produces a weird error with input {{1+1}} but I haven't found a useful bug yet, todo investigate more".
If it's the latter, we don't report those if we couldn't find anything clearly wrong with it. If it's just not following the spec (HTML entity encoding issues are a common example) but we don't find a security flaw, then it's an informational note at best. Maybe we should be more pushy with these in case it turns out to be exploitable.
> the question that started this thread was whether having Google breathing down your neck would make this a hard engagement to deliver, and I think the answer to that is almost certainly "no".
I mostly agree there. It's definitely a bit more work with our process because what the test team sends out for internal review is supposed to be basically ready to publish, so they'll do more double checking of the facts before git committing the finding and such. But at the same time, because the routine is almost identical for private and public reports, it's not very special even for big customers that people here would recognize.
Is the engagement harder? Slightly, because you don't want to make the firm look stupid by doing something wrong. Does it qualify for the statement "hard to deliver"? Nah, I agree that this answer is "no".
> the stressful part of this engagement would be coming up with a report that never gets anything more than a sev:med
This. Or even nothing above 'low' if the attack surface is small and the devs are up to date on the relevant security best practices.
Tentative findings happen all the time! You get partial code for things, or you get the code for something that sits in front of some service you don't get the code for, or the environment might not allow the input that you've determined from the code might be a vulnerability. Or, to flip the script: you're primarily testing and not reviewing code, and you come across weird behavior --- a timing-variable 500 response, for instance --- and you don't have log access. Even when you have complete code, if you fuzz the target, finding the code that could have generated the exception can take days, and it's sometimes better to doc and move on than to waste time figuring something out the client can work out in a minute.
Another class of tentative finding: authz issues in very complicated enterprise software, where your "supply clerk" user can get access to some "inventory list" data that the ACLs suggest should be restricted, but it turns out that supply clerks are just supposed to have access to this stuff.
A lot of crashers are also tentative findings. Is it memory corruption or is it a null pointer dereference? That's the difference between a sev:hi and a sev:info.
But, like, again, I'm a pentest nerd, and I'm super interested in processes that are deliberately honest about public reports. I think public reports are pretty problematic, even from firms that I like. So don't read me as pushing back on you, you've just perked my ears up.
Never did a security audit, but I did regulatory and financial audits of banks. Most of our work involved looking at last year's work, re-performing it and changing dates. Writing reports + financial statement notes followed a similar process.
Exceptions are immediately escalated to the audit committee and sometimes end up as a small footnote in the public reports. Most of the time it says "we were unable to collect sufficient evidence" to provide assurance. Almost never "this was done wrong".
This is a good call-out. The software security field uses the term "audit" in a deeply weird way. A real audit, like the kind done in a SOC2 assessment, really is about the report, and is more like 90% paperwork than 50%.
Here we're talking about software pentesting, and consulting firms have historically used the word "audit" to elevate that work. But these engagements almost never follow the shape of a real audit; there's no spelled-out audit criteria, there's no reconciliation against previous results, and the focus of the engagement is almost purely the exceptions, with little to no documentation of the tests completed without findings.
You've noted elsewhere in the thread that pentesting has the concept of a "retest". A retest purely examines the findings of some original earlier report. Any other vulnerabilities are out of scope, no matter how serious.
This seems like a better match to the regulatory audit model, but it's a bad match to the problem people would like to think of audits as solving, which is determining "are there any problems?".
The normal pentesting use of "audit" draws on that intuitive meaning of the word; the concept is that you answer whether problems exist. But it's deeply weird anyway, because the answer can't be known, only guessed.
I seem to remember PCI being called out as a very different report type from the usual, closer to the regulatory audit model at least in that the process was heavily regulated and old problems had to be fixed and retested. I never did a PCI report; don't hold me to anything.
Yes, exactly. There is a weird conflict of interest thing happening with a lot of public-facing security assessment work. The client wants a clean bill of health. The delivery consultants can't honestly sell that, at least not without a huge project scope stretching into double-digit person/months. But the firm wants to sell engagements, and public reports are a condition of the engagement.
So we have this phenomenon of "audit reports" that are really anything but that. Very few people in the industry know how to read them (for instance, how to locate and evaluate the scope of the project). But they're effectively used as seals of approval by clients. Which creates an even bigger incentive for firms to sell them, to the point where there are firms that almost specialize in doing them.
PCI is closer to the audit model, and yet even less effective than the pentest model, because the standardized delivery model created a race to the bottom effect in the market.
My oddball position on this stuff: firms shouldn't do external reports at all, and clients should just be able to post their internal reports.
I worked at Matasano for two years and did many tens of assessments. None even remotely approached 50% of the time spent on the assessment.
I left there before the Litany of Failure was completed. I did security assessments on my own and wrote something similar to Litany of Failure (I think it is better) and the thing is that you should be documenting as you go along so it is not such a painful crush at the end.
The point of an assessment like this is to identify potential risks. That one is categorized as “informational”, and isn’t a defect.
These types of recommendations are useful in context. You may have a moderate finding or series of findings that in context are more significant because of some informational finding.
I'm going to stay away from the tire fire of accusations between you and the former NCC boss. But I do want to say that having run a pen testing firm for many, many years these claims aren't hard to believe. It does depend on what you consider to be time spent working on a report, however.
50% time actually hacking even sounds kinda high depending on how the organization is structured. We had some engineers that just absolutely sucked at writing documentation but were wizzes at hacking so they got away with passing some of the documentation off on others (dramatically increasing the other engineers workloads, but sometimes that's ok). We also had some folks that were just great at formatting and dealing with client requests for docs and they could take workload off of engineers and make everyone happy.
An engagement ranged from 3 days to 6 weeks, with the average around 3 weeks. A typical internal pen test would be 2 weeks onsite followed by a week for writing the report and handling feedback (and travel and meetings and blah blah blah). It wasn't unusual for a client to request that all references to X be changed to Y or the word "should" be changed to "shall" or whatever other thing client management cared about. That time can add up quickly and make your job miserable.
Every single pen testing firm goes through a phase where they realize they are "wasting" weeks of effort on writing their reports. And they all decide they're going to automate as much as they can to improve their profit margins and employee happiness. They all usually come up with a snarky name for the new tool that does this automation. Most of the time the generation tool saves pain when it comes to formatting tables, table of contents, glossaries, etc., but wastes time wrestling with client expectations that their report be unique or focus on unusual areas. Then the engineer has to figure out how to modify the auto generated report without breaking any future changes to finding severity or writeup.
Some people would also consider time spent presenting the report to a board of directors as time spent "working on a report" and I wouldn't really disagree. It's not unusual to send an engineer and a manager to do a presentation at a remote site after the engagement finishes. That'll include 1-2 days of travel time and the time the group spends trying to "up sell" the client on further engagements.
It's been some years since I've been in that field but back in my day a break down of a 3 week engagement could look something like this:
First week is the fun part. You probably spend 30-40 billable hours on actual discovery, scanning, testing, hacking. And if you're smart you spend 3-5 hours on the early design of the report. The client might want a focus on the API or on WiFi or whatever and you arrange things so the report is organized properly early on and the team can input their findings as needed (or input them to the automated system which might not have a dedicated category for this focus area).
Much of the second week could be considered part of writing the report. You've ideally root'd most of the infra in the first week and now you're using tools and manual inspection to find every single vuln you can in the networks. You're using automated tools to find common mis-configurations and missing patches. Those tools are (hopefully) connected to your automated report generating system somehow so you don't need to format them manually. But there's always new stuff on each engagement that'll need some custom write-ups and severity determination will need to be adjusted depending on the client's security posture and risk profile.
The third week is 1-2 days getting the report ready, though that can extend further for unique engagements. Then there's submitting it to a tech editor, submitting it to management, submitting it to the client and dealing with corrections from all of them. It's extremely rare for a client not to have comments and changes that need to be addressed and that's another 2-8 hours.
Pen testing is difficult -- the travel, the presentations, the sales, the report writing. I'm not saying what this commenter is claiming is true, but the job is not how most people imagine it to be. I'll also say that that's not a criticism of NCC at all. All firms I've worked for, managed, or talked with are like this.
It goes through multiple rounds of review, and every small detail is caught and corrected. Which is how it should be. It’s just the opposite of fun.
The report is literally the product. The implications of that didn’t sink in until a few months on the job.