Whilst I'm not the parent commenter I do work in the same industry..
>>These are not real infosec firms, they are parasites.
>The entire consulting penetration testing market is setup to encourage this behavior. There is no way to prove you actually did anything correct. Someone can write a wonderful PDF analysis by hand and still leave the system full of glaring holes. Customers can't tell a system is broken until it gets hacked.
So a good company should be able to provide a methodology detailing the tests they do, you'll also see some who report the tests conducts and the results (positive or negative), so asking for sample reports from consultancies would help to find one closer to your specific needs. Personally I prefer reporting all test results as it keeps both parties straight on what has and has not been covered.
>>More specifically, we ask for it and we receive it, and we do exceedingly well. If people keep paying us five figures a week to perform a penetration test, we're not going to stop asking for it or reduce our prices.
>Right, but many times I've seen companies do it because they are desperate to do it for compliance purposes. :/ Essentially there is a non-trivial portion of the market held up by regulatory demand.
Yeah where people are getting tests for purely compliance reasons there is a tendency to go with cheap suppliers as there's not really good perceived benefit.
>>Strictly "policy" audits such as PCI compliance differ a bit, but in general they should still involve a technical deep dive into your product's infrastructure, conducted by consultants with expertise in multiple tech stacks and overall experience in a variety of frontend and backend frameworks.
>I'm curious. Do you review every line of code in a customer's codebase? What about the code of every library they import? If you don't review imports, do you leave a big caveat in your report that says their code looks okay, but the libraries could be full of vulnerabilities?
Heh this is one of the huge gaping holes in security at the moment. Most applications are now constructed of piles of code acquired from repos (npm, nuget, rubygems etc) that provide absolutely no curation of content and anyone can put any code they like up there. There is (from what I've seen) very little appetite from companies to actually try and audit their whole stack, generally due to the cost of doing that. Manual code review is expensive and when you start importing 100Kloc of 3rd party code into your solution it would not be a cheap excercise to validate...
>>These are not real infosec firms, they are parasites.
>The entire consulting penetration testing market is setup to encourage this behavior. There is no way to prove you actually did anything correct. Someone can write a wonderful PDF analysis by hand and still leave the system full of glaring holes. Customers can't tell a system is broken until it gets hacked.
So a good company should be able to provide a methodology detailing the tests they do, you'll also see some who report the tests conducts and the results (positive or negative), so asking for sample reports from consultancies would help to find one closer to your specific needs. Personally I prefer reporting all test results as it keeps both parties straight on what has and has not been covered.
>>More specifically, we ask for it and we receive it, and we do exceedingly well. If people keep paying us five figures a week to perform a penetration test, we're not going to stop asking for it or reduce our prices.
>Right, but many times I've seen companies do it because they are desperate to do it for compliance purposes. :/ Essentially there is a non-trivial portion of the market held up by regulatory demand.
Yeah where people are getting tests for purely compliance reasons there is a tendency to go with cheap suppliers as there's not really good perceived benefit.
>>Strictly "policy" audits such as PCI compliance differ a bit, but in general they should still involve a technical deep dive into your product's infrastructure, conducted by consultants with expertise in multiple tech stacks and overall experience in a variety of frontend and backend frameworks.
>I'm curious. Do you review every line of code in a customer's codebase? What about the code of every library they import? If you don't review imports, do you leave a big caveat in your report that says their code looks okay, but the libraries could be full of vulnerabilities?
Heh this is one of the huge gaping holes in security at the moment. Most applications are now constructed of piles of code acquired from repos (npm, nuget, rubygems etc) that provide absolutely no curation of content and anyone can put any code they like up there. There is (from what I've seen) very little appetite from companies to actually try and audit their whole stack, generally due to the cost of doing that. Manual code review is expensive and when you start importing 100Kloc of 3rd party code into your solution it would not be a cheap excercise to validate...