> Too often, they're also product marketing documents, beginning and concluding with "overall assessments" of the target software that is almost invariably positive
We have some customers that clearly come to us to get reports they can share with investors, and not that many come back so either they weren't happy or (I like to think) they've checked that box and aren't doing further tests. But never did one of them ask us to rephrase something for overt marketing reasons or provide even a single sentence of marketing material to include. Perhaps we were clear that this is not something we do and that's why they don't come back, though since they also don't ask and still order a test, that doesn't seem logical.
I also can't say that I've seen a Cure53 report where this is the case, including this one. If anything, the statement about a critical vulnerability in the previous pentest could make one weary, so I applaud Mozilla for publishing this rather than silent fixing and keeping it under wraps.
> Vendors pay for public-facing reports, as an extra line item in the [statement of work]
This is the first I hear of that, though the public scrutiny does mean we spend quite a bit of extra review time on such reports (from spell checking to going over every substantial statement and rating) so I could understand if Cure53 has different practices from ours. That does not mean they were bought (assuming, for the sake of argument, that a premium was paid, which I don't think there was).
> that norm should be that public-facing reports can be provided only in the same dry, technical form they're presented to development teams in ordinary, non-public projects
That is exactly what we do, and I am quite sure Cure53 works the same way in this regard. We review more deeply if we know ahead of time it's going to be public, as I wrote above, to make sure things are correct (especially since English is none of our native languages), but we don't alter the contents.
> I think third-party assessment reports like these are a real problem in our industry.
I really don't understand where this comment is coming from unless you have very different experiences with customers or public reports where you are based. At a minimum it doesn't apply to the report at hand. Too often these things are kept secret and it's a rare opportunity that we don't have to swear silence on our work. I'd love to do that more.
We have some customers that clearly come to us to get reports they can share with investors, and not that many come back so either they weren't happy or (I like to think) they've checked that box and aren't doing further tests. But never did one of them ask us to rephrase something for overt marketing reasons or provide even a single sentence of marketing material to include. Perhaps we were clear that this is not something we do and that's why they don't come back, though since they also don't ask and still order a test, that doesn't seem logical.
I also can't say that I've seen a Cure53 report where this is the case, including this one. If anything, the statement about a critical vulnerability in the previous pentest could make one weary, so I applaud Mozilla for publishing this rather than silent fixing and keeping it under wraps.
> Vendors pay for public-facing reports, as an extra line item in the [statement of work]
This is the first I hear of that, though the public scrutiny does mean we spend quite a bit of extra review time on such reports (from spell checking to going over every substantial statement and rating) so I could understand if Cure53 has different practices from ours. That does not mean they were bought (assuming, for the sake of argument, that a premium was paid, which I don't think there was).
> that norm should be that public-facing reports can be provided only in the same dry, technical form they're presented to development teams in ordinary, non-public projects
That is exactly what we do, and I am quite sure Cure53 works the same way in this regard. We review more deeply if we know ahead of time it's going to be public, as I wrote above, to make sure things are correct (especially since English is none of our native languages), but we don't alter the contents.
> I think third-party assessment reports like these are a real problem in our industry.
I really don't understand where this comment is coming from unless you have very different experiences with customers or public reports where you are based. At a minimum it doesn't apply to the report at hand. Too often these things are kept secret and it's a rare opportunity that we don't have to swear silence on our work. I'd love to do that more.