Practical neural networks operate in enormous parameter spaces that are impossible to meaningfully test for all possible adversarial inputs and degraded outputs. Your FSD could well recognize stop signs in your battery of tests but not when someone drew a squirrel on it with green sharpie.
Something a bit similar is clinical trial and it is accepted without problem.
You make a black box test on several thousands (sometimes only hundreds) patients, and if patients who received the drug perform better the patients who received the placebo, then the drug is usually accepted for commercialization.
Yet one isolated patient may be subject to several comorbidities, her environment could be weird, she could ingest other drugs (or coffee, OTC vitamins or even pomelo) without having declared it. In a recent past women were not part of clinical trials because being pregnant makes them very "non-standard'.
> Something a bit similar is clinical trial and it is accepted without problem.
Clinical trials also have strict ethical oversight and are opt-in. If clinical trials were like Teslas, we'd yeet drugs into mailboxes and see what happened.
First of all, clinical trials are typically longer and more thorough than you imagine, they span years. The fact that COVID vaccines were fast-tracked gives people wrong idea about it.
Secondly, even after the product hits the market the company is still responsible for tracking any possible adverse effects. They have a hotline where a patient or doctor can report it, and every single employee or contractor (including receptionists, cleaning staff, etc.) is taught to report such events through proper internal channels if they accidentaly learn about them.
Your run of the mill computer program also "operates in enormous parameter spaces that are impossible to meaningfully test for all possible adversarial inputs and degraded outputs".
This is hardly similar as the state of a typical computer program can be meaningfully inspected, allowing both useful insights for adversarial test setups and designing comprehensive formal tests.
Right, if you consider the internal state, it is hardly similar. You talked about black box and QA though. Black box by definition holds the internal state as irrelevant, and QA mostly treats the software it tests as a black box, or in other words the tests are "superficial" as you call it.
Black box testing in typical software is, however, less superficial, because the tester can make inferences and predictions about what inputs will affect the results. When you're testing a drawing program, for example, you may not know how the rectangle tool actually works internally, but you can make some pretty educated guesses about what types of inputs and program state affect its behavior. If the whole thing is controlled by a neural network with a staggering amount of internal state, the connections you can draw are much, much more tenuous, and consequently the confidence in your test methodology is significantly harder to come by.
