Or the problem where the developers will not know why the AI did something, say an AI decides to do some bad thing, the judge will ask the developer why and the dev will say:
-I don't know, we feed them a ton of input and it worked fine on my machine and without test data so we shipped it