Author here. Serious systems undefined, fine - I'd view them as systems that take decisions, not just make predictions (see final quoted line!). "Want to trust a system at the same level as ... a theorem" - straw man, explicitly denied in the article. Components .previously used and tested in diverse ways, testing coverage of some sort, would already be really good. Yes AI systems have architectural components such as embeddings, but they relate to types of functionalities and applications, not to functionalities.

The point about solving more problems at a low reliability level rather than solving problems at a higher reliability level is highly interesting though!