A sequence of AI models trained on each other's output gets mutations, which might help or hurt, but if there's one dominant model at any given time then it's like asexual reproduction with only living descendant in each generation (and all the competing models being failures to reproduce). A photocopy of a photocopy of a photocopy — this seems to me to also be the incorrect model which Intelligent Design proponents seem to mistakenly think is how evolution is supposed to work.
A huge number of competing models that never rise to dominance would be more like plants spreading pollen in the wind.
A huge number of AI there are each smart enough to decide what to include in its training set would be more like animal reproduction. The fittest memes survive.
Memetic mode collapses still happen in individual AI (they still happen in humans, we're not magic), but that manifests as certain AI ceasing to be useful and others replacing them economically.
A few mega-minds is a memetic monoculture, fragile in all the same ways as a biological monoculture.
A different biological analogy occurred to me which I've mentioned
before in a security context. It isn't model degeneration but the
amplification of invisible nasties that don't become a problem until
way down the line.
Natural examples are prions such as Bovine spongiform encephalopathy
[0] or sheep scrapie. This seems to really become a problem in systems
with a strong and fast positive feedback loop with some selector. In
the case of cattle it was feeding rendered bonemeal from dead cattle
back to livestock. Prions are immune to high temperature removal so
are selected for and concentrated by the feedback process.
To really feel the horror of this, read Ken Thompson's "Reflections on
Trusting Trust" [1] and ponder the ways that a trojan can be replicated
iteratively (like a worm) but undetectably.
It isn't loss functions we should worry about. It's gain functions.
A sequence of AI models trained on each other's output gets mutations, which might help or hurt, but if there's one dominant model at any given time then it's like asexual reproduction with only living descendant in each generation (and all the competing models being failures to reproduce). A photocopy of a photocopy of a photocopy — this seems to me to also be the incorrect model which Intelligent Design proponents seem to mistakenly think is how evolution is supposed to work.
A huge number of competing models that never rise to dominance would be more like plants spreading pollen in the wind.
A huge number of AI there are each smart enough to decide what to include in its training set would be more like animal reproduction. The fittest memes survive.
Memetic mode collapses still happen in individual AI (they still happen in humans, we're not magic), but that manifests as certain AI ceasing to be useful and others replacing them economically.
A few mega-minds is a memetic monoculture, fragile in all the same ways as a biological monoculture.