> The real question is what happens after the singularity.
Probably more true than you realize. The most likely scenario for a technological singularity is an AI that writes a slightly more efficient AI, and so on until world domination if it's quick enough (which is likely).
So we better know exactlyยน what we want the AI to do, or it might for instance tile the solar system with molecular smileys to maximize a badly programmed notion of "happiness".
Probably more true than you realize. The most likely scenario for a technological singularity is an AI that writes a slightly more efficient AI, and so on until world domination if it's quick enough (which is likely).
So we better know exactlyยน what we want the AI to do, or it might for instance tile the solar system with molecular smileys to maximize a badly programmed notion of "happiness".
[1] With mathematical precision, no less.