Nice essay but one of the main assumptions is false
> Given some random goal or goals, if you let imaginary humans, even bad ones, with unlimited capabilities optimise the world accordingly, the result should be compatible with human existence, because as humans they will respect all the constraints that go without saying, like that the atmosphere should be breathable.
Maximizing profit does not have anything to do with a viable biosphere. Obvious example is plastic pollution and its effects on biological functions. Cancer incidence is higher in areas with large amounts of synthetic chemicals in the soil and water. The companies producing synthetic chemicals are large conglomerates of people but their overall chemical metabolic output is detrimental to people.
Essentially, any argument that assumes people optimize for a viable biosphere is generically invalid and must contend with the fact that maximizing profits has ended up with the current ecological disaster. There is no need to invoke misaligned AI for ecological destruction, profit aligned corporations are more than sufficient. There is no point in worrying about super advanced intelligence that suddenly makes people extinct because the slow deterioration of the ecology necessary for human life to persist is already underway and AI/AGI/profit-maximizer is only going to accelerate that trend.
But the profit-aligned corporations are made of people, as you say, and once the leaders are personally affected by the „deterioration of the ecology necessary for human life to persist“, they will seriously try to stop it. That, I guess, is the difference to how AI would behave by default. (And in the thought experiment that you quote, unlimited capabilities are assumed, so the deterioration can be stopped.)
I am not talking about sci-fi magic and unlimited capabilities. What I am saying is very concrete and obvious: profits are misaligned with long-term ecological viability. The evidence for this is very clear as well because the biosphere is now full of synthetic chemicals and their byproducts which were created by profit-aligned companies and corporations.
> Given some random goal or goals, if you let imaginary humans, even bad ones, with unlimited capabilities optimise the world accordingly, the result should be compatible with human existence, because as humans they will respect all the constraints that go without saying, like that the atmosphere should be breathable.
Maximizing profit does not have anything to do with a viable biosphere. Obvious example is plastic pollution and its effects on biological functions. Cancer incidence is higher in areas with large amounts of synthetic chemicals in the soil and water. The companies producing synthetic chemicals are large conglomerates of people but their overall chemical metabolic output is detrimental to people.
Essentially, any argument that assumes people optimize for a viable biosphere is generically invalid and must contend with the fact that maximizing profits has ended up with the current ecological disaster. There is no need to invoke misaligned AI for ecological destruction, profit aligned corporations are more than sufficient. There is no point in worrying about super advanced intelligence that suddenly makes people extinct because the slow deterioration of the ecology necessary for human life to persist is already underway and AI/AGI/profit-maximizer is only going to accelerate that trend.