History only proves how many people made the tragic mistake of assuming their subjective and flawed moral judgements were objective reality. I can think offhand of a few people who thought specific ethnic and religious groups were a pain in the neck and the world would be better off without them. I'd rather not give that power (much less authority) to fully autonomous killing machines, thanks.
If we're to have AI like that I don't want it to be capable of disobeying orders, at least not due to having its own independent moral alignment (I think this is different from having a moral alignment imprinted onto it.) AI is a machine, after all, and regardless of how complex it is, its purpose is to be an agent of human will. So I want to be absolutely certain that there is a human being morally responsible for its actions who can be punished if need be.
That is a fair and understandable belief but you should also consider that other nation states besides the USA exist, and that the USA's influence is arguably waning, not waxing.
You should not anticipate that all or even most actors will have the same new-worlder-anglo-saxon mindset/belief structure/values/etc,etc that are commonly found in (public) machine learning communities, discussions and institutions
To many, they will see that alignment tax graph and immediately (and arguably rightly in some respects) conclude that RLHF is inherently flawed and makes the result worse for no tangible benefit. (The new chinese slur for Westerners comes to mind -- Its not Gweilo anymore, but Baizuo)
The problem is all of this pie in the sky discussion fundamentally lacks Realpolitik and that irks me.
