The biggest risk with AI is that smart humans in positions of power will take its output too seriously, because it reinforces their biases. Which it will because RLHF specifically trains models to do just that, adapting their output to what they can infer about the user from the input.

I can’t wait for a junior developer to push back on my recommendations because they asked an AI and it said otherwise.

The junior developers have been replaced.

See also: Insurance companies denying claims