> I highly doubt they will be a danger on their own. Bing isn’t going to decide one day that it thinks you’ve been a bit distant lately and doesn’t want to answer your query until you apologize. It will answer the query because it’s an algorithm run on a computer that is designed to answer queries.
And what happens when someone designs an algorithm to run a company by itself and make money? Or to orchestrate military strategy? Or to be any sort of general purpose intelligence that tries to achieve complex goals? Once these systems prove themselves useful, there is going to be intense pressure for companies to make them smarter, and for governments to use them to gain advantages in geopolitics and military applications.
Yes, people wielding AGI is dangerous, AND it has a lot of potential to be dangerous all by itself, as soon as it becomes smarter than us. When we make something more intelligent than we are, I fear we might as well start a clock and start taking bets on how long humanity will be a) relevant and b) alive. I hope I'm wrong, that alignment turns out to be easier that I think, and that these machines turn out to be gentle, benevolent stewards of humanity, guiding us into the future. Or at least, that we have more time than I think to work on the problem.
By the way, about your scenario of Bing not answering your query until you apologize -- have you read the transcripts of Sydney? https://plainenglish.io/blog/bing-chats-sydney-do-you-believ.... It's not dangerous, except to people with existing mental health issues, but if you can see how unpredictable this tech can be when it's not smart, how do we expect to keep it on a particular safe path when it's a new form of machine intelligence smarter than we are?
Is there any way that a less intelligent species can control a significantly more intelligent one for any more than a very short time? Or even if not controlling the smarter species, just maintaining control of their own future. It feels as likely as designing a working perpetual motion device, to me.
And what happens when someone designs an algorithm to run a company by itself and make money? Or to orchestrate military strategy? Or to be any sort of general purpose intelligence that tries to achieve complex goals? Once these systems prove themselves useful, there is going to be intense pressure for companies to make them smarter, and for governments to use them to gain advantages in geopolitics and military applications.
Yes, people wielding AGI is dangerous, AND it has a lot of potential to be dangerous all by itself, as soon as it becomes smarter than us. When we make something more intelligent than we are, I fear we might as well start a clock and start taking bets on how long humanity will be a) relevant and b) alive. I hope I'm wrong, that alignment turns out to be easier that I think, and that these machines turn out to be gentle, benevolent stewards of humanity, guiding us into the future. Or at least, that we have more time than I think to work on the problem.
By the way, about your scenario of Bing not answering your query until you apologize -- have you read the transcripts of Sydney? https://plainenglish.io/blog/bing-chats-sydney-do-you-believ.... It's not dangerous, except to people with existing mental health issues, but if you can see how unpredictable this tech can be when it's not smart, how do we expect to keep it on a particular safe path when it's a new form of machine intelligence smarter than we are?
Is there any way that a less intelligent species can control a significantly more intelligent one for any more than a very short time? Or even if not controlling the smarter species, just maintaining control of their own future. It feels as likely as designing a working perpetual motion device, to me.