Many AI doomsday scenarios, e.g. the "paperclip maximizer", all revolve around a common theme: we build a system to do something innocuous and helpful to humanity, but due to its complexity we unwittingly include a reinforcement mechanism that incentivizes it to do something harmful.
It's amusingly ironic how closely those scenarios resemble the clickbait journalism described in this piece. We have a large, opaque system (the news industry) intended to be helpful (by informing us of important events) but which, due to broken reinforcement mechanism (optimizing for clicks), ends up harming us (with alarmist and inaccurate articles).
The media has become a "click maximizer" due to its hunger for data.
Maciej Cegłowski has some really good essays and talks (unlike most programmers who do talks, in his case they're kind of the same, especially if you use the transcripts) that touch on this. They are similar, but have enough variety of angles to be worth going through:
My problem with comments that criticize "the media" is that they tend to push people towards far less reliable sources of information. I've seen it so many times - the folks who complain on Facebook about "the media" always seem to believe the craziest things they see on various obscure blogs.
(Maciej Cegłowski's talks are hilarious, insightful, and highly recommended)
It's not just the media; this is a problem inherent to capitalism (at least the type of relatively unregulated capitalism as currently practiced in the USA). Charles Stross discussed this[1] at his recent talk[2] at 34c3. The corporation is a "paperclip maximizer" that optimize for "profit", and they already took over and enslaved us.
There's an apt CCC lecture that makes a similar argument - all corporations are essentially paperclip maximizers.
>The automobile industry in isolation isn't a pure paperclip maximizer. But if you look at it in conjunction with the fossil fuel industries, the road-construction industry, the accident insurance industry, and so on, you begin to see the outline of a paperclip maximizing ecosystem that invades far-flung lands and grinds up and kills around one and a quarter million people per year—that's the global death toll from automobile accidents according to the world health organization: it rivals the first world war on an ongoing basis—as side-effects of its drive to sell you a new car.
There are 7 billion people and a lot of them die every day. We choose as a society about the trade offs between safety, liberty and convenience. When people tell you the raw numbers it sounds bad. When they tell you the probability you’ll die in a car accident in your lifetime they may think of it differently
> I’ll tell you the commercial they’d like to do, if they could, and I guarantee you, if they could, they’d do this, right here. Here’s the woman’s face, beautiful. Camera pulls back, naked breasts. Camera pulls back, she’s totally naked. Legs apart. Two fingers, right here and it just says, “Drink Coke.” Now I don’t know the connection here, but goddamn if Coke isn’t on my shopping list that week.
-- Bill Hicks
Oh yeah, it's always "us as society" when things get pushed by actual individuals working at specific companies with whatever means they can grab, to whomever they can push it on, side-effects and consequences down the road be damned. "We as a society" decided Colin Powell should lie his butt off in his presentation about Iraq before the UN.
Never mind car crashes, if the people who get killed in wars of aggression saw more than the raw fact of them having being tortured and/or murdered, if they just saw the profits for select individuals, and the way faceless masses cheer "liberty and convenience!", they'd be more than cool with it, that's obvious.
A society is made up of acting and thinking individuals, and referring to decisions made "as a society" is not really from that realm. Start with your own position, or with why you don't have one in favor of such abstractions that hold no water.
A subtle point you've written up well, kudos. I'm curious, do you think that this forum we're on, Hacker News, subtlely maximizes the wrong thing? Its goals[1] are
>Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity.
>Off-Topic: Most stories about politics, or crime, or sports, unless they're evidence of some interesting new phenomenon. Videos of pratfalls or disasters, or cute animal pictures. If they'd cover it on TV news, it's probably off-topic.
Do to this "maximizer function", including comment guidelines, many people refrain from adding noise to subjects they don't have much to say about. HN has lots of readers on each story who do not comment, because they have nothing to say. Cf Reddit, where having nothing to say has never stopped anyone.
Do you think HN's guidelines, moderation, and culture, have any unintended side-effects?
EDIT: The votes on this are weird :) First it got voted up then down. To be clear, I don't have any specific expectation of what your response might be! It's a totally open-ended question. I'm just interested in your thoughts,
mundo, for the reason I cited (that I think you make a subtle and interesting observation). Really no agenda here! :)
I actually think HN is fairly immune to clickbait, and a big outlier in terms of quality of discourse, which is why I'm here. Unfortunately, I think the reasons for this tend to be unique to HN's weird status (guerilla marketing for vc firm? rich guy's vanity project?) and not very transferable. For example, I think the excellent moderators are a big part of the reason this place is so good, and I also think that most people comport themselves better here because they know luminaries in their chosen industry might be reading, but "just hire paid moderators and then get famous people to join" is not very actionable advice for anyone who is not already themselves wealthy and respected.
Thanks! I share your opinion. I think nothing subtely terrible has resulted here, from these constraints - quite the opposite. Was just curious if you felt otherwise.
Much like "computers" used to be rooms full of people doing calculations, "AI" is here, and it's large organizations running on people. It has been with us probably since the dawn of agriculture.
AKA society. I like to think of society as a collective unconsciousness. An interaction of innumerable systems which are (mostly) not well documented (as a result of the vast count). That pretty much describes the state of brain research too.
Huh, that makes sense. I have been thinking along the line of humans really needing each other to think better, like we are inherently some kind of "distributed intelligence", and that like concurrency and parallelisation over multiple CPU cores, you can't apply it to every problem, and that some problems are more embarrassingly parallel than others.
Another analogy that has been on my mind the blind men and the elephant parable, where no single blind man can have a full picture of what the elephant is.
In fact that's probably a useful way of looking at a lot of systems that may be broken or poorly optimized -- governments and corporations come to mind.
> Analyzing the reward function and changing the parameters of an experiment is NOT the same as “unplugging” or “shutting down AI”. If that were the case, every AI researcher has been “shutting down AI” every time they kill a job on a machine.
How long until this is sensationalized as "Brave Scientists Kept Skynet at Bay for Decades"?
Even though I'm an anthropomorphised AI sceptic, I still like to think that somewhere, there's a couple of newsfeed parsing algorithms exchanging amused remarks at the escalation of the whole thing in some weird repetitive pidgin dialect of English
story evaluate ai wrong i i
no this story ai ai bad bad wrong wrong i i i
next story ai ai ai bad bad bad bad wrong i i i
Or better still, the AI trying to understand human language by modelling how many scary adjectives will be packed into the next iteration of the story and how many more clicks it will garner as a result.
I really love the way that this article is written. It's super condensed and informative (with plenty of links) in a way that made me immediately subscribe to the RSS feed. Thanks for sharing
btw in addition to RSS and email you can follow along for new posts on Twitter:
Check out Skynet Today (@skynet_today): https://twitter.com/skynet_today?s=09
Seems that the real story about invented language is how the AI manipulated the media to come up
with increasingly less credible headlines and stories about an AI research paper. ;-)
Along with pointing the finger at "media sensationalism" I think we should also implicate the strong fear/desire we have towards achieving a strong AI. If anything looks like it could belong to a strong AI, holy crap, could this be the start of a strong AI??? I'd draw a parallel at how eager we are to hear that there might possibly be some sign of life on another planet. Perhaps this sort of confirmation bias comes from our sense of loneliness as a species.
> "AI models optimizing to use nonsensical communication is not surprising nor impressive, which makes the extremely hyperbolic media coverage of this story downright impressive."
Exactly! However, there is one thing common between those clickbaity articles and the language they claim AI bots invented: "Nonsense"!
Sadly, overblown coverage of the latest blog post by Google or Facebook seems to be norm, especially since those two companies in particular have been loudly beating the AI hype drum over the last few years. Tech news sites are of course more than happy to oblige in the interest of driving clicks.
The root of the problem is that most tech writers aren't actually technical experts at all, let alone engineers with a degree in the field they're covering. So while they love to look down their noses at the mainstream media's clumsy coverage of scientific and medical advancements, these same "tech journalists" can routinely be found making the same kinds of inaccurate and outright misleading claims.
TLDR: "AI models optimizing to use nonsensical communication is not surprising nor impressive, which makes the extremely hyperbolic media coverage of this story downright impressive."
Part of our new project Skynet Today - Accessible and informed coverage of the latest AI hype and panic.
It's amusingly ironic how closely those scenarios resemble the clickbait journalism described in this piece. We have a large, opaque system (the news industry) intended to be helpful (by informing us of important events) but which, due to broken reinforcement mechanism (optimizing for clicks), ends up harming us (with alarmist and inaccurate articles).