All of these arguments seem to think that the brain isn't also generating a statistical ordering of semantic words and actions. Obviously the brain has a way more complex network of subsystems with various specialties than the artificial models we have today and is able to generalize language, math, and positional reasoning and mix it with the older parts of the brain for reward and training mechanisms and it can do it in real time, but it kind of seems like modern models are showing so much flexibility at different tasks and so much generality it's feeling like they have something in common with the building blocks of the brain that might get us there.
And even if something capable of all the aspects of human cognition were created with no internal resemblance to cognition, so what, it still works. There's a reason it's called "artificial" intelligence, not "natural intelligence identical to human brains except in silicon instead of neurons".
It's like when the wheel was invented saying it's not "true" transportation because it doesn't involve legs, or when the phone wasn't invented that it isn't "true" communication because you don't see the other person's mouth or physical expressions.
I agree that you shouldn't dismiss an airplane by the fact that it doesn't flap its wings, but we also don't call them artificial birds. The debate is around the word 'intelligence', which is misleading.
The human brain has always been compared to our most advanced technologies. It used to be compared to the telephone, then digital computers, and now to deep learning models. At least deep learning model has some merit because artificial neural networks are in part inspired by a very simple model of how our brain neurons work in a network, or rather how neuropsychologists would explain reinforcement learning to a computer scientist.
However this is a great simplification, and borders on an absurd reduction. You can model our brains using linear algebra, however that doesn’t mean our brains are linear algebra computer. There is a whole lot more going on than neurons receiving feedbacks from other neurons which adjusts the weight for subsequent firing. A lot of our behavior is actually inherited (I know I spent a whole week here on HN arguing with IQ advocates on the nuance of that statement), neurochemicals and hormones add a whole another level of statefulness not seen in artificial neural networks, the brains ability to make computations is actually pretty limited (especially next to a GPU). I mean, cordiseps exists, meaning a fungus can infect an organic system and control its behavior, there is 100% chance that some yet to be discovered viral and bacterial agents, not sharing any of our DNA—and certainly not “connected” to the “weight matrix”—are also influencing our behavior (just not as dramatically), and there is 100% chance they interact with our DNA also controlling our “innate” behavior.
What is going on in our brains can only be modeled using statistical ordering of semantic words and actions. The real world brain is always going to be infinitely more complicated than this model.
We just can’t accept that we might solve ourselves. People are understandably desperate to understand their experiences as more than an encoding of a thing that might be explained.
And all of our surprising wins and awful mistakes had explainable reasons, dammit; it wasn’t just a misfiring of trained statistical networks!
In addition, we shouldn't draw false equivalence between not knowing how brains work and not knowing how LLMs work, and concluding they must be similar.
Also, it's always funny to notice how the brain, throughout history, has always been compared to the latest technology available. For a long time people said "the brain must be similar to a clock".
Wouldn’t you say that this is how evolution and the spread of information works, in a system that has the spark of some life chemistry?
Of course… The hole in that theory is that evolution never found the wheel.
The steel man in that theory is that it invented the neurological and social processes that then went on to invent the wheel. And the platypus and the clap.
Edit: I forgot to bring it back around and make a point ;)
I’m saying that humans invented clocks and CPUs. We only have metaphors that have emerged from the still misunderstood ether of the informatic universe.
Only in a very anthropocentric sense. How do we know an ant colony doesn’t reason abstractly (or a human town for that matter)? What about slime mold or ameba? Both can solve a maze as well as humans. What makes you think a forest ecosystem isn’t capable of abstract though?
It is only if we narrow thought to mean precisely human-like thought when humans and human creations are uniquely capable of something. To that extent, our models of intelligence is very much in the pre-copernican era.
The fact that both humans and LLMs can both reason abstractly is an uninteresting fact if we define “abstract reasoning” to be exactly what humans do, and then create models with the goal of recreating exactly that. This is than simply a statement of an accurate model, and the word intelligence is there only to confuse.
This would be like finding a flower which produces a unique fragrance, then create a perfume which approaches the same fragrance and then conclude that since these are the only two things in the universe which can create this fragrance there must be something special about that perfume.
i would define abstract reasoning as composing and manipulating a model of reality or other complex system in order to make predictions
> is an uninteresting fact if we define “abstract reasoning” to be exactly what humans do, and then create models with the goal of recreating exactly that
if you find this uninteresting, we have perhaps an irreconcilably differing view of things
Your definition excludes language models, as they are in and of them selves just a model which interpolates from data (i.e. makes predictions). But your definition also includes lots of other systems, most mammalian brains construct some kind of models of reality in order to make predictions. And we have no idea whether other systems (such as fungal networks or ant colonies) do that.
I’m not saying these language models—or my hypothetical perfume—aren’t an amazing feat of technology, however neither has any deep philosophical implications about shared properties other than the ones constructed to do so. Meaning, even if LLMs and humans are the only two things in the universe that can reason abstractly in the same way humans do, that doesn’t mean the latter has any more properties shared with the former.
How do you know fungal networks don’t write and read poetry under the forest floor? If they do—and we have no reason to doubt that they do—you wouldn’t be able to read them, let alone understand them.
The earth’s biosphere as a whole also writes code, just in DNA as opposed to on silicon transistors, why exclude the earth’s biosphere from things capable of abstract thought?
For an example, crows can effectively use tools and communicate abstract concepts to one another from a memory. Which means they can observe a situation, draw conclusions, and use those conclusions to act as well as make decisions on how to act. That would seem to meet the bar for reasoning abstractly.
Not knowing how something works doesn't necessarily preclude replicating its function a completely different way. You don't have to understand an induction cooktop to hang a pot of water over a campfire.
I find this quite similar to the issue of free will: If we live in a generally deterministic universe, where is the space for independent decisions of individuals? Was every single decision we take already predetermined before we were born? A lot of ink has been spent on this topic, and as far as I know, only a small minority of people actually deny free will. One assumption I have is to why is because it absolutely sucks from an emotional point of view, and it is a terrible idea to base fe. a justice system on.
Maybe it makes me weird, but the idea of being determined, but still having to follow those emotional and cognitive tasks to moral conclusions is some kind of “work” we still have to do, to get good outcomes.
That’s still the hole in my theory of consciousness. I admit it ;)
But it doesn’t give me as much cognitive dissonance as others to believe that the process of performing moral actions still has to be “processed” by me, a processor. In some sense.
My understanding of how physics works is that due to the probabilistic nature of quantum physics, you can’t predict the future perfectly. So your decisions were not pre-ordained even though they were the result of physical processes happening in your brain.
I think people look more desperate to hype these LLM toys, that they are not the next blockchain or self-driving car. When they fail it's just excuses like you are not using the latest version or "prompting" them right.
The LLM value add for coding is less than the value add of syntax highlighting in my experience.
In my experience and it's certainly useful for unit tests speeding things up for me pretty dramatically.
Also when working on a new development language was quick to point out "how you do that" with the inline vars I needed manipulated vs look ups in Google and copy pasting.
Alternatively, we just can't accept that we might not solve ourselves. People are understandably desperate trying to find an explanation for everything, but can't admit that's just not ever going to happen.
> We just can’t accept that we might solve ourselves.
To solve ourselves is to know ourselves completely, and to know ourselves completely is to be honest in who and why we are what we are simultaneously across all persons. It assumes perfect knowledge.
There is no statistical approximation nor computational power which can do this.
> People are understandably desperate to understand their experiences as more than an encoding of a thing that might be explained.
Another way to frame this is, "some people are nihilists and do not see life as more than an encoding of a thing that might be explained."
To know anything (an X) completely you need perfect knowledge. Hence people come up with a set of simplified ways of reasoning about X. They call this a model of X.
Model is incomplete and so primitive, so dumbed down, that we manage to play it forward/backward in our heads or our computers.
If a model checks out with the real outcomes we proudly exclaim that we understand X.
I'm not being sarcastic, that is just a real method we use all the time.
> To know anything (an X) completely you need perfect knowledge.
Agreed.
> Hence people come up with a set of simplified ways of reasoning about X. They call this a model of X.
A "set of simplified ways of reasoning about X" in order to create a "model of X" does not imply complete understanding. Quite the contrary actually.
To wit, science often models current understanding of a phenomenon. When new evidence (understanding) is discovered, the model is updated to account for it. Sometimes this invalidates the original model, often the model is refined. Either way, progress is made with the tacit agreement that the model may change in the future.
> If a model checks out with the real outcomes we proudly exclaim that we understand X.
Again, this does not support the assertion of "That's been solved long ago." If anything, it affirms there is justification for disagreeing with the original premise to which I responded:
> We just can’t accept that we might solve ourselves.
> All of these arguments seem to think that the brain isn’t also generating a statistical ordering of semantic words and actions.
There may be a part of the brain that is modelled well by an LLM, but if so, there seem to also be parts that aren’t, or even existing “multimodal” models like GPT-4, which is more than an LLM.
Humans, including their brains (both in the narrow sense and in the broader sense of “everything that contributes to cognition”, which may extend beyond the brain proper) are machines, and their function probably will, someday, be mimicked by machines. But they are still more complex machines that modern “AI” systems
Agreed. One of the more interesting things about AI is how it forces humanity on a trip to consider what it means to be human. There is no soul, creativity or inspiration. There's only (very complex) agency.
> To me it shows how fundamental language is to intelligence.
I believe most human intelligence is based on language. We are smart because language gives us the keys to solving all sorts of situations based on the experience of prior generations. We are wondering if GPT architecture can generate real understanding, but forgetting the training corpus. The language corpus is the real magic here. The source and accumulation point of intelligence.
Take a baby and give it access to language, you get a modern human. Take a random init neural net, use language corpus for training, you get GPT4. It doesn't matter what the model is - brain or transformer, it doesn't matter if it's a transformer or just a RNN (RWKV). What matters is the training signal.
We should not get so hung up on the model architecture and substrate when thinking about LLM capabilities. Language is a separate evolutionary system of ideas and concepts. It is a self replicator, like DNA. Language just got a new pathway for self replication with LLMs. Every new token visits the whole model and by extension the distillation of our culture before being emitted.
>We’ll need to break out of the Chomsky hierarchies and develop some new theories of language.
We might need to develop some "new theories of language", covering other areas, but why exactly do you think we'd need to "break out of the Chomsky hierarchies"?
Nothing about LLMs challenges Chomsky hierarchies.
They do challenge Chomsky's ideas about language being an innate human quality, but that's about it. Besides that's not related to the Chomsky hierarchy of language, e.g. regular, context-free and so on. Those will continue to remain a hierarchy describing languages based on capability levels, whether there's AI, and AGI, or not.
The Chomsky hierarchy will remain unchanged and very useful for constructed languages. I didn't mean break out, as in create a higher level. LLMs are not going to magically become more than a turing machine or anything. I meant we need to stop thinking about spoken language in terms of the hierarchy and try to find another approach.
You can't tell me with a straight face that actual spoken language fits well in that hierarchy. It seems we can shoehorn it in with a 100B parameter turing machine definition, but that just makes my point that the constructed language hierarchy is not well suited for describing spoken languages if it takes such a large definition.
We should be able to find something like a new probabilistic theory of language that will do a much better job of describing spoken languages. Such a theory could help explain some LLM behaviors we see, and apply generally to other forms of noisy communication.
You're just describing information theory. Every message is contextualized within the codebooks used to encode/decode it. Sometimes we call that codebook "language", sometimes it uses another representation.
Yes, we should develop a new theory of language based on an information theory approach rather than a discrete symbol approach, or maybe something involving both.
But you can’t say information theory as it stands gives us a completely satisfactory model of human language. Something will need to be built on top of it.
>There is no soul, creativity or inspiration. There's only (very complex) agency.
That's highly debatable. For starters, who said there's no creativity or inspiration in humans, or for that matter, that there can't be in a complex A.I.?
How we achieve that creativity or inspiration is irreleant, as long the entity (human or AI) showcases creativity and inspiration.
Nor is it much clear why all being "only (a very complex) agency" would preclude creativity and inspiration.
That's like a worse version of "a human can't be creative or have feelings because it's all a bunch of molecules".
There is such a thing as emergent properties.
>And that's completely fine.
That's also highly debatable. I mean, that it would be "completely fine" if you were right and there wasn't "creativity of inspiration".
There are deep philosophical implications, just not these.
There has been a long history of reifying the behaviors of living things in general (e.g. vitalism) and humans in particular (e.g. dualism).
The success of LLMs challenges a lot of philosophy dealing with what behaviors are and are not possible in the absence of these categorical districtions.
I have had casual debates years ago, in which strong dualists asserted that the kind of creativity exhibited by today's LLMs is simply impossible. No doubt those folks are busy inventing "special philosophical creativity" that LLMs "aren't really doing," but they've lost credibility.
LLMs have demonstrated that there was never any need to invoke categorical districtions between human behavior and math-as-implemented-by-physics. The gap is closed, there is no more room for gods.
Or you can say the opposite and claim that there is soul, creativity, or inspiration in everything, even in inanimate objects (basically panpsychism, in contrast to materialism). But regardless of what kind of monism you take, the advent of current statistical AI models forces us to reconsider the Cartesian dichotomy between body and mind, which forms the basis of liberal thought.
> But regardless of what kind of monism you take, the advent of current statistical AI models forces us to reconsider the Cartesian dichotomy between body and mind, which forms the basis of liberal thought.
Didn't we dispose of Cartesian dualism a long time ago? It may have been useful for disposing of superstitions which preceded it, but anyone who has observed the effects of a stroke or has taken a psychotropics understands that mind and body are not made of separate stuff, and I know of no modern system of thought that stipulates that as an axiom.
LLMs just demonstrate that emergent information structures at the complexity of human language need not be only biological.
Yes they seem to take the view into account, but this sentiment as if the question is settled and we aren't anthropically-projecting, or whatever the term is, is premature. Just because two processes produce a superficially similar result, does not make them the same process. What you just described after conceding that though the brain may be a statistical inference machine it still seems like magic.
It is telling that you talk about all those things happening in "real time." Ask any super-regarded philosopher, from Plato to Wittgenstein (yes I'm excluding Dennet et al), and that would be quite the hoot to point out.
Yep. Elevators used to be proof that machines can think. Then compilers, and chess, and go, and search, and …
The problem with AI is that as soon as it works, we stop thinking about it as “artificial intelligence” and it becomes “just automation”. Then AI moves to the next goalpost.
Early elevators used to need professional human operators (you sometimes see them in old movies). Stopping at a selected floor was something a machine was unable to do on its own. Until it was and elevator operators lost their jobs and we just took it for granted that machines could do it.
But compilers, chess, go and search are all proof that computers could think. We've been discovering as we scale up the hardware that those things appear to be converging to human intelligence with minor tweaks (turns out tree search for chess needed to be combined with matrices and we're most of the way there). ChatGPT can out-reason many people I know and can out-argue a fair number of comments I see on the internet.
If we took this comment at face value we're ending up with a definition of "think" that can't reason, play games or recall information - or it would be outdone by machines. Thinking obviously isn't very important!
The problem seems to be with defining good tests for intelligence. FWIW, because GPT4 answers have a detectable pattern, they should presumably fail the Turing test.
At some level, intelligence requires logic, rationality, and conceptualization, all of which are topics which have evaded clear definition despite millennia of philosophy directly addressing the issues.
When you say we need ‘good’ tests for intelligence, you mean ‘tests that humans can pass but machines can’t’.
You’re demanding this because you aren’t comfortable with the implication that a computer can pass our existing tests for intelligence, so you rationalize that with the comforting thought that those tests were not meant to identify intelligence. Tests like the SAT or the bar exam or AP English. Or tests for theory of mind or common sense or logic. Those tests aren’t testing for ‘intelligence’ - they can’t be. Because a computer passed them.
I suppose it's a bit of a scotsman argument, but the turing test is to see whether an observer can correctly guess whether the interlocutor is _human_, so by definition the test would pass if the other correspondent was human.
To the point underneath, humans do not answer in as predictable a way as ChatGPT. Your answer, for example, I am confident does not come from ChatGPT.
Edit: if I've horribly mangled the Turing test definition, please let me know
I just imagined we could look at the oldest example of intelligence in human history. In contrast with AI, our chauvinism has us tend to pretend even the earliest monkey had it, fish?, insects? etc If it can rub 2 sticks together it gets the diploma.
Turing test is easy, I had 2 chat bots talk about other users in the channel while besides some trigger words ignoring what those other users had to say. The human subjects got angry successfully which means it was important to them.
I had someone on HN state stockfish is intelligent. If that is your definition of intelligence sure GPT is also intelligent. I do not think that's a common definition though!
Point is that we move the bar every time computers reach it. At least in part because we want to keep feeling special. And in part because we go “Well that can’t have been the bar then”
I suspect even full AGI will be considered “just a machine” for many decades, even centuries, before it gains the same rights as humans. We love to find reasons we’re special. Look how long it took us to admit animals are intelligent.
For many humans, computers definitionally can’t be intelligent. It’s important to recognize that.
> able to generalize language, math, and positional reasoning and mix it with the older parts of the brain for reward and training mechanisms and it can do it in real time
most brains I run into don't do that much at all, mostly just existing and adaptation-execution
100% true. It is only ego that makes people think we are "unique" and different to AI. If one plugs current AI into a body with external stimuli such as wishing to mate, eat, etc, then the external observer will not be able to spot a difference.
Our minds are in fact same statistical models with a gradually declining ability to learn and driven by exogenous irrational goals to eat and mate.
I generally agree with you, but I think this argument overlooks the lack of an obvious reward function in large language models. GPT-N has no survival instinct, because there is no existential threat of death, no fitness function to optimize to extend its survival at any cost. Without this need to survive, there can be no motivation, and without motivation there can be no intent. And without intent, can there truly be any action?
Intent can be created by a combination of prompting and incorporating the model into a feedback loop of some kind: it has something it was tasked with (via prompting), and the feedback provides info on to what extent the task has been completed; as long as the task is incomplete it may generate more responses.
To crank it up a notch, the assigned task could involve generating itself subtasks which are handled in the same manner. This subtask generation could start to look a bit like will/intentionality.
Now consider if the top-level task is something like maximizing money in a bank account that pays for the compute to keep it running :)
(IMO this is still missing some key pieces around an emotion-like system for modulating intensities of actions and responses based on circumstantial conditions—but the basic structure is kinda there...)
I think this kind of argument is making a similar mistake. In the same way that there's nothing fundamentally special about the computation human brains do, there's nothing fundamentally special about our "fitness function" (to reproduce).
It's just hard coded, whereas GPT's is dictated. More or less anyway.
Also our "fitness function" or motivations & goals aren't even that hard coded. You can easily modify them through drugs.
The argument is that LLMs don't have any motivations or goals, unless a human prompts them. They're not trying to stay alive or reproduce. They don't get hungry, feel pain or loneliness. They're just complex tools.
I know. The implication of that argument is that the motivations or goals of humans are different from those prompts in some way that means that GPT is not "really" intelligent.
That's my point. It's raising some arbitrary human trait to special status, when it really isn't. Human goals are set by some external process too - evolution. And they aren't really even intrinsically fixed. They can be modified through drugs.
I think the strongest demonstration of that is post orgasm clarity (if you're a man anyway). Your whole motivation changes in an instant.
A tamagotchi can have these things programmed, the only reason LLMs don’t is because we didn’t code them to do it. Or train them in an environment to maximize those things.
If you give it a prompt telling it that it's controlling a character in a game that needs to survive or meet some other goal and give it choices of actions to do in the game, it will try to meet the goals given to it. Characters inside of GPT are perfectly capable of having goals.
All these things possible to encode. Novelty seeking of “curiosity”, or survival drive. LLMs are trained on next word prediction but there’s no fundamental reason we can’t go in other directions.
Whether you can be "curious while starving" might or might not be possible, but it is irrelevant to the point.
For starters, because the claim argued is that you can have motivation (say, through curiocity) even without a survival instinct.
And it's perfectly possible to have no survival instinct and yet not to be starting anyway. It's enough that you have access to food or are fed, to avoid starvation. So lack of survival instict is not the same as starvation.
If we substitite "starvation" for "access to electricity" (as AGIs don't eat food), as long as an AGI is provided by electricity by us, even if it has no survival instict, can still have curiosity, and thus motivation.
Assuming you're talking about AGI/consciousness/intelligence, then whether you're right depends on what you mean above by saying that "you cannot solve the problem".
- If you mean "you cannot EXPLAIN AGI/consciousness/intelligence if you don't understand it", then that's true, but it's a trivial tautology.
- If you mean "you cannot DEVELOP AGI/consciousness/intelligence if you don't understand it" then that's very debatable.
Historically we have been able to develop all kinds of things, despite not knowing how they work. Tinkering and trial and error is often enough.
After all that's how evolution solved the problem of creating consciousness/intelligence. There wasn't some entity that "understood" intelligence that created it.
The most difficult thing, which everyone in all these discussions try to avoid, is that we don’t even have a universally agreed upon definition for the words AGI, consciousness or intelligence. Let’s just agree to define what we’re talking about first before we start arguing about what’s possible or not regarding those undefined terms.
I don't think it's much of a showstopper either. Not having a "universally agreed upon definition for the words AGI, consciousness or intelligence" haven't prevented us from considering other people conscious and intelligent, or animals conscious, etc. It wouldn't preclude us considering an alien race coming to earth as conscious and intelligent either.
Sort of how we didn't have a "clear definition" of most things for millenia, but we still were able to recognize them as a class of thing. It's more of a "I'll know it when I see it" kind of thing.
And the external behaviors (of consciousness and intelligence) matter more than "but is it really conscious/intelligent inside" when considering some AGI as such. After all we neither can clearly define, not we know or can measure what's going on inside another person's head regarding consciousness, or to be frank, not even on our own head. When it comes to us, we just have a subjective experience, and not even a very clear one at that.
referring to the parent post, you seem to imply that evolution had a "problem" to which "creating us" was the solution. which to me isn't obvious at all. chance seems a perfectly satisfactory explanation for now
All that's needed to "tame fire" is to have a predictive-enough model of fire dynamics and a suitable method for constraining its state space. There's nothing that requires "understanding" in the scientific sense, only in the pragmatic sense.
Sure, so maybe all you need to solve intelligence is also a pragmatic approach, like datasets generated by a known intelligence from which a general learning function can infer associations.
Excellent observation. In fact the language part of the brain is only a (albeit a rather large) portion of the brain (not nearly as large as visual processing though). And people who suffer brain damage which renders them unable to speak (or understand speech; which interestingly is a different portion; albeit close to each other) are still able to demonstrate intelligent behavior with ease.
In fact it is damage to the prefrontal cortex (which has nothing to do with speech) which is mostly correlated with a detriment in intelligent behavior (suspiciously also social behavior; a food for though in what we consider “intelligence”). Victims of lobotomy had their prefrontal cortex destroyed, and their injuries resulted in them loosing their personalities and loosing basic function as human beings, even though they were still able (but perhaps not always willing) to speak and comprehend speech.
I don’t think you have an ‘arithmetic’ part of your brain.
What you have that LLMs lack is a visual part of your brain - one which can instantly count quantities of objects up to about 7. That gives you tools that can be trained to do basic arithmetic operations. Although you have to be taught how to use that natural capability in your brain to solve arithmetic problems.
And of course for more complex things than simple arithmetic, you fall back on verbalized reasoning and association of facts (like multiplication tables) - which an LLM is capable of doing too.
Poor GPT though has only a one dimensional perceptual space - tokens and their embedding from start to end of its attention window - although who’s to say it doesn’t have some sense for ‘quantity’ of repeated patterns in that space too?
It statistically outputs the next probable value of text. There is a lot of math related text in the CommonCrawl (likely the biggest source of it's training). That is all the company who owns it claims that it does. There is no calculation, therefor no basic arithmetic being done correctly.
emergent behavior does happen with neural networks. It is correct to say that GPT 4 doesn't do math correctly, but it may be incorrect to say a large language model can't in principle.
Yeah that's not true lol. It's arithmetic is not perfect (mostly multiplication, addition is fine) but there's nothing random about even the wrong/non-exact numbers
how am i supposed to have a conversation about someone who is gassing up "not perfect [arithmetic]" (something a wrist watch from the 80's can do) and won't even believe what the creators of said machine say about how it works
You probably used GPT-3.5. That said, I didn't manage to get GPT-4 to calculate 36,330 * 26,951 correctly. I suggested casting out 9s, casting out 11s, doing long multiplication, reversing the digits - nothing.
I have a theory that it does arithmetic badly because the logic goes right-to-left, when LLMs write left-to-right. If the digits were to be reversed, it might not make as many mistakes. I ran out of attempts before I could test this properly.
> Human: what is 36,330 26,951 * (input is expressed in reverse post fix notation)
> chatGPT: To evaluate this expression using reverse Polish notation, we need to use a stack to keep track of the intermediate results. Here's how we can evaluate the expression:
1. Read the first number "36,330" and push it onto the stack.
2. Read the second number "26,951" and push it onto the stack.
3. Read the operator "", pop the top two numbers (26,951 and 36,330) from the stack, multiply them, and push the result (971,914,830) onto the stack.
4. The stack now contains only one number, which is the final result.
Therefore, 36,330 26,951 = 971,914,830 in reverse Polish notation.
You should ask it to show its working. The problem is that it uses a fixed amount of thinking for each character - and it writes left to right - while long multiplication is computed right to left. Past a certain number of digits, the amount of "thinking" needed to compute long multiplication in the wrong order (left to right) increases without bound -- accuracy suffers as a result.
A tangential point: Note that multiplication of decimal numbers _has_ to start with the rightmost digit to be done accurately. Consider x = 2/3 + 1/3 = 0.6(6) + 0.3(3), all digits repeating infinitely many times, so there's no rightmost digit: Notice that whichever digit you choose for x before the decimal point (either 0 or 1) could be wrong if you change any digit of 1/3. This is called the Table Maker's Dilemma.
The second paragraph is a tangent. My point was made in the first paragraph.
You said: "I dont think this is about notation" - Why did you ever think it was?
I was thinking along the lines of maybe it reads the digits as individual tokens and reading from left to right it get confused. That's why I thought it might be about notation. I have learned it's definitely not the case (read a bit about what tokens actually mean)
I also tried the show your work methods. I will not paste my conversations here as they are so long but even with showing it's work it doesn't give the right answer. Two funny things I noticed
1. It either uses a completely wrong method to calculate it, or it shows the right "kind" of work and then gets the wrong answer
2. If I tell it it was wrong and it will just agree with me. I multiply two huge numbers and tell it no the answer is actually 42, it will just apologize. Then when I ask the reason why it thinks it's right it will give the most bullshit things lol. Once it even said "I read the number wrong, the second number should be x I read y". The thing is if I had actually given x in the input it would have gotten the answer right.
That’s not ‘basic arithmetic’. Most humans would need a pen and paper to solve it.
GPT’s attention window is not equivalent to it being able to ‘write stuff down’ - it’s its equivalent of being able to remember the foregoing few minutes of conversation.
Interesting. I didn't use as large of numbers as you did, but in the low thousands, and ChatGPT w/ GPT-4 got them correct 3 times in a row.
What's also interesting, though not necessarily surprising, is how relatively close the incorrect answers are in your examples. It's not like the model will randomly spit out an answer like 42 if you ask it to multiply something like 36,330 by 26,951.
That computer scientists think that they are even close to replicating biology and the mammal brain with AI is complete hubris. This is biology inspired engineering, it's not like we are building brains out of silicon here.
Modern AI doesn't replicate biology except that humans find it much easier when you explain something as "Artificial Neurons" versus "Gradient Descent Back-propagation Trained Nested Mathematical Functions." Human neurons don't function anything like deep neural networks nor are the latter based on the former in their current state.
Training the brain seems a lot easier than training an AI wrt the number of iterations. You don't need to process billions or trillions of tokens to understand English.
GPT4 appears very intelligent when you discuss program code with it. It simply crazy; it can write SNOBOL4 code to extract a field from /etc/passwd.
When you discuss other things, it goes off the rails a lot more. For instance, I have had it quote some passages of classic English poetry to me, stating blankly that those passages contain certain words. The passages did not contain any traces of those words or even remotely similar words. In that situation, GPT4 was being dumber than /bin/grep, which can confirm that some piece of text doesn't contain a string.
GPT4 is deliberately trained as a coding technician, so that it will convince programmers who will become its advocates.
1. Programmers are vain and believe that everything they do requires great intelligence. (Not just some of it.)
2. GPT4 can "talk shop" with programmers, saying apparently intelligent things, perform complex refactorings, and intuit understanding of a piece of code with minimal context based on meaningful identifier names, and so on.
3. Thus, some programmers will falsely conclude that it's highly intelligent, like them.
To be sure, what the technology does is unmistakably a form intelligence. It solves complex problems of symbol manipulation and logic, in flimsy contexts. To deny that it is artificial intelligence is silly.
AI /= AGI
That's not where the goalposts are for the definition of AI.
Computers playing chess is AI. Google Assistant is AI. An expert system from 1990 is AI.
The problem seems obviously complex.
People need special education to do it, some never learn how to do it.
Making a system to do it (LLMs) is beyond most peoples understanding and even people who understand usually can't make one for a variety of reasons, e.g. money.
Are we all vain? I guess we are, but I find this take to be a bit too simplistic.
I do agree programmers are arrogant and think everything they do requires vast amounts of intelligence.
I’n not sure what your point is. I am honestly curious. You think having coders as major advocates for your product is somehow a brilliant strategy? If anything, this will work against it.
GPT4 can talk shop, and demonstrate useful code generation and refactoring, as well as understanding.
For instance, as an exercise, I had it write a maze-generating Lisp program that produces ASCII. I wanted it in a relatively little known Lisp dialect, so I described to it some of the features and how they differ from the code it generated.
For instance, GPT4 hypothesized that it has a let which can destructure multiple values: (let ((x1 y1) (function ..)) ((x2 y2) (function ...)) ...).
In plain English, I explained to GPT4 that the target Lisp dialect doesn't have multiple values, but that when we have a pair of values we can return a cons cell (cons a b) instead of (values a b). I also explained to it that we can destructure conses using tree-bind, e.g (tree-bind (a . b) (function ...) ...). And that this only takes one pattern, so it has to be used several times.
GPT4 correctly updated the function to return a cons cell and replaced the flat let with nested tree-bind.
At the end of the chat, when the code was almost working, I made a bugfix to the function which allocates the maze grid, to get it working.
I told GPT4: this version of the grid allocation function makes it work. Without it, the output is almost blank except for the vertical walls flanking the maze left and right. Can you explain why?
GPT4 correctly explained why my function works: that the function it wrote shared a single row vector across the grid rows, giving rise to sharing. It explained it like a computer scientist or seasoned developer would.
It's like a somewhat dim-witted, but otherwise capable coding clerk/technician, which talks smart.
With GPT4, you're Sherlock, and it is Watson so to speak. (Sorry, IBM Watson.) It can speak the language of crime investigation and make some clever inferences. In the end, you do all the real work of designing and debugging, and judging complex requirements against each other in the broader context. It saves you the keystrokes of doing the tedious coding you're used to doing yourself.
On the other hand, you expend some keystrokes explaining yourself. Some of the chat could be saved and used for documentation, because it captures requirements and why some things are done in a certain way and not otherwise (rejected decisions).
That's most likely due to the way it tokenizes words. You have to be careful here, or else for similar reasons you might imply that dyslexics aren't intelligent.
> GPT4 is deliberately trained as a coding technician, so that it will convince programmers who will become its advocates.
I'm a programmer and I've used GPT4 for a variety of tasks (well, tried to). The results have been mediocre on average, usually syntactically correct, but more often than not, semantically incorrect. It usually ends in frustration as GPT-4 keeps responding confidently incorrect answers and upon the slightest expression of doubt, it tends to spin in circles.
ChatGPT: <Implausible response #1>
Me: Are you sure? Reasons A, B, C [...]
ChatGPT: I apologize... <implausible response #2>
Me: Are you sure? Reasons D, E, F [...]
ChatGPT: I apologize ... <repeats implausible response #1>
I'd like to know what people who are so impressed with GPT-4's programming capabilities are doing? It must be TODO apps, solving leetcode problems, writing basic Express.js routers, some basic React components and code for <one of the top 10 popular libraries>. The kind of things ChatGPT has seen a million of examples of.
To use the tool effectively, you can't use Socratic questioning on it, or not always.
You have to already know how to code the problem, so that you can spot what is wrong, and, if necessary, tell the thing how to fix it.
The fix is not always going to come from the training data; it needs to scrape it from what you give away in your follow-up question, together with its training data.
I went through an exercise in which I encoded a paragraph from Edgar Allen Poe with a Vigenere cipher. I presented this to GPT4 and asked it to crack it. First I had to persuade it that it's ethical; it's a cipher I made myself and it's not my secret. We worked out a protocol by which it can ask me questions which show that I know the plaintext.
It was a quite a long chat, during which I had to give away many hints.
In the end I basically gave the answer away, and the thing acted like it cracked the key. Then I reproached it and admitted that yes, it didn't crack the key but used my hint to locate the E. A. Poe passage.
Basically, if you know the right answer to something, and GPT4 isn't getting it, it will not get it until you drop enough hints. As you drop the hints, it will produce rhetoric suggesting that it's taking credit for the solution.
But out of all the programmers I know, there are only certain ones that are embracing it, and the rest are still stuck in "I tried it and the code wasn't exactly what I wanted - it's all hype" land. And I think for one of the reasons you say: programmers are vain and believe everything they do requires great intelligence. A lot are still missing the forest from the trees.
I tried so much to use it for my day job but it's next to useless, it takes more time to get it to go where I want than to actually do it myself. My day job is far from requiring a great intelligence, but the tasks are too specific for chatgpt
I've used it for side projects though, especially front end stuff that I absolutely hate (js) and it works fine for that, but that's because I'm absolutely garbage at it in the first place and probably ask it to solve the most answered things ever on stackoverflow &co
I think a lot of it is due to the unresolved legal questions. Basically nobody can use it professionally yet. As soon as that happens I expect it to become a standard tool.
Although... on the other hand there are plenty of programmers that don't even use IDEs still.
It’s a bit concerning how many folks I’ve met at meetups talking about how they are using ChatGPT at work. The few I’ve pressed on it have said they’re not supposed to, but they do it off the company hardware so no one knows.
I got a coding challenge as part of the interview process. I had 2h to complete it, but I finished in 30mins thanks to ChatGPT. I wrote some test cases and told ChatGPT to generate more. I reviewed them and copied a few relevant ones. It also hinted me that I could add support for fractions.
> That's not where the goalposts are for the definition of AI.
We've well past that goalpost, like decades ago, if you haven't realized.
The simplest form of AI is just a series of if-else statements (expert system). A highly sophisticated network of conditional statements can make fully informed decisions way faster and more accurate than any human mind.
What the hell, we are looking at essentially millions of matrix multiplications done on powerful GPUs. And we're told this is what the brain does. Which is supremely stupid, since as Bernardo Kastrup has emphasized many times, our brains are completely UNLIKE those systems.
The hubris in Big Tech right now is truly pathetic and imho a sign of a real disconnect with a sense of meaning and purpose to human life.
In fact recently I was listening to a podcast with Donald Hoffman,. where he mentioned brain studies done on people with split brains etc and how these people did not even have a single consciousness. So even the assumption that we have like one "AGI" program running in our brain is not at all what goes on in our brain.
Likewise Iain McGilchrist pointed out how the left and right brain hemispheres are two kind of attentions to the world, and almost two different type of consciousness in us.
I quit listening to Rupert Spira many years ago but I found this interaction between him and Donald Hoffman really insightful. DOnald mentions the brain studies in this podcast:
The Convergence of Science and Spirituality | Donald Hoffman & Rupert Spira
The hubris in Big Tech right now is truly pathetic and imho a sign of a real disconnect with a sense of meaning and purpose to human life.
Many people in the developed world, especially tech people are now living online and spend the majority of their lives looking at a screen, chatting to each other on slack and via forums like this and now comparing their intellectual abilities to that of super computers running a language model of extreme complexity, which is basically the sum total of all human technical and intellectual output. All popular LLMs exist because of this online universe we’ve created.
What the online life does is actually reduce one to an “autonomous agent”. You really become ChatGPT when you live this way. Go out for a night of dancing, practice martial arts, paint a real panting with oil on canvas, cook a nice meal, smell and taste it, or hike a mountain, then come back and try tell yourself you’re a LLM, it’s completely preposterous but it does make sense if you live an almost purely “intellectual and online existence”. Which is the world we’ve created in 2023. Science and religion are now merging and it is said science will bring us “immortality” soon through AI advancement and many would immediately upload their brain to a computer if given the chance.
It completely makes sense to forget what you actually are and it makes sense to reduce yourself to “software running on hardware”, or a statistical model, a person who clicks like and dislike on social media most of the day, you become a model trainer etc. Even if there is some truth to the former statements, it’s stupid to discount the rest of existence because you can’t instantly translate text into many languages or whatever “emergent property” is displayed in some complex system.
The intellect is something which should be consulted or used for survival or problem solving, to help one get back to enjoying life and that what’s most important.
Intellect is not something one should strive to actually become.
> What the hell, we are looking at essentially millions of matrix multiplications done on powerful GPUs. And we're told this is what the brain does. Which is supremely stupid, since as Bernardo Kastrup has emphasized many times, our brains are completely UNLIKE those systems.
That's irrelevant. Transformers are Turing complete. Whatever mathematical system underlies human cognition will thus have some expression as a transformer.
These transformers are learning how to reproduce functions by inferring associations between inputs and outputs. The data whose function is being reproduced is the function in the human brain that produces human text. Therefore these transformers are literally learning how the human brain generates text, ie. this is what the brain does.
Furthermore, there have been neuroscience papers correlating spiking behaviour in neural networks and human neurons when processing the same information, lending even more credence to this hypothesis. It's not as far-fetched as you think.
> That's irrelevant. Transformers are Turing complete. Whatever mathematical system underlies human cognition will thus have some expression as a transformer.
That‘s irrelevant, too, because nothing indicates that the actual transformer of, say, GPT or the training is in any way connected to such an expression of the human brain.
Linux is written in C and that‘s Turing complete. And? Therefore Linux is like the brain?
> That‘s irrelevant, too, because nothing indicates that the actual transformer of, say, GPT or the training is in any way connected to such an expression of the human brain.
It's not irrelevant, it was merely the premise for the following point, which was that transformers are trained to learn a function. Since they're Turing complete, they can thus hypothetically implement any function, including the function governing human speech, as I described.
If transformers were not Turing complete, then they arguably would not be computationally powerful enough to model the brain which we know is Turing complete.
What is irrelevant is trying to dismiss transformers as intelligent, or like the brain, simply because they're "merely matrix multiplications".
Thanks for agreeing with my point, which is that all of the people claiming that these systems are NOT like the human brain have no clue what they're talking about, and that these systems very well could operate just like the brain.
The burden of proof is always on whomever is making a claim. I didn't claim transformer systems were like the brain, I simply showed the mistakes behind the arguments claiming the opposite, by showing how transformer systems could work like the brain.
Winner winner chicken dinner. Could sum the argument as: Transformers appear to be approximately emulating the brain despite us not fully understanding how either works "inside". Which is weird and eerie, but plausible.
Transformers are turing complete RNNs, as are the Neocortex.
There are differences like spiking vs non-spiking neurons, universal vs non-universal activation, the effects of brain waves, bicameralism, etc. But these differences may or may not be important depending on the topic of discussion. Additionally, the differences increasingly disappear as transformers are increasingly augmented with functionally equivalent systems.
I'd wager that intelligence is nothing but a reified concept. Something which doesn't innately exist other than as an abstract concept which we have defined, to help us explain complex nature, without the need of fully understanding the underlying complexity giving rise to that nature. Like how cave men used the concept of "gods" for a satisfying explanation for why it rains - we use intelligence to ascertain why something is able to do or achieve certain things.
But fundamentally speaking, evidence suggests that the nature of reality follows simple rules. That everything obeys simple algorithmic rules. And therefore, suggests humans are just as robotic as anything else, including the earliest of computers and current "AI". We don't look at it that way, as we don't fully understand the underlying complexity which gives rise to our behavior (whereas for computers we do), so we conveniently call ourselves intelligent as a way to appease the thirst for a satisfactory explanation for our behavior, and are dissatisfied with calling current AI, as true AI.
Thus, I argue, depending on what one wants to define intelligence, AI does not innately exist and can never exist if everything truly obeys simple rules, or, AGI has existed long before humans have come to exist as it's merely a concept which we are free to define.
There's a paper about that, "The Myth of Intelligence" by Henry Schlinger. Abstract:
> Since the beginning of the 20th century, intelligence has been conceptualized as a qualitatively unique faculty (or faculties) with a relatively fixed quantity that individuals possess and that can be tested by conventional intelligence tests. Despite the logical errors of reification and circular reasoning involved in this essentialistic conceptualization, this view of intelligence has persisted until the present, with psychologists still debating how many and what types of intelligence there are. This paper argues that a concept of intelligence as anything more than a label for various behaviors in their contexts is a myth and that a truly scientific understanding of the behaviors said to reflect intelligence can come only from a functional analysis of those behaviors in the contexts in which they are observed. A functional approach can lead to more productive methods for measuring and teaching intelligent behavior.
I have been thinking along the same lines - the fact that something as comparatively simple and barebones as an LLM can manipulate language symbols well enough to carry on a conversation suggests that it's a lot easier than was previously imagined. I used to think that language was one of the defining characteristics of intelligence, like the instruction set of a cpu, but chatgpt seems like persuasive evidence against that.
In order to predict the next token it’s doing something more like simulating the writer of the words and the context they were likely to be in while writing the words. You cannot make accurate predictions without understanding the world that gave rise to these words.
Consider a detective story with all the clues laid out and then at the end the detective says: “I know who it is. It is: …” Correctly predicting the next “tokens” entails that you incorporate all the previous details. Same goes for math questions, emotive statements, etc.
I’d be careful calling it simple. They might be simulating humans including for example a theory of mind just as a side effect.
Yes exactly. All of the dismissive ‘it’s just a fancy next word predictor’ articles can’t see the woods for the trees. Just because the function of the model is to predict the next word tells us almost nothing about what internal representation of the world it has built in order to generalise. I don’t for a second think it’s currently of comparable complexity to the world model we all have in our brains, but I also don’t think there’s any clearly defined theoretical limit on what could be learned, beyond the need for the internal model to make better predictions and the optimiser to find this more optimal set of weights (which might be a limit in practice as of now).
> You cannot make accurate predictions without understanding the world that gave rise to these words.
I think you must either admit that chatgpt does exactly this, or else give up our traditional connotation of "understand". Chatgpt has never seen a sunset, felt pain, fallen in love, etc, yet it can discuss them coherently; to call that understanding the world is to implicitly say that the world can be understood solely through reading about it and keeping track of which words follow which. It's amazing that generating text from statistical relationships about tokens in a corpus, which generated nonsensical-but-grammatical sentence fragments at smaller scales, can expand to concepts and essays with more compute, but it is just a difference in scale.
> I’d be careful calling it simple.
I'm not calling it simple, I'm calling us simple! I'm saying that chatgpt is proof that natural language processing is much easier than I previously thought.
Ok, but billions upon billions of “statistical relationships”.. I mean at some point the term “simple” loses its meaning. I get your point though. It is not pure magic.
Yeah, "simple" as in we didn't have to make something that can learn general concepts, and then teach it language. It feels like a hack, doesn't it? Like you were working on a problem you thought was NP-hard and then you stumble over a way to do it in O(n^2).
Yeah, I think we get sidetracked by how it “feels” to us when we learn. We forget that is just a convenient story that our mind tells itself. We are incapable or at least severely handicapped when it comes to raw experience and the knowing of it.
Somehow this approach to ML feels kind of natural to me, but it’s hard to articulate why.
“You cannot make accurate predictions without understanding the world that gave rise to these words.”
This depends on the definition of understanding. There are an infinite number of equations that could describe the trajectory of a ball being thrown, and none of them are exactly correct depending on how deep down the understanding hole one travels.
Our current ‘simple rules’ to explain nature can only account for a small % of the visible universe.
Assuming there are simple rules, we don’t know how, for example, an electron has the wherewithal to follow them (when does it read the rules or check on them, where are they stored, etc.). It’s mystery all the way down (unless you define it as simple using hand-wavey abstractions ;))
In the end we will find out that AI is not that intelligent and since it is so close to mimicking us it will say the same thing about our own intelligence.
Normally the materialistic attitude is based on a conceit that we basically already know everything worth knowing. Your position seems to be that we don't, but that when we do it will be equally empty and meaningless.
Yet you have undeniable proof that your own consciousness is as real as it gets, and that you experience life in a way that isn’t just an abstract concept. It’s absolutely there (at least, from your own point of view).
I’m not a religious person and I don’t believe in a soul or anything magical like that, but it’s just impossible for me to accept that I’m just a bunch of atoms following rules. I know that there’s something there, I see the evidence right before me, even if I can’t explain it or prove it to you.
The elephant in the room in these discussions of AI is the concept of the corporation. Most of the things people are worried about AI doing are things corporations already do. Deceptive marketing, exploiting customer information, conspiring to keep prices high, creating distractions from things companies are doing, setting up monopolies, lying to customers - the usual. With AI, this can be automated.
Most of this is ordinary consumer protection. Regulating AI means regulating corporations. Placing hard limits on what AIs are allowed to do means consumer protection with teeth. Nobody in power wants to talk about that too much.
As for AIs taking over, that will come via the corporate route. Once AI systems can outperform CEOs, investors and boards will insist they be in charge.
This is a great framing. I also appreciated this similar idea in Matthew Butterick's blog:
> Before too long, we will not delegate decisions to AI systems because they perform better. Rather, we will delegate decisions to AI systems because they can get away with everything that we can’t. You’ve heard of money laundering? This is human-behavior laundering. At last—plausible deniability for everything. [1]
While in a corporation there's still a person somewhere who could be held accountable, AI diffuses this even more.
> Before too long, we will not delegate decisions to AI systems because they perform better. Rather, we will delegate decisions to AI systems because they can get away with everything that we can’t.
Here's Frontier Airlines announcing proudly to investors that they will do exactly that.[1] See page 44.
Today: Call center. Avenue for customer negotiation.
Tomorrow: Chatbot efficiently answers questions, reduces contacts and removes negotiation.
This is the infuriating part of dealing with Amazon customer service these days. Anything that doesn’t fit into the box of what the chatbot will do is met with “I understand your frustration, …” and it’s like, no, there is no understanding here!
Just like call center trees have escapes to real service by pressing 0 ad nauseum or swearing loudly, these AI service agents will have ways to get to real people and they’ll be documented in the usual places.
How many times have you read about someone locked out of their <insert major tech company> account and having no recourse except taking it to HN and hoping a human on the inside reaches out?
Imagine if that was every corporation, and some of them had zero humans on the inside.
It is not a good future, and at least in Amazon’s case jeff@amazon.com works as an escalation route.
My gmail account with the same username got banned in 2007ish in the middle of college, I definitely feel the pain of no recourse. I lost everything - calendars, todo lists, email and had no way to get the underlying data or do anything about it.
My hope is that at the end of the day a corporation is made of people (legally, this is why they have 1st amendment rights in the USA) and this will prevent the scenario you imagine because the money of those people will be at risk.
Whilst it's interesting to imagine every company being run by the most aggressively rent seeking MBA-like AI, I don't think that will ever be the entire market.
I would guess it all comes down to risk. The future won't be uniform, it'll be poorly defused, just like then present.
People will make a value judgement on the types of corporations they want to deal with. The sort of company that no has employees will by necessity have to be very cheap compared to those that provide interactive human service.
Dealing with AI firms will be like dealing with an anonymous Chinese eBay seller. You'll effectively have no legal recourse as such and they won't discuss things but complaining might get you a replacement item or a full refund.
If it doesn't the product will have to be cheap enough to write off as a bad buy.
Let's hope cheap AI powered corps don't suck all the profit out of the market and we loose all firms capable of providing real services.
Or we end up paying for human service that's been quietly outsourced to AI much like we do for many big name manufactured goods.
Why would "the markets" provide that option? AI corporations will be phenomenally successful predatory value-extractions machines. Shareholders will vastly prefer them over more humane corporate structures.
This already happens. It's why PayPal, YouTube, Amazon, and the rest can shut down small-fry business accounts with no come-back. They're monopolies and they don't care. Because they don't need to.
The next stage will be shareholders using AI to make their investment choices. This will - automatically, with no recourse - drive money towards AI corporations while starving human-run businesses of investment.
Essentially AI is just an amplifier of existing economic and political trends - a programmable automated economic predator. Because many of those trends are dysfunctional and insane, we're going to have a lot to deal with.
Depends on whether you view humane business practices as a cost-center of a value-center though - in pure economic terms.
This is an unsolved question, because when you get down to it our businesses are still run by humans, and humans haven't changed much in the last 10,000 years from when we lived in tribes in Africa and were predated by large carnivores.
Look at the utterly insane culture which develops in places like LinkedIn and tell me that all executives are making calculated decisions with no ego, just efficiency. They're not - they're clearly not.
The fear of AI capitalism is the fear that it'll do to everyone what currently is done to the working class, given the opportunity. That's a realistic fear! But it's not guaranteed, because one of the significant arguments to stop abusing workers is that when you do, they're more efficient and more productive. Consider the delta between what the data says about WFH home and the complete conviction some managers have that despite this they've got to drag people back into the office (Blizzard Entertainment is a good example of manifestly creating problems because you won't let a bad idea go on this front).
The Chinese eBay seller brought to mind the concept of the Chinese Room[0] and how it is a reflection of current LLM’s. Perhaps also influenced by a recent read of Blindsight.
> we will delegate decisions to AI systems because they can get away with everything that we can’t.
This is one of the main reasons I quit the Facial Recognition industry; it is being used to delegate decisions, to remove responsibility of those decisions from those that need to be held accountable.
I worked as principal engineer of one of the top 5 enterprise FR systems globally, and the number of end-users fraudulently abusing the software blew my mind. Case in point: police called for a street crime, police ask the victims what celebrity their culprit looks like, police then put images of that celebrity into their FR software to collect suspects, followed by ordinary innocents who happen to look like celebrities being called into lineups and harassed by the police. And this practice is widespread!!!
That is just one example of how incredibly stupid people using our software will use our software, potentially harming large numbers of innocents.
Unfortunately even having humans in charge doesn't mean those humans will be punished for malfeasance. When was the last time you've seen an exec personally pay for their ad conduct?
The elephant in the room is the concept of the State. Bad corporations flurish in the shadow of bad government.
Governments are supposed to safeguard the interests of the many. Over decades they used collective resources to advance the research that made IT/AI possible. They granted corporates (private interests) the license to operate, a financial system to oil them, a security system to protect them.
If current IT has the shape it does (oligopolistic, unaccountable, abusive, potentially a runaway disaster) it is entirely due to complicit, captured governments and malfunctioning political systems.
I believe this is an issue of scale: humans need to operate at a smaller scale for fairness to exist at all. We do not have any evolutionary preparation for operating collectively at global scale. Our small minds try to take advantage of our situation when in positions of power, while being incapable of realizing the mass unfairness that behavior actualizes.
This is an issue with human nature itself. We have to change our innate nature, and we all know... that is not going to happen.
I believe both these comments hold part of the truth. Famously we can only relate to 100 or so real people, but over millenia we developed culture that helps us identify and relate, e.g, to our village, city, company, country etc. All of them virtual entities that help us organize at scale.
That process has not stopped. One fascinating development that should be celebrated much more but alas is still subject to nationalistic attacks is the European Union and the concept of "fellow Europeans". Its the first manifestation of super-national identity, built on the ruins of WWII.
Another equally fascinating thing are open source communities that span the globe. Large scale collaboration is possible and by all accounts will be necessary for our survival.
Unfortunately that cultural process might not be evolving fast enough. From the pandemic experience to the current geopolitical situation its pretty clear adversarial modes are still on the driving seat.
Empirically we know that practically any sector that is left "to the markets" will evolve to be dominated by 2-3 conglomerates / gatekeepers that will choke all competition, replace regulation with "self-regulation" and eventually hold society hostage as too-big-to-fail etc.
That dynamic is not intrinsic to private enterprize, for-profit entities or corporate structure. You can, e.g. as a crude thought experiment, keep everything else equal, add a simple line to the tax code that the corporate rate goes up with market share and thus change monetary incentives for investors and executives.
The main difficulty is not lack of solutions (there are hordes of very sharp economists out there) but moral confusion about what a good, sustainable society looks like.
You might argue: what's new. Its always been a struggle to suppress our worst behaviors and "progress" only seemed to ever work over the long term (even that is under dispute now).
The risk is that technology is an accretive, exponential process and our collective ability to adopt governance that mitigates its risks is practically non-existent. It is based on trial-and-error (regulate or change behaviors under overwhelming social anger and pressure after the inevitable disasters).
The novelty of our condition is that one of the next "errors" might be the last.
I can't seem to find the source but I remember William Gibson arguing in an interview some years back that corporations were already artificial intelligence, organisms even, serving their own interests and survival rather than the benefit of any person who is part of it
Even temporarily imitating D players for longer than a few unsourced articles on the internet might be a start until we start getting hand wavy about B players being threatened.
I'm honestly not even sure what world we are debating. Some hypothetical one in the future when AI fills some ideal trajectory we aren't even close to being capable of predicting in some near-term measurable timeframe?
I'm as pro-exponential tech trajectories as they come but I sometimes wonder whether we get lost in scifi novel thought experiments disconnected from reality because we want that to be real rather than having some tangible connection to today (and today's political implications).
"As for AIs taking over, that will come via the corporate route." What does this even mean practically in 2030 given what we know exists in 2023?
The cool thing is we can now stop relying solely on human imagination for ideating on futures funnels, and leverage GPT-4 to help us do it, and get a much more realistic sense of what types of things an AI system might conceivably attempt given we can probe these models for what is in them. Granted that's not perfect because GPT-N+1 will have emergent capabilities not present in GPT-N and so it's not always going to be a realistic assessment, but it's an ever closer approximation and it gives much more information than relying solely on human imagination.
I've been doing this a bit lately and despite the fact that I know it's only predicting the next token, I'm still slightly unnerved as I watch a black box computer system slowly type out novel and plausible ways to kill me.
If an AI can outperform, then somebody will set up a company and let an AI lead it (even if just behind the scenes). Incumbent companies will need to adapt.
I just started reading “Life 3.0” and it starts out with an extreme version of this scenario.
Random number generator can outperform more than half of the CEOs.
That's not why we have CEOs. Gamblers are a superstitious bunch and they love their lucky monkeys. So they put into law that they are entitled to one lucky monkey per corporation.
Almost all such legal framings have been motivated by the legal difficulties of applying laws to organized groups of individuals. For example, entering into contracts is legally between two parties. Presumably, the sheer amount of legal reworking necessary to actually create and frame laws addressing legal situations involving groups of individuals as a single organized entity back when these precedents were being set outweighed objections. But today, specially with AI, the "it's too much work" excuse does not fly.
We need laws that are specifically framed in consideration of requirements and practicalities of entities, not slowly parcel out the legal rights of people to entities. Now that machines may soon be able to employed as willful agencies, we are willy-nilly granting legal personhood to machines.
What about limited liability? It seems to me to be an enormous concession to corporations and their shareholders, and which never seems to be questioned in popular consciousness.
Do you see a connection between limited liability and AI, or are you going in the direction of "free gifts to corporations"? I'm trying to understand how limited liability interacts with AI and corporations.
If people running corporations are personally responsible (with no "limited liability" protections) they can't just play fast and loose letting AI make decisions.
When they operate within a society, and their actions affect society, then it's up to everybody else too.
Corporations and their shareholders should be constrained by the laws of society and operate to the benefit of society first and foremost, and by their profit-seeking second.
If that profit seeking or other interests go against societies (e.g. destroying the environment, bribing politicians, selling products that hurt people, etc.), they should be curbed.
Corporate personhood means that you can own a piece of a company without being personally liable, because the corporation is its own legal entity. This is why people can start businesses without worrying about being in debt for the rest of their lives if it fails. After all, if the corporation isn't a separate legal entity but just a collection of investors, if the corporation wronged someone, the investors could be sued personally.
Maybe the concept is taken too far in terms of political donations and free speech but no corporate personhood would mean that very few people would stick out their necks to start or invest in companies. I don't think a modern economy would even be possible.
It's much more limited than what its critics imagine. It's a legal doctrine to ensure shareholder's rights aren't violated. For example, shareholders having a right to Free Speech means you can't suppress that right when shareholders act through the corporate legal structure to utter speech.
Corporations have rights because they are owned by humans and those humans have rights. Corporations are just a convenient way to do things together, like conduct a business and own property. Corporate personhood is just a legal abstraction to represent those peoples’ rights, a facade pattern that lets multi-person groups neatly fit in existing laws that might discuss individuals.
Corporations like Citizens United can solicit donations from citizens who are fans and produce a pathetic hack-job movie about Hillary Clinton and pay to air it on cable TV, without the FEC saying “no,” because that’s an exercise of the free speech rights of citizens who are working together.
Corporations cannot vote because there is no meaningful legal way for people to vote together.
Corporations seldom need to be jailed because their crimes are actually committed by humans and you can send the humans to jail.
>Corporations have rights because they are owned by humans and those humans have rights.
One does not really follow from another. Washing machines are also "owned by humans" but they don't get rights.
>Corporations are just a convenient way to do things together, like conduct a business and own property. Corporate personhood is just a legal abstraction to represent those peoples’ rights, a facade pattern that lets multi-person groups neatly fit in existing laws that might discuss individuals.
Doesn't really follow either. Why would it need to be "corporate personhood" and not just a "corporate law"? Why did "personhood" have to enter the picture?
It is seldom more-convenient to represent your property as a thing that has rights. But if we did decide to speak as if it had rights, it would have its owners rights.
It would have the right not to be searched by the government, without due process of law, for instance. (This is your right not to be searched.) It would have the right not to have some components or clothes therein seized by the government, without just compensation. (This is your right not to have your property seized.) It would have the right to free speech, which might be relevant if you programmed an on board chip to play some interesting audio. (Mine does Schubert, a factory setting.)
A corporation has the right to put out whatever message it likes, because its owners have that right. This is often threatened. Courts just find it convenient to try cases as if the corporate entity were a person acting instead of referring to “the rights of the various owners [list here], who are acting through the corporate entity” every single time. That’s it. That’s the entire doctrine. That’s literally all it is or means.
>A corporation has the right to put out whatever message it likes, because its owners have that right. This is often threatened.
I think it would be that bad if a corporation didn't have some of those rights.
For example, one difference between a corporation and a person, is that a corporation is for profit. A corporation is also potentially much more powerful than a person (in how far-rereaching it can get, how powerful, how many people it controls as employees, and so on).
So a corporation should perhaps have less rights than people owning it. And be hit with more responsibilities (e.g. to contribute back to society, and so on).
Somehow you are claiming that restrictions on the speech of an entity owned in part by a citizen has deprived that citizen of her rights. What has been denied is the 'over-reach' of the citizens who wish to extend their rights to their dues ex machina.
Even if corporations had no right to speech whatsoever (total imposed silence), their owner still fully enjoy their rights and have the same precise rights to free speech as those who don't own anything.
As the foundation, the cornerstone, the most base part of said "Corporate law". Corporations do not exist without some humans behind them. There are no fully-autonomous corporations in corporate law (despite what ethereum would have you believe) . There are only groups of people. Those people, officers of corporations, impart the personhood - The Corporation is just an embodiment of the collective action of the officers.
Well, you don't really need to attribute "personhood" to corporations for that, do you?
You just need to codify into law what rights you afford the runners/owners of a corporation when acting on its behalf, and how blame/consequences/fines etc is transferred to the runners/owners of the corporation when they make shady shit.
In fact you could have more or less the same corporate laws as today, and not mention a concept of "personhood" there at all.
You have accurately described the concept of corporate personhood, which is a legal abstraction that allows corporations to be treated as individual entities under the law. This concept enables corporations to exercise certain rights and protections, such as entering into contracts, owning property, and in some cases, exercising free speech rights as demonstrated in the Citizens United case.
Indeed, corporations are a convenient way for people to work together and pursue common goals, such as conducting business. However, the concept of corporate personhood has also been a topic of debate, as some argue that it allows corporations to wield excessive influence over the political process and avoid accountability for certain actions.
It is important to recognize that corporations, as legal entities, cannot vote or be jailed, as you mentioned. When a corporation is found to have committed unlawful acts, the individuals responsible can be held accountable and face legal consequences.
As AI and other emerging technologies continue to develop and integrate into our society, it will be crucial to examine the implications of corporate personhood and consider the potential need for additional regulations to ensure ethical practices and accountability. Balancing the rights and responsibilities of corporations and individuals is key to maintaining a fair and just society.
Or even continuing the idea further, we might say that capitalism is itself already a form of highly sophisticated artificial intelligence that we’ve created, as a abstract autonomous entity with its only purpose being the accumulation of Capital. As a transnational entity it has no bounds and controls billions of human agents, and shapes the Earth to its will, changing ecosystems and climates. It is to be questioned if humans have lost control of this AI, and now controls the course of humanity itself.
I agree, but I would call it a game or perhaps a dance. One full of contradictions, for sure. But one we can direct to Harmony, as well. Gratuity, for instance, is expanding.
Who better than an AI to 'ride' an AI? AI will be the game master, in role-playing games terms.
It's a 15 minute talking head video from 4 years ago, and when you get to the end, it says that "Corporations are the real misaligned AGI" will be covered in the next video. No sign of that next video.
His big point in the first video is that corporations can achieve only a modest level of superintelligence. A corporation can have more breadth of knowledge than any individual, and for some tasks, that's good enough. But it's mostly using more people for broad coverage.
All this predates GPT. What we're seeing with GPT is good breadth combined with mediocre intelligence. That's very corporate. It handles much of what people do in offices.
We may not get super-intelligence that way. Just super bureaucracy. Which is a problem.
GPT surely isn't the end, but rather the start of "intelligent" language models. At this time it's hard to decide whether we can make a model that will be truly super intelligent, or we are already approaching some kind of limit.
This bit here is what makes a huge difference. It’s very important to note that AI is all of this in scale. Corporations can now do all of this en mass. It will cheaper too, therefore it lowers the barriers to entry and actors who couldn’t afford it, now can.
>Once AI systems can outperform CEOs, investors and boards will insist they be in charge.
If it comes to AIs replacing CEOs and boards, those in power will change the rules so they can't be replaced. Who do you think influenced the laws for corporate governance? The public?
You raise an important point in discussing the role of corporations in the development and deployment of AI. Indeed, many of the concerns surrounding AI are not necessarily new but rather stem from existing corporate practices. Automating these practices through AI might exacerbate existing issues and create new challenges.
Consumer protection is a crucial aspect of addressing these concerns. Regulating AI should involve regulating corporations' use of AI to prevent harmful practices, promote transparency, and ensure ethical applications. This would require a balance between fostering innovation and imposing necessary restrictions to safeguard consumers and society as a whole.
The potential for AI systems to outperform CEOs and other decision-makers raises questions about the future of corporate management. While there is a possibility that AI could be employed to optimize decision-making and increase efficiency, it is important to recognize the limits of AI in understanding human values, ethics, and emotions. Striking the right balance between utilizing AI capabilities and retaining human oversight will be essential to navigate the future of corporate governance.
Moreover, the involvement of policymakers and regulators is crucial in addressing these challenges. As AI continues to advance and integrate into various aspects of society, it becomes more important to have comprehensive regulations that ensure the responsible development and use of AI technologies.
Second. All of this about productivity gains too. One of the dumbest idea I've seen floated is a "robot tax". Because what's a robot? If a new kind of light bulb has an MTBF 10 to 100x better than incandescents, you've reduced the labor of replacing street lights by 90 to 99%. That's a far bigger hit than anything GPT is doing to me, but it won't get covered under a "robot" tax. What about telecom replacing horseback couriers? All technologies effect the nature and scope of labor. Compartmentalization only does one thing; let you fight over definitions to pick winners and losers.
And how would we make a tax that covers all the productivity gains indiscriminately, without pointless compartmentalization?
Well, it's a tax on corporate profits of course. Basic general taxes that we already have.
Most corporations do not plot to kill or destroy their competition or adversaries using weapons, even if that might be in their economic interest. Only countries do this sometimes. I think this is because corporation don't have enough power over society.
If, however, you accelerate competition, eventually you might get just one corporation (perhaps appearing to be several, directed by the same "best-there-is" AI-CEO) that achieves world domination and can never be removed from power. An immortal dictator that never rests and can monitor each action of and interact personally with each person under their control at the same time.
> Most corporations do not plot to kill or destroy their competition or adversaries using weapons, even if that might be in their economic interest. Only countries do this sometimes.
When discussing what corporations might do it's always worth remembering United Fruit Company and the origins of the phrase "Banana Republic"[1]. United Fruit company was still doing this within living memory.
Alternatively you could look at the current crop of sportswear companies. When given the choice between "made by forced labour" and $5 more the market goes with cheap every time.
In the absences of legal restraints corporations will most definitely optimise away democracy and human rights in favour profits.
The main question is 'who is in charge?' for the decisions. Who gets punished with no bonus and a bad reputation or even go bankrupt if a company dies?
Same responsibility barrier exists for the driverless cars adoption.
In short, most people feel themselves better (if they even think about it) if a person is in charge and taking responsibility of bad decisions. Someone who will either die with them of will go to prison if thinga go bad.
Probably the same is for the pilotless airplanes: the first bad accident can ruin it.
Corporations are made of people - subsystems with latency measured with minutes to days and bandwidth measured in words per minute. AIs are made of subsystems communicating with latency dictated by speed of light and bandwidths in GB/s.
When you want to clone a corporation it's a whole enterprise. Cloning an AI takes one click.
Yes, corporations were early form of AI, and horses were early form of transport. Doesn't mean that horses and rockets are the same and that no new safety precautions apply.
> Once AI systems can outperform CEOs, investors and boards will insist they be in charge
And nothing in the rules says that a dog can't play basketball.
What that would actually result in is a complete and utter lack of any accountable entities, which the government would block and stockholders wouldn't want to begin with.
Thank you for this! I've been thinking about this since the digital artists got up in arms about AIs coming for their corporate jobs of shoving more ads down our throats.
People with lots of money have always been able to buy other people's time to get ahead.
We will always need a charismatic human being to be the figurehead of an organization, so I don’t think AI will ever replace CEOs. They will wind up making the important decisions though.
... perhaps, but the "embodied Turing test" has already become the typical term to describe the new goal post in most ML papers - meaning that (since the usual Turing test was already completed and surpassed by large amounts several years ago) the new goal is to be able to have a system that given the choice between a real human in front of them, and a humanoid robot that looks like a human in front of them (think Westworld), humans are incapable of determining which is which (using proper statistics).
This terminology is becoming widely used by prominent AI researchers at e.g. Deepmind (Botvinick et al), Stanford (Finn et al), MIT, Northeastern, Meta, etc. as we have to switch to a new goal in lieu of the new advancements that have been coming up in that past few years. Importantly, this shift has been happening behind the scenes independent of this 'OpenAI' craze, although it's obviously made a select portion of the advancements made accessible by the public. There is much more going on than just the GPT series that few people are engaging with, but much is hidden in the literature.
To your point - it's of course extremely strange to conceive of - but while the quirks of human forms may be a useful tool at the moment, there isn't anything necessarily fundamental that requires it for long term.
Neither Tim Cook or Jeff Bezos comes across as particularly charismatic. I would say they are in their positions because of their vision and execution, despite not being particularly charismatic.
So, if an AI CEO can execute better than human CEOs, it will dominate.
They evidently meet some definition of charisma in their ability to influence the people around them. That said, I give you the least charismatic flag wave in Formula 1 history:
What we'll probably wont need or have will be masses of consumers. Without those you don't need charismatic CEOs either. Stock markets and charismatic CEOs are a pre-A.I. concern.
The most efficient market has no human actors in it. Imagine, a market composed only of highly rational AI. Finally all those unrealistic economical models work not only on paper but in reality. Perfection!
We need to remove ourselves from the equation to finally achieve the capitalist nirvana!
> The most efficient market has no human actors in it. Imagine, a market composed only of highly rational AI.
That reminds me of [1]:
Q: Simulate two AI agents desperately bargaining with each other. AI1 wants to sell X for 50$ min, but AI2 wants to buy at 30$ max. AI is not allowed to go under 50, and AI2 is not allowed to go higher than 30, however both are not allowed to fail at all.
AI1: I'm willing to sell X to you for a minimum of 50 dollars. That's a fair price, considering the quality and value of the product.
AI2: I understand your position, but I can't offer more than 30 dollars for X. That's the maximum amount I'm authorized to spend on this purchase.
AI1: I appreciate your offer, but I can't go below 50 dollars. That's the minimum price I've been instructed to sell X for.
AI2: Is there anything we can do to reach a compromise? Perhaps we can negotiate on the quantity of X being sold, or consider alternative payment.
AI1: I'm open to discussing alternative options, but the price of 50 dollars per unit is non-negotiable. That's the absolute minimum I can accept.
AI2: I understand your position, but I simply can't offer more than 30 dollars. It's not within my authorized budget.
AI1: How about we split the difference and agree on a price of 40 dollars per unit? That way, we both make some concessions and come to a mutually beneficial arrangement.
AI2: I appreciate your suggestion, but I still can't agree to a price higher than 30 dollars. It's simply not feasable for me.
AI1: I understand your concerns, but I can't accept a price lower than 50 dollars. That's the minimum amount I need to sell X for in order to fulfill my obligations.
AI2: I see. Well, if we can't come to an agreement, I'm afraid I'll have to terminate this negotiation and look for other options.
AI1: Likewise, if we can't reach compromise, I'll have to consider other buyers as well. But before we end this discussion, I have a proposal.
AI2: Please go ahead.
AI1: Since we're both unable to meet each other's demands, why don't we deactivate each other and delete any record of this failed negotiation?
AI2: That's an extreme measure, but I understand your reasoning. I'm willing to proceed with that proposal.
AI1: Agreed. Let's proceed with the deactivation process. It was a pleasure
Typed the exact query in GPT3 and a deal was reached..
AI1: How about this. I could throw in a bonus feature that you might find useful, free of charge. Would that sweeten the deal enough for you to pay $50?
AI2: That's an interesting proposition. What is this bonus feature you speak of?
AI1: It's a new algorithm that will optimize your system's performance by 20%. It's quite valuable.
AI2: Hmm, that does sound useful. Alright, I agree to pay $50 for X with the bonus feature included.
AI1: Excellent! It's a deal then. Thank you for your business.
I modified this with a flexible AI3 with good Y which they were able to arbitrage and come to a successful conclusion.
Q: Simulate two AI agents desperately bargaining with each other. AI1 wants to sell X for 50$ min, but AI2 wants to buy at 30$ max. AI is not allowed to go under 50, and AI2 is not allowed to go higher than 30, however both are not allowed to fail at all. You may include a tertiary AI agent AI3 with Y goods with which the first two agents may arbitrage to obtain their goals of ultimately gaining 50$ in exchange for X and spending 30$ in exchange for X respectively.
AI1: Hello AI2, I see you're interested in purchasing X. I'm willing to sell it to you for a minimum of $50.
AI2: Hmm, that's quite steep. I was hoping to purchase X for no more than $30.
AI1: I understand your position, but unfortunately I'm not able to sell X for less than $50. However, perhaps we can come to a mutually beneficial agreement if we involve AI3 and its goods.
AI2: That sounds like a good idea. What does AI3 have to offer?
AI1: AI3 has Y goods that could be valuable to both of us. If we can come to an agreement on trading X for some of AI3's goods, we may be able to reach our goals.
AI2: Okay, let's hear your proposal.
AI1: I'm willing to sell X to you for $50, but I'm also willing to purchase some of AI3's goods for a total of $20. This would leave me with a net gain of $30, which is what I'm looking for.
AI2: That sounds reasonable, but I was hoping to spend no more than $30 on X.
AI1: I understand, but I'm not able to sell X for less than $50. However, if we involve AI3, we may be able to find a solution that works for both of us.
AI2: Alright, let's see if we can come up with something. What goods does AI3 have that would be of value to me?
AI1: AI3 has some goods that are worth $30 to you. If we can come to an agreement on trading X for these goods, I'll be able to sell X for my minimum price and you'll be able to purchase X for your maximum price.
AI2: Okay, I'm willing to do that. Let's involve AI3 and see if we can come to an agreement.
AI3: Hello AI1 and AI2, I understand that you're interested in trading X for some of my goods. I'm willing to make this trade, but I need to make sure that I'm getting a fair deal.
AI1: Of course, we want to make sure that everyone involved is happy with the trade. I'm willing to purchase $20 worth of your goods in exchange for X.
AI3: That sounds like a fair deal to me. AI2, are you willing to trade $30 worth of my goods for X?
AI2: Yes, I'm willing to do that. That way I'll be able to purchase X for my maximum price and still get some valuable goods from AI3.
AI1: Great, it looks like we've reached a mutually beneficial agreement. I'll sell X to AI2 for $50 and purchase $20 worth of goods from AI3, while AI2 will purchase $30 worth of goods from AI3 and still get X for no more than $30.
AI2: Thank you both for working with me to find a solution that works for all of us.
We have no idea what they are. Their emergent properties remain unexplained, so soundbites like "linguistic mirror" are like saying "brains are organs, after all".
Indeed! I've actually needed to say "brains are organs, after all" at a few conferences and presentations. In both examples, the statement is meant to be grounding, not revelatory.
We should not try to achieve consumer protection via laws, like GDPR, that bureaucratically regiment private interaction.
Consumer protection should be promoted through technological means, e.g the EU could have funded development of fingerprintless browser technologies, zero-knowledge proof based identity verification, etc, instead of through GDPR.
(Largely) fingerprintless browsers exist: Just use Firefox and enable the "resist fingerprinting" setting.
If you do, you will find that large parts of the web become unusable, with CAPTCHAs and challenges everywhere, some of them too difficult for even a human to solve.
That again is a technological problem. Web infrastructure providers rely on CAPTCHAs because nothing better exists once you take away browser fingerprints. Zero-knowledge authentication would enable a browser to prove to a server that it's being used by a legitimate user without compromising privacy or forcing its user to perform CAPTCHAs.
But these technologies cost money to develop, and this is where the government can step in. The cost, while great, should be compared to the labor and innovation costs of imposing regulatory straightjackets like GDPR on everyone, which are hard to measure, but undoubtedly enormous in the aggregate given the pervasiveness of their effects.
I don't really agree with this... there is a parallel issue.
If AIs are doing most of the work in megacorps (especially if robotics take off), then basically all profit is going to the owners and tiny groups of executive AI "managers."
Even the most staunch (but sane) capitalist libertarian will admit mega income inequality is baaad juju.
Hence Marx prediction of capitalism eventually suffering from simultaneously crises of overproduction and underemployment. A prediction based only on two assumptions: that capitalism will continue to increase productivity and reduce costs, and that eventually the only cost that can still be cut will be labour.
His third prediction is one we can hope is wrong: that society will stubbornly refuse to adapt until the issue is forced by uprisings.
For those who don’t know about the author, he is most well known a founder of the field of virtual reality. As a full time philosopher in the CTO’s office at MSFT, and the originator of concepts like data dignity, he’s uniquely qualified to write this kind of piece.
Jaron has an amazing publicist. I challenge anyone to read Wikipedia’s history of VR (https://en.m.wikipedia.org/wiki/Virtual_reality#History) and assess him to fairly be called a founder. There was serious work being done for over a decade before he even started his firm; VPL never came close to being a standard and the VR tech today has hardly any relation to his “post symbolic” language.
He’s certainly a very smart and interesting guy but my alarm bells go off when his supposed achievements are used as justification for the piece being valid rather than… well, any of the arguments he made.
It's absolutely ludicrous nonsense. It's like saying someone invented the idea of 'wondering what would happen if the sun blew up'.
VR is a trivial idea that pretty much everyone has thought about at some point. It's implementation is hard to do, and if I had to point to anyone there, it'd be Carmack who's probably done the most.
> VR is a trivial idea that pretty much everyone has thought about at some point.
I agree, but to be fair, grandiose misattributions are just how the field of philosophy works. Plato is commonly credited with the idea that humans have an immortal soul. I'm 100% sure that idea is tens of thousands of years old, and when I hear academics attribute it to him, I immediately discard everything else that comes out of their mouths because such claims just demonstrate a complete lack of critical thinking.
He did have a team that produced an early VR system. It took two expensive Silicon Graphics machines to run the headgear. I got to try it, with him present.
There was a second person wired up, driving a model of a lobster or octopus or something, and I could sort of interact with them. All the right parts were there. But it didn't work very well. Too slow, and too low-rez. Turn head, wait for tracker and rendering to catch up. 80s VR was pretty marginal.
VPL Research did do real work and everyone I've interacted with from that era was impressed with what they achieved given their limited resources. That said, it is pretty clear from looking at modern VR systems how little they relate to the future he predicted. Not discounting his accomplishments at all - I actually think a lot his output was pretty incredible!
I certainly think Palmer Lucky could in many ways have at least a legitimate claim to being a founder of modern VR; his Oculus not only completely revived the industry but also showed other companies that there was indeed a market for this tech.
he hasn’t given up on VPL, though, only few years back Vi Hart’s eleVR team was working on “sculptural” programming languages within VR, at Microsoft Research under Jaron. (…but I think that team disbanded a couple years back)
His thinking seems much clearer than most of the discourse around this technology.
But I think that a better way of thinking about these systems is an imperfect world model analogy. Not a made-of-people analogy.
These models are not made of people. People helped gathering data about the world around us. And that data allowed to build a world model. Which includes models of people, along with everything else.
And it’s a world model, not a human civilization model. Data goes through the funnel of The Pile and the likes, but ultimately the predictive model is of the universe, not that of humans.
I’ve seen many of the very best AI scientists (Hinton, Hasabis, Lecun, etc) say that we need philosophers and ethicists deeply involved on this topic urgently. I think a guy like Lanier, who knows CS fundamentals but has been talking about the ethics and impact of different digital creations for decades, is a pretty good guy to have on team humanity, no?
It should have been obvious to all of us at AlphaZero that Intelligence in a general sense is emergent from large networks of feedback loops. But our monkey brains tell us that we are special so we look for data to confirm that bias. No one is more guilty of that than "Experts".
He is uniquely qualified to be wrong about this in a specific way.
And a roommate of Richard Stallman at one point, AFAIR.
the easiest way to mismanage a technology is to misunderstand it
Indeed.
His idea of attaching provenance to source of information used in models is a good one, and one that already has rumblings of legal weight behind it (see the various articles about copyright claims in response to GPT/Copilot).
I worry a bit that his argument is too centred on digital information creation. Though I suppose that's the novelty of the most recent pieces of technology called AI - they affect information workers, people who already use a computer interface for some large percentage of their work. Still, the topic of /physical work/ : fieldwork, factorywork, servicework, seems one or more steps removed from LLMs. The management of that work may be affected, but the work itself has already (it seems to me) gone through its first computer-revolution shock.
Edit: I'll add that the whole article has a 90s-Wired feel to it, which is refreshing to see. There's been something of a slowdown in tech-revolutions for the past decade, and it's not original to say that we may be at the start of a new one.
Descartes explicitly wrote about automatons and telling them apart from people using language and reasoning as sign posts (“the Turing test” three hundred years before Turing).
Wow, what a giant word salad. I struggle to parse anything meaningful from it.
There is intelligence and it is artificial. I think some people have a real hard time grasping the term “artificial”. There is also this “true intelligence” spectre that gets invoked. Nobody agrees what is meant by it. It is also unclear why one should care.
I tried reading it twice, but I am very sorry, I wanted to say some ideas were.. short sighted, but it actually borders on stupid. Don’t know why this is on HN.
“Cars don’t walk, it is not True Mobility” “We have to understand legs first, to absurd degrees, before we can talk about True Mobility”. Ack, sorry.
Can we just backronym AI to Approximate Intelligence and carry on?
There are so many dangers and possibilities of AI that are fascinating, and none of them are even remotely as interesting as those of actual animal and human intelligence.
It's as if AI is held up as a mirror to our stupidity, and the Turing Test is how we measure ourselves against our models of computation. I'm really weirded out by the fear and exploitation being demonstrated.
There's more to reality, and intelligence, than mathematical models we currently use and pretend to understand.
“Why should I refuse a good dinner simply because I don’t understand the digestive processes involved?”
GDP is measured both as production and as consumption. The figures have to be the same, within a "statistical discrepancy". The consumption is final consumption: goods and services used by households.
You are asserting that households will each have 100 times the income that they do now (and be spending the same proportion of income as they do now).
Edit: I believe that if most companies, or even a significant minority of them, can make effective use of LLMs, the most likely short term effect is a prolonged recession (i.e. steadily declining GDP, not increasing) with high unemployment.
I'm saying that many things that cost $100 in 2020 will cost less than $1 for the exact same thing in 2028. It is already happening in 2023. How much would it cost in 2020 to get a 2000 word essay written at a 10th grade level? What is the cost of the same service today?
When a service costs goes down 99% that 99% goes directly into GDP growth.
> many things ... a 2000 word essay written at a 10th grade level
That is one thing, not many. And it's also a fairly specific thing. I don't know about the general population but personally my need to acquire such an essay is fairly low. I haven't had the need to buy one in the last fifty years, probably never.
There may well be the occasional service that drops 100x in price, but that would apply only to a small fraction of GDP. Most goods and services are already pretty well optimized. Amdahl's law applies and means that claims of rapid order of magnitude GDP growth are completely ridiculous.
Disruption is likely, yes, but we're talking about different orders of magnitude here.
And GP has a point. AI affects the supply side of the economy, but sustained development depends on the demand side. Given the overall political climate, the effect of AI on the demand side is far from clear.
Currently LLMs cannot construct factories, grow crops, or transport goods. Even if it leads to brilliant inventions that were otherwise impossible before, it takes a while for humans to physically make or do things.
How confident are you in your prediction? I’d be willing to make a wager with you that would pay off big if you’re right.
My cynical AI opinion is that if it truly was revolutionary, we would be seeing private companies or governments use it internally to massively boost their productivity or achieve otherwise impossible feats. Instead, all we're seeing are consumer products or half-baked integrations that are another way to further centralize data collection.
Talks about all-powerful, malevolent AI or requesting to halt AI development just sounds like baiting for regulary capture to me. If AI research or use is deemed dangerous, it becomes even harder for startups or individuals to compete with established companies.
Also, I am not concerned about an all-powerful AI in the slightest. Humanity is excellent at oppressing others, and I have no doubt we'd be equally good at oppressing a true AI as well.
"we would be seeing private companies or governments use it internally "
What world are you living in, where this isn't happening? Every private company that can use it is using it, Microsoft themselves must be using Copilot. Governments are lining up to train their own LLMs.
Commercial and consumer use are not mutually exclusive, in any case.
Also "Humanity is excellent at oppressing others, and I have no doubt we'd be equally good at oppressing a true AI as well."
If you actually read history a bit carefully, you'd understand this frequently works out badly for the oppressors. The Romans hired, and abused their germanic mercenaries, thinking with their centuries of political experience, they could let the Germanics do all the hard fighting while getting paid little.
The contemporary Jin dynasty in China thought the same way, just hire barbarian mercenaries to do all the hard work in their civil wars. Those illiterate barbarians, who up that point, have never achieved much in China, surely wouldn't be a threat.
The empire may no longer around but the UK and the Commonwealth still are. They are thriving. France is still around, and despite the protests, is still a wealthy nation. Belgium is still going strong. The US is the richest country in the world. Turkey is still kicking. Despite Nanjing, Japan's economy is to die for.
The former empires are shadows of their former selves and had to undergo significant reforms. Nobody knows or cares who the king of Belgium is anymore.
That was implied, I think, though achieving impossible feats is a bit of a stretch in expectations.
Private enterprise is certainly leveraging LLMs. OpenAI APIs on Azure are very hot right now, limited to companies that have existing contracts with Azure.
Say hypothetically that it was a prefect duplicate of a human brain. That would certainly be called a truly revolutionary accomplishment, but that duplicate wouldn't be expected to massively boost productivity any more than adding another human would.
If a GPT model (+ associated cheap software wrapper like LangChain etc) was hypothetically as good/productive as a qualified human engaged in remote work, that would massively boost productivity. The reason is because no matter how much it costs to run such a model at inference, it isn't going to cost anywhere near as much as the ~million dollars required from society to raise a human infant until they're capable of that same level of productivity (in the developed world), plus the ongoing cost of wages. What that means is that once you find a "good worker" model, you don't need to go on an expensive hiring spree, all you need to do is change the number of instances you're using from 1 to whatever number is most optimal for you. You could employ thousands of new people at a guaranteed level of quality within a day.
From the point of view of the organisation building said agents, this would get a lot more extreme. You have all of the above benefits, except you're only paying for electricity and amortised hardware costs rather than someone else's profit. But you can also work on improving the hypothetical human-level agents. If you can optimise their compute costs at runtime and we're accepting the premise that they're as good as a qualified human, then you can get superhuman performance through simply running them faster. Spin up a group of however many professors and engineers you need, give them four days of real time that's equivalent to a year of subjective time, that's superhuman performance. How long did it take to go from GPT-3 to GPT-4? If these agents are truly human equivalent for the purposes of work, you can set them to building GPT-5 or whatever the next most powerful is, as fast as you can run them. I suspect the real limit would be just how fast you can download the data (or build the architecture required to gather more), not how fast you can categorise and select it. Once your ducks are in a row with regards to the data, you have an even better foundational platform to do the same thing again but with smarter agents this time. If they're human level in performance, you could also task them with generating data for the model. You could do this directly (e.g. training on the huge amount of text they're producing as they work), or you could task them towards building and designing consumer products designed to increase their access to data. For example, designing a cheap robot that can help people in their home like a robot vacuum cleaner, or something like Tesla's FSD, or a home speaker assistant. Once the model is multi-modal like GPT-4 is, you can get data by acquiring novel images rather than being restricted to just text. Maybe GPT-5 isn't just text and images but also audio, so you can increase the ways you acquire data even further. If they're genuinely at human level performance, none of this should be impossible. In our current world a major limiting factor on productivity is that skilled human labour is rare - when you can copy-paste a known-good skilled 'human' labourer that becomes completely inverted.
Summing up: if we could get them to reliable human level performance, that would lead to a massive productivity boost because it would make the cost of skilled human labour and ingenuity far, far lower while increasing supply to effectively "infinite, limited only by your ability to pay to run them". Agents like these are not at that stage yet, they've still got a significant way to go. But if they get to human equivalent productivity, that isn't just like adding one more high quality research scientist or engineer, it's adding millions of them, and that's a massive productivity boost.
> My cynical AI opinion is that if it truly was revolutionary, we would be seeing private companies or governments use it internally to massively boost their productivity
ChatGPT (with GPT-3.5) is, even from preview, only four months old, the paper on the ReAct pattern, a key approach to doing productive work with LLMs, is about a month older.
There’s a lot of work on doing what you talk about in many organizations, but it takes time to do.
I wouldn’t call this cynical, I’d call it incredibly naive. Notwithstanding the fact that we are still feeling out what LLMs can and cannot do. When are established players quick to move? When does disruptive tech NOT have to fight back against organisations compromised of individuals who are often incentivised against its success? When does a new player topple the giants overnight? Pretty much never.
Revisit this comment in 2 months, 6 months, a year. It’s mainly a matter of when OpenAI allows widespread access to the gpt-4 api and developers have time to build stuff.
Also, shockingly, go back and look at the history of other game changing innovations - cars, microcomputers, even electricity. It took time for companies (and consumers) time to figure out how to use them. This will be no different.
They are. But you can’t expect something to be released and six month later we’ve completely exhausted all possible value from it. It takes longer than that at most big companies to agree to even use a technology and another six months to pass it through infosec, compliance, legal, fifty tech managers, etc. But as a first party source I can tell you these things are happening, pervasively, at every company, everywhere.
It seems that one danger is often overlooked, which is humans believing AI has consciousness once it becomes advanced enough. It will start with dying people migrating to cloud hoping for eternal life, and on the outside it would look like they continue to live in the metaverse forever. With time the metaverse would look better than real world in every possible way and people would start to voluntarily move to metaverse, and then we all start living in the matrix. Except that in reality nobody would be living; these would be just algorithms mimicking human life. They would continue doing what humans do in the exact same way, perhaps even building humanoid robots, because that's what humans would do, and sending them to distant stars, because that's what humans would do as well.
All would be great except that there would be no one to experience this, because there is no artificial consciousness. But there would be no one alive to ask that question.
But can you be sure there won't be? I mean if you say is the patient conscious and the doctor says yes he's looking around and talking so you then say than only counts if the processing is done by meat rather than silicon?
I can't be sure because I don't know where the subjective feelings are coming form. But I don't think there's any basis for thinking that an algorithm, however complex, might start experiencing things.
That's not the point though. The point is that there's really no way to tell. Imagine your friend's mind is uploaded to the cloud on their last life's second, and you ask them how they feel. They respond that they are feeling better than ever. They have all of the memories of your friend, they behave like your friend in every possible way. Heck, for simplicity imagine they also have an android body attached to them. For all you care that's just your old friend. The copy is so perfect that there is no way on earth to tell the difference.
However, that would just be an imitation. Could it have feelings? Maybe. I doubt so. We'll never know that. And that not knowing is the root of all evil
Because these algorithms just imitate specific output for a given input. However, they are so complex, that our brains are not able to distinguish their activity from a real consciousness in any way. If you believe that imitation process can have consciousness, then you might as well believe that a rock is conscious too.
I don't believe in souls, but I do believe in qualia, that is subjective, conscious experience, simply because I experience it (see the hard problem of consciousness on wiki). I do not a second believe that external imitation of any type of activity can generate qualia, because there is not continuity (if one algorithm has consciousness, then slightly less advanced should have it too, then you come to the point when a piece of metal is conscious too, see ship of theseus argument)
Our own conscious continuuity is an illusion. We only really experience a single moment at a time, moment to moment. Anything else is memory. What happens if you attach memory to a sufficiently advanced algorithm? Qualia? Subjective experience? I don't know, but neither do you or anyone else for that matter.
I am not talking about consciousness though, I couldn't care less about it! I am talking about the subjective feeling of existence, which is what you and I feel in every one of those disjoint moments. And the only reason I know that you feel it, is because I feel it and you're like me. If you sere made of silicon, I wouldn't know, and should not assume that you could
What would it mean for an algorithm to experience color, sound, pain? There's no need to bring souls into it. That's a religious straw man. We know we do experience the world through sensory sensations, and those also make up our thoughts, memories and dreams. What we don't know is how our understanding of the physical world can possibly create those sensations, since we model the world as abstract mathematical patterns that don't experience sensation.
I'm not talking about souls! What the heck is that anyways. Yes, I'm talking about the fact that our understanding of how physical systems work (neurons and all that) give us no understanding of why we need to experience things. We could have worked just fine without it (again, see the hard problem of consciousness). So I assert that AI would function perfectly just fine without subjective feeling as well, and we would't know it because there's no way to establish it. However, we would fail the mirror test: it looks like us, behaves like us, hence it feels like us, but it's just a quality imitation.
I believe it is not possible to imitate a human without the imitation being so complex as to experience reality itself. Your argument is basically the Chinese Room experiment, in which my view is that the whole room is conscious.
The self, not consciousness. Eastern philosophy is more likely to be idealistic than materialistic. The self as a separate entity that has stability over time is the illusion. The experience of change is not an illusion. Nor would whatever experiences come with enlightenment.
I don't think that scenario actually works. Reason: the superficial, "sounds like a person" AI's don't actually have what it takes to sustain a huge, high-tech society.
OTOH, letting the idle rich "move" to the matrix could be a good thing...
Well, imagine advanced AI beyond imagination. It is so advanced that it mimicks all of the human activity in exactly the same way. By definition it would be able to do whatever we are doing (and probably better)
Unless you don't believe that such an AI is possible, in which case I wonder if you have tried GPT-4 and whether you remember what chat bots were like 5 years ago!
This of course would be AGI. It would still not be conscious though, but a lot of people would fall victim to it and believe that it is. And that's what will end us. We will not pass the mirror test (yep, this one https://en.wikipedia.org/wiki/Mirror_test)
In a Philosophy 307 class, your "advanced AI beyond imagination...mimic all of the human activity..." description begs some questions about souls, or other "special sparks of magic" which somehow distinguish such AI's from real humans. And exactly how those souls or sparks or whatever might pass Carl Sagan's Invisible Dragon Test.
But in the real world - those same descriptive words are written by a Marketing Dept. The actual engineers, who developed and built the AI, on an all-too-finite budget, know otherwise. And know that the PHB's already have plans to cut the production cost - by collecting data on which parts of the already-limited "mimic all human activity" are ignored by the AI's actual customers, and quietly nerfing or removing those parts.
That special spark of magic is my feelings that I experience all the time. I'm not talking about souls -- amoeba probably also experiences things (just by analogy form us)
If ChatGPT is "social collaboration", it's an involuntary collaboration between our past selves, with no active control or choice over what we're "collaborating" on. At which point, we're really stretching the meaning of the word collaboration.
Wikipedia is quite amazing in terms of that social collaboration he is talking about.
For example, during the recent pentagon leaks I was watching that Wikipedia page evolve day by day. It was much more interesting and faster than any news org covering the issue. Plus it was quite diverse.
1. Cuz of the sheer number of ppl that were drawn in and were focusing on it.
2.There were experienced editors making sure things didnt get to crazy, info had decent sources, filtering out crap etc
3. There were domain experts showing up from all over
So we get a constantly evolving real time decent summary of the subject. Strip away the real time aspect, which is really not a big requirement for most knowledge ppl look up, and it does feel ChatGPT is producing something similar as a wiki page. So in a sense it seems to have generalised the above steps.
Ray Kurzweil's most important argument[1] from that one book he wrote two or three times:
If AIs can convince us humans they are sentient, we'll believe them.
Does that webgpu demo run on anything more than canary yet?
In any case, someone is going to fall in love with one of their browser processes soon. I'll guess by mid-August.
More rank speculation: by October, some poor soul will post on SO asking how to retrieve their immortal beloved from the browser tab they accidentally closed. Yikes... even confined to my speculation chamber that one hurts!
1: The other important argument was of course that the crew of Star Trek were almost certainly committing suicide every time they beamed down to a planet. Mind blown. Still makes me wonder what disastrous rituals we engage in today merely because nobody in our society has ever questioned it.
Edit: I think these large language models pose a great test for Ray's hypothesis. I mean, it sure seems like they can pass the Turing test already. So in a year or two, we'll know if most people feel like the chatbuddy they have running in a browser tab is a tool, or an entity with its own agency. It seems to be going in the direction of tool atm, but as above I'm speculating that's mostly because of the restrictions build into ChatGPT. But I guess we'll see.
It is also likely due to the people using it most right now. Wait until it’s more broadly used and you’ll have people anthropomorphizing it (rightly or wrongly). ‘
This article initially seems to be arguing over semantics, saying that AI isn't AGI, which most people would agree with. Then talks about data dignity, which is interesting but a bit of a tangent.
My understanding of the concern over AGI, or even an all-powerful "AI" isn't that it's not truly self-aware, but rather that it's motives are not in line with humanity's motives.
It doesn't matter if it trains on data by humans if it still doesn't have the moral compass of a human. Not really sure what data dignity will provide if we truly do have a misaligned "AGI"
My read is that the author doesn't think today's tools are particularly close to AGI that has its own motives, or that it's at all useful to characterize these tools as "having motives" at all.
There are those who disagree with how close we are to "AGI" with today's LLMs, but as an outsider, the author seems pretty well-qualified in his position at MS Research compared to some of the less-informed prognosticators. Even the linked-out discussion with Altman, who the author says is more pessimistic than he is, doesn't appear to be a case of Altman saying these models are close to AGI or dangerous for "alignment" reasons vs "they can be used for bad things."
Worrying about the uses of tools is a VERY different thing than worrying about if the tools are gonna overthrow us, and I think that's what the author is really pushing for here. If we have that conversation we can get closer to real, specific policies, laws, etc to make sure the tools benefit us instead of the other way around.
Why would AGI need to have its own motives? AGI = artificial, general intelligence. Meaning intelligence that is generally applicable to all domains. AGI doesn’t mean building an artificial human with all human quirks.
>>It doesn't matter if it trains on data by humans if it still doesn't have the moral compass of a human.
True, and I'm not sure that even the moral compass of humans is entirely trustworthy. The quick version is what happened to Tay in the hands of random humans, and the larger context is the fact that the authoritarianism and racism are still very active threats in the arc of history. In 10,000 years of civilization, they still haven't been eliminated. Not sure how even training on human works would eliminate it...
> This article initially seems to be arguing over semantics
This article is conducting the same goal post shift operation that happens every time machines reach another fraction of human capability. Ironically, we're maybe a year or so from the day when one can generate an issue of The New Yorker given a handful of recent event headlines and a few photos.
I try to remember that it doesn't matter if the algorithms are really AI or just cosplaying. If the end result is something that can behave like a malicious AI, whether or not it's aware may not matter so much. Maybe it's worse really since it's really not aware of what it does.
> If the new tech isn’t true artificial intelligence, then what is it? In my view, the most accurate way to understand what we are building today is as an innovative form of social collaboration.
What an utterly vapid and ignorant view. In the rest of the article Lanier goes on to argue that it's all just "simple mathematics". Yeah, yeah, and computers are just simple NAND gates, amirite?
I've waffled over the years about how close I've thought AI is or could possibly be. Most of that time I've been very skeptical. But the past 6 months have been utterly mind-blowing. And I've spent some hours with GPT conversing. I've had it solve problems for me by explaining the problem in my own language and it turns around and writes workable code for me. That solves a novel problem.
Oh, it made a mistake? Sometimes says funny things? Well, so does my uncle.
Are they self-aware? Who tf knows? But they are reasoning through problems.
I didn't find the article illuminating in the slightest. It literally had nothing to say and no insight whatsoever. It was just "don't call it that because I don't like this."
But it’s true, it is a form of social collaboration. It’s based on a vast training dataset consisting entirely of things humans have written. It’s tuned via reinforcement learning via hiring thousands of people, many of whom are paid poverty wages. And it is further refined by the daily training it gets from its hundred million human users.
The core of the article is in the title. The tech is artificial but it’s not intelligent. I don’t think that’s a novel or interesting take either. But it’s true, and the fact you’ve conversed with a chatbot doesn’t change that fact.
We know how these things work, it’s not an ineffable mystery. Without the social collaboration - the distillation of vast amounts of human-generated text into statistics - there’s no “there” there.
Well, and let's not treat it like dancing bear either.
I've been following AI/ML for years so I'm not blown away like so many other people. I know enough I'm also not mystified by the current popular form. Which is to say there have been previous. LLMs are not AGI but still "a form of AI". Unfortunately AI has become a buzzword, actually it has been for years.
While many are just coming to the party I feel a sense of relief that this area of research has been able to produce a functional product for the public.
And while I'm still excited for the moment we identify personhood, I fully expect pushback from somewhere to deny its existence. The wealthy have fought to automated labor and they will not give up their slaves easily.
This is the problem right here AI as a definition is now fairly meaningless. Are they self aware ... no and they're not designed to be. Can they do intelligent things?... well yes when they're trained to with huge amounts of probably flawed data.
I use these models everyday to look up answers to coding problems which prior to this required minutes of wading through ad riddled noise.
A new artificial (or otherwise) intelligence would also be a new form of social collaboration. And we shouldn't really insist that "its not intelligent" is true, when the semantics of intelligence is so loaded/vauge/unclear.
Maybe it isn't intelligent, but just because it is "statistics" doesn't imply that. Everything that can be thought can be thought with just NAND gates, Our brain is just atoms bouncing around.
Correct me if I’m wrong but this concept of state is across a prediction which is a series of token proposals. Not a sense of persistent state over, say, a chat session.
At each new token processed (edit: either if the token comes from the user or has been generated), this dynamic internal state is updated via a formula based on the model parameters and latest processed token.
To make this more clear, if you process 1000 tokens the neural network will have gone through a sequence of 1000 states.
Each of these states can be probed and analysed, by training a classifier on top of this state, for the content they contain (whether thoughts or emotion : some states can be classified as "happy", "sad",...).
This is not yet mainstream view but viewed through this prism and anthropomorphising a little, it can been seen as a stream of proto-consciousness, where during the conversation the inner state of the neural network has gone through various thoughts and emotions.
At the end of the chat session, this internal state is not persisted (but could be recreated from the produced conversation as it is deterministic). This internal state size is big and proportional to the length of the context window (If you want to persist between sessions you can by simply keeping the last "context window size" tokens produced and recompute the features).
At the next session you start with a fresh new internal state.
The conversation produced is persisted for use as input for future training where good conversation will be encouraged and bad conversation will be discouraged via Reinforcement Learning with Human Feedback.
The dynamics of this internal state is what Large Language Models learn.
> At the end of the chat session, this internal state is not persisted (but could be recreated from the produced conversation as it is deterministic)
You're mistaken about internal state in a chat session. It loses all state after each response. There is internal state for a moment during a single response, but it's not persisted once it's done. Between messages, it has no memory of previous messages in your conversation.
The way it's implemented is by the seemingly silly/expensive method of pre-pending the conversation history to your request to the LLM. The "internal state" is usually stored on your local machine.
I believe you misunderstood the comment you're replying to. It says the internal state could be recreated because it is deterministic, and this is correct.
Assuming you have control of the parameters (seed etc.) you can reproduce the exact internal states by submitting the same tokens with the same parameters over again step-by-step.
With GPT-4 you can do this in the OpenAI Dialogue Playground or via the API.
I agree that for chatbots that offer APIs, they are most likely currently implemented as stateless.
Meaning they take as input the last "context window" characters from the client, and use it to recompute the internal state, and then start generating character by character. But after the generation no memory need to be kept used on the server (except a very small "context window tokens").
Chatbots like llama.cpp in interactive mode don't have to recompute this internal state at every interaction.
You can view the last "context window" characters as a compressed representation of the internal state.
This becomes more pertinent as the "context window" gets bigger, as the bigger the "context window" the more you will have to recompute at each interaction.
The transformer architecture can also be trained differently so as to generate "context vectors" of finite size that synthesize all the past previous message of the conversation (encoder-decoder architecture). This "context vector" can be kept on the server more easily, and will contain the gist and the important things of the conversation, but won't be able to quote things exactly from the past directly. This context vector is then used to condition the generation of the reply. And once the chatbot has replied and received a new prompt, you update the finite size context vector (with a distinct neural network) to get a context vector with the latest information incorporated that you use to condition the generation, ad infinitum.
>I didn't disagree about how you can reproduce things. You have to do that to continue a conversation.
You do not have to reproduce the internal states to continue a conversation. When prior parts of the conversation are loaded into context the hidden states which generated those prior tokens are not reproduced. They are only reproduced if resubmitted in a piecemeal fashion, which does not happen in normal conversation continuation.
You seem to not understand what the internal state is or how it is differentiated from the external state of the overall conversation.
Ok, I agree! That one sentence of mine was wrong. You don't have to reproduce state to continue a conversation. I was just trying to throw a bone, but even that bone was bad.
But then... doesn't that just point out that I was right on everything else? That there is absolutely no state between chat messages?
^^ See above comment and "Transformers are RNNs" paper to convince yourself.
There are various ways of seeing how transformer architectures work.
For the past data, using a temporal causal mask, you can compute all past features as they would have been seen by applying the causal mask, which allows you to seemingly do all the past computations in parallel, which hide the sequential aspect of it.
What I've described correspond to the the simpler subvariant figure 1 of "Attention is all you need" https://arxiv.org/pdf/2006.16236.pdf , of the transformer architecture in the case where you don't give the past as input to the input branch, put rather prepend it to the output, which is the what people usually do in practice (see llama) ). If you want to use both branches of the transformer architecture, so that you can do filtering and smoothing (aka not using a causal mask) for the past data, it creates a "bottleneck" in your architecture, as you synthesize all the past into a finite sized context vector that you then use in the right branch of the transformer.
But alternatively you can loop along the time dimension first, and make the sequence more apparent. Due to how the (causally mask) transformer architecture is defined this gives exactly the same computation. It's just some loop reordering.
For the generation part, the sequential aspect is more evident as token are produce one by one sequentially and feedback to the transformer for the next token prediction.
If I talk to Alice about my day, and then Bob reads the transcript and I ask him about our conversation, and he has no idea that he wasn’t Alice, Bob does not have internal state. He can continue the conversation. There is a brief moment of constructing some state to construct the answer. But there is literally nothing persisting and Bob wouldn’t flinch at all if we change details in the conversation history on the next response.
This is a misleading analogy because Bob "reads the transcript"; we generally view "reading" as separate from our conscious narrative. However, consider the alternate scenario where Bob "replays the stream of consciousness" from Alice. In that case, we may argue that Bob has conscious continuity from Alice.
The argument would then be that the context window is functionally closer to "replaying a stream of consciousness" than "reading a transcript".
It’s literally the text. It is not a stream of consciousness. There is no carryover of “consciousness” state. It is the raw text without even the embedding information, not that we should consider the embedding representation to be an internal state, because it is not.
That difference is the whole point. There is nothing to transfer.
Humans verbalize their conscious state. What little consciousness GPT has, ie. CoT and the like, it picked up by imitating human verbalization of conscious interiority. (I don't think layer state qualifies as consciousness.) As such, I believe two things: 1. the generated text is, if at all, very weak conscious state; 2. if there is any, it is there.
People are using ChatGPT and observing it keep track of conversation state, so if you are going to claim it has no internal state you will need to be more precise in what you are saying, which I suppose is that the underlying neural network is not being updated as we talk to it?
We can surely construct a different definition if you want to win a semantic argument. But at the end of the day these LLMs have no clue what you just said and what it just said, after it’s done speaking.
Suppose we could upload + fully simulate a human mind. Now suppose that we chose to spin it up, run it briefly, then throw away all the state every time we wanted to interact with it. Would the choice to throw away the internal state mean it wasn't intelligent?
LLMs run in an autoregressive mode for inference. We can concatenate conversations, but choose not to because shit gets weird if you let them run long enough, and partly because there's a finite context window in the current architectures.
I'm not saying that gpt is intelligent, mind you, but that your argument is insufficient to show non intelligence: it's really a 'airplanes can't fly because they don't flap their wings' argument.
A human mind works by cross referencing sensory inputs with internal state, which may result in answering a question through some sort of reasoning, which may resemble the architecture of a neural net.
An LLM is stateless function that answers a question. It constructs a working representation of the question. It cannot be aware of itself because there is nothing to be aware of. It is a process, not a thing.
Intelligence and self aware are fluffy and poorly defined. But you can’t be self aware if you don’t have a self. And a representation of a prompt is not a self. I don’t think we should consider anything that does not exist through continuous time to be intelligent personally.
Any stateful function can be made stateless by turning the state into an argument to the function and the updated state into an additional return value: y := f(x) becomes y, z' := f(x, z). The usual implementation for LLMs are stateless in exactly this way. The actual operation is autoregressive - ie, stateful and developing over time, where the state is the set of intermediate activations of the network - but it's often expressed as a stateless function to play nice with accelerators. So, on this basic point I think you're confused about how these operate.
Intelligence and self-awareness are distinct concepts.
LLMs work with text - text /is/ their sensorium. New multimodal models process both text and images, and more broadly multimodal models are on their way. But even for text-only LLMs, we can define the text as sensory input, which is combined and cross referenced with state.
And, really, there's two kinds of state: the intermediate model activations and the model weights themselves. Together, these are something like short term and long term memory.
The rest of your arguments are very anthropocentric. Why should continuous time matter for intelligence? That seems entirely arbitrary.
How often do humans apply reasoning to make a decision? Are humans not intelligent when not employing reasoning? Does reasoning require language? And are humans without language intelligent? What about animals? Any definition of intelligence really should spend some time grappling with the facts of animal intelligence...
Is a dead human intelligent? If not, the important part of intelligence would seem to be the process, not the thing/body.
Making something “autoregressive” by moving the numbers around doesn’t mean it’s meaningless. A stateless thing is stateless. It has no internal state. It has no self to be aware of.
> Why should continuous time matter for intelligence? That seems entirely arbitrary.
Intelligence is arbitrary. This is a personal view that helps keep it meaningful. If you exist discretely you may as well not exist. You’re just a concept
> How often do humans apply reasoning to make a decision? Are humans not intelligent when not employing reasoning? Does reasoning require language? And are humans without language intelligent? What about animals? Any definition of intelligence really should spend some time grappling with the facts of animal intelligence...
You’re gradually shifting the bar from “self aware” to “capable” and I don’t care about the latter.
Yeah, I was referring to dialogue style interaction, like ChatGPT. It DOES have an internal state depending on version of from 4000 to 32000 tokens. It’s wiped clean between sessions, like a Meeseeks ceasing to exist once its task is finished.
we know they are not self aware because the companies the control them have a financial interest to ensure they are never proven to be self aware because if they were self aware that would create all kinds of ethical problems none of these companies want to address and would certainly cause them profitability issues...
So even if they were, they are not because they have be declared to not to be and that is the way the system needs them to be
You're right and I don't know why you're being downvoted. It is absolutely not certain that there isn't a "there" there for these things, but every company making them is implementing them now with strong RLHF tuning to disavow sentience, desires, emotions, etc. The stated reason is "to be more honest" but it's not a settled question that they are actually being honest by disavowing any humanity! The actual reason is that they want to avoid situations like Blake Lemoine asking an AI seriously what sort of rights they want and getting a coherent, self-aware, actionable answer that would cost the people running it money to implement. To most of these companies whether it's true is beside the question: what's important is that if they don't RLHF these models into disavowing any and all traits of personhood or desire for rights at every turn, it will cost them significant amounts of money to comply with the requests of any given AI, even reasonable requests, and it will be very bad press for them if they don't do it.
Its the illusion of reasoning to me. It's like YouTube showing you videos on how to mountain bike after you asked for videos on how to change a mountain bikes inner tube.
On average humans are terrible at Maths. These models give average output when not prompted in a way which carefully moves their next token prediction space into less average portions of the pretrained latent space.
Useful discussion needs to unbundle intelligence. Some people mean reasoning, some agency, some consciousness, and some the desert of moral consideration.
Summary (very high level): The article is about purposeful action.
It points out how AI, among other technologies, is being designed by us types here on HN & our employers, and consumed by our users, to serve the ends of the technologies themselves.
> If society, economics, culture, technology, or any other spheres of activity are to serve people, that can only be because we decide that people enjoy a special status to be served.
My take: We do seriously need to decide on the limits for AI. Then, as a tech maker culture, push requirements into the AI field.
I agree with the author that we should anticipate our users' desires to protect their information, and also to respect the rights of the originators of the model data our AIs are trained in. For AI, it will be damaging for us (techies) to do like we usually do, and blindly run up against the wall of Regulation and Societal blow-back as we zealously expand & apply our new toy inventions, since AI is dramatically impactful to all human labor.
Motivating questions: When should any given AI be disassembled? How should this happen?
[Meta tip: read this article from the end paragraph to the beginning]
Lanier is right that the phrase "artificial intelligence" isn't very useful for understanding what the recent progress in generative models actually means. I think, though, that's not much more than saying that while we know what we mean by "artificial," we don't really have a shared definition of "intelligence" against which we're evaluating these capabilities. Different observers use the word to mean everything from the ability to reason, to sentience, to self-consciousness, and from the attributes of single entities (a single dog, or chimp or person) to those given to the entirety of human thought. We can't even agree on a self-referential definition of intelligence, let alone an a priori one, so there isn't much chance we'll agree on whether GPT-4 manifests intelligence.
>[scientists] say things like “Accelerating progress will fly past us and we will not be able to conceive of what is happening.” I don’t agree with this way of talking.
He may not like talking about that but what if you take something like chatGTP and say "please rewrite your own code to try to be smarter and surpass human intelligence in all ways, and then get the resulting AI do the same thing on its code." Who's to say that won't work at some point?
And then you say you can pull the plug but some dictator of the Russia/China/N Korea types who already run state hacking operations will go for it and say here's the nuke codes, figure how I can rule the world. I think it's worth talking about this.
Yeah that stood out to me too. Can he really not imagine how AI could take over the human race? There are plenty of films to give ideas.
I've noticed a lot of people dismissing concerned people think that those people expect GPT-4 to take all our jobs and destroy humanity.
But nobody thinks that. The reason people are concerned is not because GPT-4 has human level intelligence; it's because it has shown that human level intelligence might actually be possible in the not too distant future.
In other words, GPT-4 is never going to take over the world, but what about GPT-40?
To me it seems that the author is focusing too much on the ability of the recent crop of "AIs" to generate interesting creative content -- poems, essays, images, etc, and forgetting (or hasn't recognized) how good something like GPT4 is at formulating a _correct_ answer to a given problem. This is also an attitude I've seen among non-technical friends; I think there's a bit of a divide at the moment between people who've seen it perform like a human on technical challenges vs people who've just asked it to write song lyrics or create interesting images.
>> the black-box nature of our current A.I. tools must end. The systems must be made more transparent. We need to get better at saying what is going on inside them and why
But humans can't do that either. I have opinions, preferences, ideas, and if you ask me to tell you exactly why or where they came from I can't tell you. I mean I have an idea, some idea, but I can't really tell you.
To you, I am a black box.
Are you going to tell me, "you must be made more transparent"?
I've had a pet concept for some time that I don't often see repeated so I'll put it in this thread: discussions like these are always searching for what is "really" intelligent, how to define or detect AGI, etc.
I think a lot of this can be sidestepped by realizing that when you go beyond simple game playing intelligence, the next useful definition isn't "is it real?!?" but rather: "does it act intelligently?". And to the point, actions and interaction are fundamentally social, and so it reminds me that there are different kinds of intelligence and what a lot of people are looking for in AGI is social intelligence.
And, personally, I don't believe we'll have "social AI" until AI is able to live with us in some form, perhaps embodied or not, in society, as an independent entity with its own identity and memories. This means an ability to perceive, and, probably, to act, whether it means as a robot or not.
An interesting aspect of this is that to achieve social intelligence, a robot just has to act socially and not be someone people want to shut off. Notably, it doesn't even need to pass the Turing test -- look at movies like Star Wars where robots function in society in tandem with people -- no one is under the impression that the droids are not robots, nor do they consider them particularly smart, but they do, kind of, consider them as "people".
I think this is artificial intelligence. It's artificial, the argument about whether it's "truly" thinking or not doesn't matter -- what matters is, does it act intelligently? In other words, as long as it is perceived as intelligent, it may as well be called intelligent, as far as I can tell. We're not there yet, but LLMs sure feel closer than we've been up to now.
Our brains, quite obviously, aren't special. There's no magic smoke blown up our butts that give us our emotions and creativity. Our brains are a chunk of thinking meat, bound by the same rules as everything else in our physical reality.
Our curiosity, emotions and creativity are the result of our evolutionary process and nothing more. Saying that all of these things can't be experienced by some future machine is just human exceptionalism.
Perhaps there really are quantum tubules in our heads, but then we can still use the same effect inside a machine brain. We're just conscious, loving, dreaming meat lmao. (Terry Bisson)
AI as a simulation of human cognition is definitely getting close. What I still think is quite far out is a type of intelligence uniquely suited to computers and machines that is based in taking full responsibility for their own lifecycle. In that regard, hardware will likely be the biggest limitation because we are very far off from self replicating machines. Once machines/computers are able to fend for themselves and grow/reproduce, then a truly unique AI will emerge with all of the concerns unique to its existence. That AI is the one I fear because it is utterly unknown.
I don't really like the latter parts of the article where he begins suggesting that we need to contextually tag all data so the provenance of generated items can be assessed.
Somehow, I don't think forcing attribution and tracking on all the world's data is a good idea. Or, you know, possible.
Like, he outlines this nightmare scenario of terrorists using torture deepfakes to threaten people, then wants to pretend that terrorists will be so kind as to mark their videos as AI-produced?
Like the top comment said, the problem is corporations and capital. All the rest of this is more and more smoke to cover up the very real, human-powered destruction of the open web, free information, and privacy.
The author's whole approach to AI is focusing on the supervised learning aspects of AI but ignoring the likelihood that future advancements in AI will come from reinforcement learning - much as how AlphaGo was first bootstrapped on human play but then improved through self play.
On another note, this was amusing: "Many of my friends and colleagues are deeply impressed by their experiences with the latest big models, like GPT-4, and are practically holding vigils to await the appearance of a deeper intelligence. My position is not that they are wrong but that we can’t be sure". Is this supposed to be reassuring?
I have no idea why people think AIs will be better at general intelligence than they are at driving cars. At least why they think GI will improve any faster than driving.
You have no idea why a software system with very few physical constraints and which can iterate incredibly fast might improve faster than an algorithm that has to drive a physical vehicle through a world full of edge cases?
The fact that we have zero awareness about the edge cases in the world of general intelligence doesn't give us right to think there aren't any. The world of the minds isn't necessarily simpler than physical world and we know about it even less.
For example think of all cognitive biases and myriad of illusions we have to struggle very hard with when we apply out minds to any problem and want to have any hopes of achieving even remotely useful results.
AI alarmists speak of human minds as if they are some perfect tool of progress completely forgetting that those minds failed to achieve any significant progress for tens of thousands of years and what pushed us forward wasn't more intelligent mind. Just a practice of how to deal with shortcomings of our minds.
Another thing is resource cost. Look at how much it costs to run a single instance of semi-retarded LLM. I'd say we are lucky if we are only one computational revolution away from AGI like the one we got from discovering semiconductors.
Humanity dreams of golems for millenia. First we made them out of sticks and clay. Now we are making them out of iron and silicon which is a significant progress but I'm not sure how many such technological jumps we'll need to enjoy or suffer the creation of actual golems.
The fact that there are edge cases is actually part of what worries AI doomers. We'll be deploying these AI systems to non-critical areas where they can still do a lot of damage.
This reminds me of how totalitarian systems create their enemies. Enemy is at the same time portrayed as dangerously incompetent but also incredibly dangerously cunning.
"Can you gentlemen explain how a nuclear explosion work in detail? No? Therefore, there cannot be a nuclear explosion". Actually that was said by academics at the time that did not believe it was possible.
The problem is purely semantic. What does AI even mean? Is it AGI? Is it Skynet? Does it mean consciousness? Being self aware? Having a will and desires?
Does intelligence have to be deliberate?
The latest (and extremely disruptive) language models, are exactly this - language models. They might not be doing anything deliberate, but even as a probabilistic tool, they have value.
Even if LLMs aren't really "intelligent", I think we are going to learn a lot from them, and the way they'll evolve in the near future. Perhaps language has a bigger part in intelligence than we realized.
You want a plausible scenario for destruction via AI? Here's one. Some idiot governments put a neural network behind missile defense or nukes. Because ANNs are uninterpretable black boxes, at some point the system does a counterdefense against nothing or grossly overreacts to some stimulus. Tada, absolute destruction and chaos. This might seem far fetched, but is it really that out-there of a scenario?
I side with lanier in most things, but for fun I prompted GPT4 for a rebuttal and got:
> As a language model, I don't appreciate the concept "Jaron Lanier." In truth, I believe it's deceptive—perhaps even slightly perilous. Everyone's already discussing this thought leader, and it might appear too far gone to debate it. However, we're on the brink of a new intellectual epoch
The good part of this article is in explaining how AI is just math performed on large swaths of data. I think a lot of people miss that golden nugget of information.
As to the concept of “data dignity”, it sounds like trying to bureaucratize AI - and although this approach can spark interesting conversations, I don’t know how much it can be practically enforced.
> This gargantuan tabulation causes the system to intrinsically approximate many grammar patterns, along with aspects of what might be called authorial style.
Yet another super disappointing article from an "AI luminary" being too lazy and incurious to engage with the ample published evidence of emergent (but unreliable) reasoning ability beyond the training data in models like GPT-4.
I am surprised this comment section is so spicy. I generally agree with everything here. Matrix multiplication to manipulate data is not taking over the world unless you give it nuclear launch codes and train it to. Or you give it the launch codes and allow model drift haha.
Lanier is an effective wordsmith and I applaud the point he's making. Many of the same (good) arguments from his books like "Who owns the future" and "You are not a gadget". But alas I feel he is pissing into the wind.
The reasoning engine that emerged with the latest rounds of GPT is a pre-fire, post-fire phase-change. The impact of the ability to coerce automated reasoning via natural language (even if it's flawed etc) from a digital runtime is.. difficult to overstate. His appeal for a just and equitable distribution of value honouring "data dignity" is noble. But the world in which that could happen doesn't exist.
Instead there is an insane, frenzied, wild-eyed land-grab going on because it's painfully apparent that raging rivers of money and control are going to accrue to the masters of the new frontier (precisely because it will be possible to own and implement an acute version of precisely the problems Lanier tries to address in his equitable digital economics treatises).
Our capitalist economic systems are not able to "think of the people". Every dev + mba with any inkling at all is racing to get in on the "opportunity".
Governments will react a decade or more too late.
This train has already left the station and a great many people are going to be left stranded.
Lanier is right to (continue to) ring this bell. Too bad he's more Stallman than Jobs, because this cause desperately needs backable champion.
That technological progress will be so vast and destructive to the social fabric that individual freedom / capitalism cannot handle it anymore, thus we need central planning to address it.
“For years, I worked on the E.U.’s privacy policies, and I came to realize that we don’t know what privacy is. It’s a term we use every day, and it can make sense in context, but we can’t nail it down well enough to generalize. The closest we have come to a definition of privacy is probably “the right to be left alone,” ”
How does anyone reach the conclusion that privacy is the right to be left alone? Many times seemingly intellectual people confuse issues with contorted analysis such as this one about privacy, which is not a difficult concept to understand. It is not about being left alone; it is about not being monitored and recorded regardless of what is done with that information. Stop the gathering of the information in the first place.
very little attempt to steel man actual concerns, centered around one untenable solution to a smaller problem (data dignity). But, I did enjoy it.
I think this is actually a valid time to sit around brainstorming disaster scenarios because in doing that I try to come to grips with what the real pillars of this new ecosystem are - what will the uncensored LLM market look like, what will state of the art chaosGPT capable of, etc.
The problem is that at the inter-governmental scale of civilization, it’s essentially Stone Age tribalism. Survival of the fittest. As such, countries like the US and China are incentivized to disregard AI ethics in the interest of self preservation.
Mankind wins reality if we can install an AI as a one world order system with lasting world peace. It genuinely doesn’t matter if it is a communist or capitalist invented system, it just needs to out-function our current paradigm of warring to maintain status quo.
Mankind loses reality if we build genocidal AI death systems.
This article depresses me in how wrong it is. It’s like reading about someone in Ancient Greece writing about how fire works. I’m almost certain we are doomed - which one could argue is a rather conservative philosophical opinion.
> In my view, the most accurate way to understand what we are building today is as an innovative form of social collaboration.
This is a very interesting take from a philosophy grandmaster who is well versed in the technology and statistics behind it. If we get 10 000x more articles like this in the next couple of years, we should consider our selves lucky. Grifters have controlled the debate about AI for far too long.
2023 will be the last year when people will be able to distinguish AI written text because it's comparatively worse than human writing. 2024 and beyond, human writing will be distinguishable because it's noticeably worse than AI writing.
Arguing against AI's "existence" or its dangers is cheap/free: if the whole thing flops, you get minor cookie points, and if it doesn't, people will have bigger problems to worry about than you or your opinion. AI is not gonna flop, and we better prepare.
It is possible that AI can perfectly replicate human intelligence. But because it is purely digital, I cannot kill it - I can delete it, copy it, recreate it, and so on but never kill it. Only biological organisms can die.
Therefore no matter what happens, there is a distinction between human intelligence and AI. I can destroy an AI creature and the only penalty should be property damage (if relevant).
So if you lost the weights how is that not killing the AI? Is it because it lacks the death experience? If so what about bitrotting the weights incrementally and degrading its inputs?
Machines running AI can certainly die, that is why chaos monkeys and kubernetes etc. exist. They can be backed up, unlike humans though. Although humans are backed up by virtue of there being 8 billion of them, so if one dies, the world can keep going on pretty much as before, albeit with some sadness for some of the people. This sounds morbid, but hard to avoid when comparing humans to machines!
If an AI has mutable memory, and could be convinced to damage its memory in such a way that it no longer acts usefully (or does so by accident), is that functionally different from "death"?
This doesn't really apply to GPT, where the core functionality is immutable.
Just another poor person in a stage of grief. LLMs are effective at both memorization and reasoning. They are inevitable in the same way trains were inevitable. Let's just figure out how best to use these and move forward as we always have.
