After reading the paper I’m struck by the lack of any discussion of awareness. Cognition requires at its basis awareness, which due to its entirely non verbal and unconstructed basis, is profoundly difficult to describe, measure, quantify, or label. This makes it to my mind impossible to train a model to be aware, let alone for humans to concretely describe it or evaluate it. Philosophy, especially Buddhism, has tried for thousands of years and psychology has all but abandoned attempting so. Hence papers like this that define AGI on psychometric dimensions that have the advantage of being easily measured but the disadvantage of being incomplete. My father is an emeritus professor of psychometrics and he agrees this is the biggest hurdle to AGI - that our ability to measure the dimensions of intelligence is woefully insufficient to the task of replicating intelligence. We scratch the surface and his opinion is language is sufficient to capture the knowledge of man, but not the spark of awareness required to be intelligent.
This isn’t meant to be a mystical statement that it’s magic that makes humans intelligent or some exotic process impossible to compute. But that the nature of our mind is not observable in its entirety to us sufficient that the current learned reinforcement techniques can’t achieve it.
Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task. You will be able to. How did you do this without thought? We’ve all had sudden insights without deliberation or thought. Where did these come from? By what process did you arrive at them? Most of the things we do or think are not deliberative and definitely not structured with language. This process is unobservable and not measurable, and the only way we have to do so is through imperfect verbalizations that hint out some vague outline of a subconscious mind. But without being able to train a model on that subconscious process, one that can’t be expressed in language with any meaningful sufficiency, how will language models demonstrate it? Their very nature of autoregressive inference prohibits such a process from emerging at any scale. We might very well be able to fake it to an extent that it fools us, but awareness isn’t there - and I’d assert that awareness is all you need.
Awareness is just continuous propagation of the neural network, be that artificial or biological. The reason thoughts just "appear" is because the brain is continuously propagating signal through the neural network. LLMs also do this during their decoding phase, where they reason continuously with every token that they generate. There is no difference here.
Then you say "we don't think most of the times using language exclusively" , but neither do LLMs. What most people fail to realise is that in between each token being generated, black magic is happening in between the transformer layers. The same type of magic you describe. High dimensional. Based on complex concepts. Merging of ideas. Fusion of vectors to form a combined concept. Smart compression. Application of abstract rules. An LLM does all of these things, and more, and you can prove this by how complex their output is. Or, you can read studies by Anthropic on interpretability, and how LLMs do math underneath the transformer layers. How they manipulate information.
AGI is not here with LLMs, but its not because they lack reasoning ability. It's due to something different. Here is what I think is truly missing: continuous learning, long term memory, and infinite and efficient context/operation. All of these are tied together deeply, and thus I believe we are but a simple breakthrough away from AGI.
There are very significant differences between biological and artificial neural networks. Artificial neural networks are mathematical attempts to replicating how the brain’s neurons work. They are not and were never meant to be 1 to 1 replications. There is the difference in scale, where the “parameters” of human neural networks absolutely dwarf the current LLMs we have today. There is also the fact that they are materially different. The underlying biology and cell structure affects biological neural networks in ways that artificial neural networks just simply dont have access to.
The idea of awareness being propagations through the NN is an interesting concept though. I wonder if this idea be proven through monitoring the electrical signals within the brain.
People like to focus on the differences between the brain and artificial neural networks. I myself believe the only thing that truly matters is that you can form complex functions with the common neuron element. This is achieved via linking lots them together, and by each having a property known as non-linearity. These two things ensure that with neurons you can just about approximate any linear or non-linear function or behaviour. This means you can simulate inside your network pretty much any reality within this universe, its causation and the effects.
The deeper your network the more complex the reality you can "understand". Understand just means simulate and run inputs to get outputs in a way that matches the real phenomenon. When someone is said to be "smart", it means they possess a set of rules and functions that can very accurately predict a reality.
You mention scale, and while its true the number of neuron elements the brain has is larger than any LLM, its also true the brain is more sparse, meaning much less of the neurons are active at the same time. For a more fair comparison, you can also remove the motor cortex from the discussion, and talk just about the networks that reason. I believe the scale is comparable.
In essence, I think it doesn't matter that the brain has a whole bunch of chemistry added into it that artificial neural networks don't. The underlying deep non-linear function mapping capability is the same, and I believe this depth is, in both cases, comparable.
While thinking that current mathematical model replicate accurately a fondamental aspect of biological neural network might be right, it doesn't mean that nothing is missing to achieve the stated goal of true intelligence.
Maybe we've just reach the ability the replicate the function of an artificially powered dead brain that would be randomly stimulated and nothing more. Is this really a path to intelligence ?
Isn't the brain randomly stimulated already? Even not being dead? Don't you think the complex reasoning is a cause of the neurons themselves and not the stimulation?
Animals are alive and are not nearly as smart. Its because their neural networks are not as deep. Its not for the lack of proper chemistry or stimulation.
Why would it have to be a 1 to 1 replication? Isn't that a strawman argument? NNs can basically store the collective of knowledge of humanity in that miniscule amount of neurons. NNs also run at much much higher frequency than human brains. Does that make human brains inferior and not worthy of being called aware by the same line of argumentation? Why do these differences even matter? I can imagine a vastly different form of awareness than humans just fine. They can both be aware and not that similar.
> What most people fail to realise is that in between each token being generated, black magic is happening in between the transformer layers.
Thank you by saying that. I think most people have an incomplete mental model for how LLMs work. And it's very misleading for understanding what they really do and can achieve. "Next token prediction" is done only at the output layer. It's not what really happens internally. The secret sauce is at the hidden layers of a very deep neural network. There are no words or tokens inside the network. A transformer is not the simple token estimator that most people imagine.
> Awareness is just continuous propagation of the neural network, be that artificial or biological. The reason thoughts just "appear" is because the brain is continuously propagating signal through the neural network.
This is just a claim you are making, without evidence.
The way you understand awareness is not through "this is like that" comparisons. These comparisons fall over almost immediately as soon as you turn your attention to the mind itself, by observing it for any length of time. Try it. Go observe your mind in silence for months. You will observe for yourself it is not what you've declared it to be.
> An LLM does all of these things, and more, and you can prove this by how complex their output is.
Complex output does not prove anything. You are again just making claims.
It is astoundingly easy to push an LLM over to collapse into ungrounded nonsense. Humans don't function this way because the two modes of reasoning are not alike. It's up to those making extraordinary claims to prove otherwise. As it is, the evidence does not exist that they behave comparably.
The sentence "It is astoundingly easy to push an LLM over to collapse into ungrounded nonsense" makes me wonder.
How easy? What specific methods accomplish this? Are these methods fundamentally different from those that mislead humans?
How is this different from exploiting cognitive limitations in any reasoning system—whether a developing child's incomplete knowledge or an adult's reliance on heuristics?
How is it different from Fake News and adults taking Fake News for granted and replicating bullshit?
Complex output can sometimes give you the wrong idea, I agree. For instance, a study Anthropic did a while back showed that, when an LLM was asked HOW it performed a mathematical computation (35 + 59), the response the LLM gave was different from the mechanistic interpretation of the layers [1]. This showed LLMs can be deceptive. But they are also trained to be deceptive. Supervised fine tuning is imitation learning. This leads the model to learn to be deceptive, or answer what is usually the normal explanation, such as "I sum first 5+9, then add the remainder to... etc". The LLM does this rather than actually examining the past keys and values. But it does not mean it can't examine its past keys and values. These encode the intermediate results of each layer, and can be examined to identify patterns. What Anthropic researchers did was examine how the token for 35 and for 39 was fused together in the layers. They compare these tokens to other tokens, such as 3 , 5 , 9. For an LLM, tokens are high dimensional concepts. This is why you can compare the vectors to each other, and figure out the similarity, and therefore break down the thought process. Yes, this is exactly what I have been discussing above. Underneath each token prediction, this black magic is happening, where the model is fusing concepts through summation of the vectors (attention scores). Then, merged representations are parsed by the MLPs to generate the refined fused idea, often adding new knowledge stored inside the network. And this continues layer after layer. A repeated combination of concepts, that start with first understanding the structure and order of the language itself, and end with manipulation of complex mathematical concepts, almost detached from the original tokens themselves.
Even though complex output can be deceptive of the underlying mental model used to produce it, in my personal experience, LLMs have produced for me output that must imply extremely complex internal behaviour, with all the characteristics I mentioned before. Namely, I frequently program with LLMs, and there is simply zero percent probability that their output tokens exist WITHOUT first having thought at a very deep level about the unique problem I presented to them. And I think anyone that has used the models to the level I have, and interacted with them this extensively, knows that behind each token there is this black magic.
To summarize, I am not being naive by saying I believe everything my LLM says to me. I rather know very intimately where the LLM is deceiving me and when its producing output where its mental model must have been very advanced to do so. And this is through personal experience playing with this technology, both inference and training.
What makes you think you can understand the subjective experience of LLMs then? It's out of reach of science, so the only way is to ask them? How can you be sure they don't have subjective experience? Remember that you forbade yourself from using science to answer it.
Also, if subjective experience has any effect on the material world, then we can measure it and test it, putting it in reach of science. If it doesn't, why does it even matter in this discussion? By definition it has no relation to the AGI discussion since that's an empirical matter.
Haha, well, I would appreciate if comments included more substantive evidence when they make claims like they did.
I see a lot of "AGI boosters/doomers" comfortable making big claims without providing research to back what, when challenged, prove to be just their model or feeling of how things function.
Oh I agree with you, I was just underscoring that.
It seems to be a case of people looking at a problem they have little knowledge or understanding of and thinking "how hard can it be"? In this case, the answer is "so hard that philosophers have dubbed it 'the hard problem'".
Philosophers are mostly unaware of artificial neural networks. The game has changed, you can understand a lot about the human mind if you understand AI. Don't get too stuck in the past.
How about an objection to what I said? A case where someone is conscious but without continuous propagation of neural signals? Or something
You didn’t provide enough of a hypothesis to seriously discuss.
> A case where someone is conscious but without continuous propagation of neural signals?
That would be irrelevant. All known conscious beings are made up of biological cells, but that doesn’t prove that all conscious beings must be made of biological cells, or that biological cells are the key causative factor in consciousness. The same goes for “continuous propagation of neural signals.”
You described a personal conjecture as though it solved a known hard problem, even throwing in the word “just” as though the solution is really simple. This is a lot like the Feynman quote about quantum mechanics: if you think you understand it, you almost certainly don’t. You may not even have recognized the core problem yet. The original Chalmers paper is a good place to start: https://consc.net/papers/facing.pdf
But coming at it from a computational perspective, in some ways it’s even easier to see the problem. We don’t generally assume that a deterministic, non-neural net program written in say Python has a conscious subjective experience. To use Nagel’s terminology, there is “nothing it is like” to be that program. But, an LLM or any other computational neural net is no different from a program like that. It’s executing deterministic instructions, like a machine, because it is a machine. We can speculate about consciousness being some sort of emergent property that arises in such systems given the right conditions, but that’s all it is: speculation.
And it’s completely unclear what those right conditions might be, or how those conditions could possibly give rise to conscious experience. Why aren’t humans philosophical zombies with no conscious experience, just reacting to input like machines? No-one has succeeded in getting past the conjecture stage in answering that question.
I so completely agree. In virtually every conversation I have heard about AI, it only every talks about one of the multiple intelligences as theorized in Howard Gardner's book Frames of Mind: The Theory of Multiple Intelligences (1983)[1]
There is little discussion of how AI will enhance (or destroy) our emotional intelligence, or our naturalistic, intrapersonal or interpersonal intelligences.
Most religions, spiritual practices and even forms of meditation highlight the value of transcending mind and having awareness be present in the body. The way AGI is described, it would seem transcendence may be treated as a malfunction or bug.
There is no way to measure awareness. We can only know we are aware ourselves. For all we know trees or rocks might have awareness. Or I could be the only being aware of itself in the universe. We have no way to prove anything about it. Therefore it is not a useful descriptor of intelligence (be it human, animal or artificial).
Agreed. Everything that looks like intelligence to ME is intelligent.
My measurement of outside intelligence is limited by my intelligence. So I can understand when something is stupider compared to me. For example, industrial machine vs human worker, human worker is infinitely more intelligent compared to machine, because this human worker can do all kinds of interesting stuff. this metaphorical "human worker" did everything around from laying a brick to launching a man to the Moon.
....
Imagine Super-future, where humanity created nanobots and they ate everything around. And now instead of Earth there is just a cloud of them.
These nanonobots were clever and could adapt, and they had all the knowledge that humans had and even more(as they were eating earth a swarm was running global science experiments to understand as much as possible before the energy ends).
Once they ate the last bite of our Earth(an important note here: they left an optimal amount of matter to keep running experiments. Humans were kept in a controlled state and were studied to increase Swarm's intelligence), they launched next stage. A project, grand architect named "Optimise Energy capture from the Sun".
Nanobots re-created the most efficient ways of capturing the Sun energy - ancient plants, which swarm studied for centuries. Swarm added some upgrades on top of what nature came up with, but it was still built on top of what nature figured by itself. A perfect plant to capture the Sun's energy. All of them a perfect copy of itself + adaptive movements based on their geolocation and time(which makes all of them unique).
For plants nanobots needed water, so they created efficient oceans to feed the plants. They added clouds and rains as transport mechanism between oceans and plants... etc etc.
One night the human, which you already know by the name "Ivan the Liberator"(back then everyone called him just Ivan), didn't sleep on his usual hour. Suddenly all the lights went off and he saw a spark on the horizon. Horizon, that was strongly prohibited to approach. He took his rifle, jumped on a truck and raced to the shore - closest point to the spark vector.
Once he approached - there was no horizon or water. A wall of dark glass-like material, edges barely noticeable. Just 30 cm wide. On the left and on the right from a 30 cm wide wall - an image as real as his hands - of a water and sky. At the top of the wall - a hole. He used his gun to hit the wall with the light - and it wasn't very thick, but once he hit - it regenerated very quickly. But once he hit a black wall - it shattered and he saw a different world - world of plants.
He stepped into the forest, but these plants, were behaving differently. This part of the swarm wasn't supposed to face the human, so these nanobots never saw one and didn't have optimised instructions on what to do in that case. They started reporting new values back to the main computer and performing default behaviour until the updated software arrived from an intelligence center of the Swarm.
A human was observing a strange thing - plants were smoothly flowing around him to keep a safe distance, like water steps away from your hands in a pond.
"That's different" thought Ivan, extended his hand in a friendly gesture and said
- Nice to meet you. I'm Ivan.
....
In this story a human sees a forest with plants and has no clue that it is a swarm of intelligence far greater than him. To him it looks repetitive simple action that doesn't look random -> let's test how intelligent outside entity is -> If entity wants to show its intelligence - it answers to communication -> If entity wants to hide its intelligence - it pretends to be not intelligent.
If Swarm decides to show you that it is intelligent - it can show you that it is intelligent up to your level. It won't be able to explain everything that it knows or understands to you, because you will be limited by your hardware. The limit for the Swarm is only computation power it can get.
We don't want awareness because it begets individuals by means of agency and we'd need to give them rights. This is industry's nightmare scenario.
People want autonomy, self-learning, consistent memory and perhaps individuality (in the discernability/quirkiness sense), but still morally unencumbered slaves.
Any definition of AGI that doesn't include awareness is wrongly co-opting the term, in my opinion. I do think some people are doing that, on purpose. That way they can get people who are passionate about actual-AGI to jump on board on working with/for unaware-AGI.
Because LLMs don't have this special quality that you call "awareness", then they cannot have "cognition", neither of which you defined? This is along the lines of "There must be something special with my mind that LLMs don't have, I can just feel it" special pleading whether you call it awareness, consciousness, qualia etc.
As long as you cannot define it clearly or even show that you yourself have this quality, I think the burden of proof is on you to show why this has any real world implications rather than just being word play. We can build thinking, reasoning machines just fine without waiting for philosophers to finally answer what consciousness is.
> Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task.
I do not have any even remotely practical definition for this, but this has to somehow involve the system being in a closed loop. It has to "run" in a sense that an operating system runs. Even if there is nothing coming on certain inputs it still has to run. And probably hallucinate (hehe) like humans do in an absence of a signal or infer patterns where there are none, yet be able to self-reflect that it is in fact a hallucination
Anything that is not measurable (i.e. awareness, consciousness) is not very useful in practice as a metric. I don't think there is even an agreed definition what consciousness is, partially because it is not really observable outside of our own mind.
Therefore I think it makes perfect sense that awareness is not discussed in the paper.
Consciousness is observable in others! Our communication and empathy and indeed language depend on the awareness that others share our perceived reality but not our mind. As gp says, this is hard to describe or quantify, but that doesn't mean it's not a necessary trait for general intelligence.
But LLMs have been measured to have some theory of mind abilities at least as strong as humans: https://www.nature.com/articles/s41562-024-01882-z . At this point you either need to accept that either LLMs are already conscious, or that it's easy enough to fake being conscious that it's practically impossible to test for - philosophical zombies are possible. It doesn't seem to me that LLMs are conscious, so consciousness isn't really observable to others.
That's still using language. My dog has theory of mind in the real world where things actually exist.
Also, those results don't look as strong to me as you suggest. I do not accept that an LLM is conscious nor could I ever unless I can have a theory of mind for it... Which is impossible given that it's a stochastic parrot without awareness of the things my five senses and my soul feel in reality.
Language is one of communication contracts. LLModels leverage these contracts to communicate data structures (shapes) that emerge when evaluating input. They are so good at prediction that when you give them a clue of a shape they will create something passable, and they keep getting better with training.
I hear there's work being done on getting the world models out, distilling the 'cortex-core' (aka the thinking without data), to perhaps see if they're capable of more, but so far we're looking at holograms of wishful thinking that increase in resolution, but still lack any essence.
This begs a question - can true intelligence even be artificial?
I'd argue the biggest issue with concretely defining intelligence is that any attempts end up falling in two buckets:
1. "Too" Broad, which raises uncomfortable questions about non-human intelligence and how we as humans treat them (see: whales, elephants, octopuses/cephalopods)
2. Too narrow, which again raises very uncomfortable issues about who and who does not qualify as human, and what we do with them.
Put in other words, it's more an issue of ethics and morals than it is definitional.
Awareness doesn't seem that hard for AI systems though - if you look at the screen on a self driving Tesla you can see if it's aware of pedestrians, cyclists etc. because it draws boxes around them on the screen as it becomes aware of them.
I guess by 'AGI' most people mean human level or above so I guess you'd want human level awareness which Teslas and the like don't have yet.
Can't "awareness" in both examples be approximated by a random seed generator? Both the human mind and autoregressive model just need any initial thought to iterate and improve off of, influenced by unique design + experienced priors.
Yep, computers execute code, they are tools. Humans have the capacity to spontaneously generate new thoughts out of nothing, solve problems never before solved and not just by brute force number crunching.
Does any of that argument really matter? And frankly, this statement:
>This makes it to my mind impossible to train a model to be aware
feels wrong. If you're arguing that human's are aware, then it is apparent that it is possible to train something to be aware. Nature doesn't have any formal definition of intelligence, or awareness, yet here we are.
From a practical perspective, it might be implausibly difficult to recreate that on computers, but theoretically, no reason why not.
Have we shown what the human brain does at a “hardware” level? Or are you just assuming that the basic building block of a computer is that same as the basic building block of a human brain?
Basic building blocks are atoms. So, yes same. If you mean cells vs transistors, sure they're different. But we don't have to demonstrate anything to know that nature already made conscious intelligent AGI without it itself not understanding anything. Therefore AGI can be created without knowing what consciousness is. It happened at least once.
> Does any of that argument really matter? And frankly, this statement.
My definition of a complete AGI is: an AI that can read JIRA tickets, talk with non-programmers and do all my job and get me and all/most software engineers fired and proves sustainable.
But in general, it's an AI that can do any remote-work just as good as humans.
agreed. There is no way to tell if someone is aware or not we rely on brain activity to say someone is alive or not there is no way to tell someone or something is conscious currently.
Does general intelligence require awareness though? I think you are talking about consciousness, not intelligence. Though to be frank consciousness and intelligence are not well defined terms either.
This seems like an unsupported assertion. LLMs already exhibit good functional understanding of and ability in many domains, and so it's not at all clear that they require any more "awareness" (are you referring to consciousness?) than they already have.
> the spark of awareness required to be intelligent.
Again, this seems like an assumption - that there's some quality of awareness (again, consciousness?), that LLMs don't have, that they need in order to be "intelligent". But why do you believe that?
> We’ve all had sudden insights without deliberation or thought.
Highly doubtful. What you mean is, "without conscious thought". Your conscious awareness of your cognition is not the entirety of your cognition. It's worth reading a bit of Dennett's work about this - he's good at pointing out the biases we tend to have about these kinds of issues.
> We might very well be able to fake it to an extent that it fools us
This leads to claiming that there are unobservable, undetectable differences. Which there may be - we might succeed in building LLMs that meet whatever the prevailing arbitrary definition of intelligence is, but that don't possess consciousness. At that point, though, how meaningful is it to say they're not intelligent because they're not conscious? They would be functionally intelligent. Arguably, they already are, in many significant ways.
> Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task. You will be able to. How did you do this without thought? We’ve all had sudden insights without deliberation or thought. Where did these come from? By what process did you arrive at them? Most of the things we do or think are not deliberative and definitely not structured with language.
Not to pile on, but isn't this actually a distinct example of _lack_ of awareness? As in, our brains have sparks of creativity without understanding the inception of those sparks?
Perhaps I'm conflating some definition of "aware" with another definition of "awareness"?
This isn’t meant to be a mystical statement that it’s magic that makes humans intelligent or some exotic process impossible to compute. But that the nature of our mind is not observable in its entirety to us sufficient that the current learned reinforcement techniques can’t achieve it.
Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task. You will be able to. How did you do this without thought? We’ve all had sudden insights without deliberation or thought. Where did these come from? By what process did you arrive at them? Most of the things we do or think are not deliberative and definitely not structured with language. This process is unobservable and not measurable, and the only way we have to do so is through imperfect verbalizations that hint out some vague outline of a subconscious mind. But without being able to train a model on that subconscious process, one that can’t be expressed in language with any meaningful sufficiency, how will language models demonstrate it? Their very nature of autoregressive inference prohibits such a process from emerging at any scale. We might very well be able to fake it to an extent that it fools us, but awareness isn’t there - and I’d assert that awareness is all you need.