I think a big difference is that humans have the ability to introspect, and examine their own mental processes to see how they drew a conclusion and what information went into it. We can then choose to update our behavior. Sometimes we "hallucinate" that information, but we do have some level of insight into ourselves. ChatGPT has no ability to examine itself, and in fact has no idea what it is doing. This, I believe, is a big shortcoming of neural nets in general. If we could overcome this hurdle, I think it would be a lot closer to an AGI.
Neuroscientists who have attempted to compare what people think about how we think and how we ACTUALLY think have concluded that self-reports are entertaining to listen to, but not very informative. That is, we have more of an illusion of introspection than a reality.
The level at which we ARE self-aware of our own thinking does not seem to me to be more impressive than the ability of something like ChatGPT to generate training data that can in turn be used to help create much smaller models with similar capabilities, or future models with better ones. In other words LLMs are capable of some form of learning through introspection, and we do not appear to be close to the limits of how much self-improvement they are capable of.
This isn't 100% true.... I think people are using something called the Reflexon model where you ask GPT for an answer, then feed that answer back to GPT and ask what is wrong with it allowing self examination and a significant improvement of its previous answer.
Now, GPT has no method of long term learning by reincorporating these answers back into its system to improve its future behavior, which is a huge limitation.
From the paper, seems like Reflexion is an outer loop that provokes GPT to re-evaluate itself, which says nothing about the self-reflective capabilities of the actual GPT model.
But overall, I find there's a bit of a boundary problem when considering current ML models. Is GPT considered self-reflective if we embed it in a system that mimics such an ability?
The boundary problem you describe seems unimportant to me. Who cares what we classify GPT as when we have successfully built a system that clearly has that capability?
Feedback loops are critical to intelligent thought. We know this, and it should come as no surprise that we'll need to explicitly add them to AI systems.
Interesting - it reminds me of the ‘strange loopiness’ in Douglas Hofstadter’s books - we have a model of ‘I’ that reflects in on itself.
We obviously have a huge evolutionary pressure for having a semi-accurate internal model of ourselves that we can use to project out into the future and avoid possible harm - I guess there is no similar pressure / incentive for LLMs.
