Humans are confidently wrong about all kinds of things all the time. We don't call it hallucination other than for very specific, limited subsets. We call each other names over it, call it mistakes, stupidity, or lies. In fact, we structure large parts of society around it (the presence of multiple contradictory world religions means the majority of the worlds population goes through life being confidently wrong). It's just being confidently wrong about some subjects is more socially acceptable than others.
ChatGPT clearly needs reinforcement to know its limits the way humans are constantly reinforced from childhood that making confident claims about things we don't actually know well is often going to have negative consequences, but until we've seen the result of doing that, I don't think we have any foundation to say anything about whether ChatGPT's current willingness to make up answers means it's level of understanding is in any way fundamentally different from that of humans or not.
I think that the ChatGPT model (at least chatgpt-3.5-turbo) has gotten impressively better at this.
- In my most recent tests, it will tell you when the data you've provided doesn't match the task (instead of inventing an answer out of thin air).
- It will also add (unprompted) comments/notes before or after the result to disclaim a plausible reason why certain choices have been made or the answer isn't complete.
You have to take into account that not everyone wants the model to not hallucinate. There is a lot of competing pressure:
- Some people would like the model to say "As an AI model trained by OpenAI, I am not qualified to provide an answer because this data is not part of my training set" or something similar, because they want it to only talk when it's sure of the data. (I personally think this use case - using LLMs as search engines/databases of truth - is deeply flawed and not what LLMs are for ; but for a large enough GPT-n, it would work perfectly fine. There is a model size where the model would indeed contain the entire uncompressed Internet, after all)
- Some people want the model to never give such a denial, and always provide an answer in the require format, even if it requires the model to "bullshit" or improvise a bit. An example is, if that as a business user I provide the model with a blog article and ask for metadata in a JSON structure, I want the model to NEVER return "As an AI model..." and ALWAYS return valid JSON, even if the metadata is somewhat shaky or faulty. Most apps are more tolerant to BS than they are to empty/invalid responses. That's the whole reason behind all of those "don't reply out of character/say you are an AI" prompts you see floating around (which, in my experience, are completely useless and do not affect the result one bit)
So the reinforcement is constantly going in those opposite directions.
I agree it's better, but it's still awful at it. But I think it's one of the low hanging fruits to improve, though, just by reinforcing the limits of what it knows.
With respect to the competing draws here, I'm not sure they necessarily compete that much. E.g. being able to ask it to speculate but explain justifications or being able to provide a best guess or being able to ask it to just be as creative as it wants, would be sufficient. Alternatively one that knows how to embed indication of what it knows to be true vs. what is speculation or pure fiction. Of course we can "just" also train different models with different levels of / types of reinforced limits. Having both a "anything goes" model and a constrained version that knows what it does know might both be useful for different things.
You seem to propose a "if it quacks like a duck" scenario here (aka the Chinese Room), and applying it to people who are mistaken or acting on incorrect information.
Those people are reasoning, however rightly or wrongly, based on that wrong information. LLMs are not.
We do not have sufficient knowledge of what reasoning entails to be able to tell whether there's any meaningful distinction between human reasoning and LLMs. It's likely there is some difference. It's not clear whether there's any major qualitative difference.
Claiming we can tell that there's a distinction that merits saying people are reasoning and LLMs are not, is "hallucination" to me. It's making a claim there is insufficient evidence to make a reasoned statement about.
EDIT: Ironically, on feeding ChatGPT (w/GPT4) my comment and your reply and asking it to "compose a reply on behalf of 'vidarh'" it produced a reply that was far more willing to accept your claim that there is a fundamental difference (while otherwise giving a reasonable reaffirmation of my argument that reinforcement of the boundaries of its knowledge would reduce its "hallucinations")
It may be true that we don't have a real fundamental understanding of what human reasoning involves but I think we can be pretty certain that it's something different from just stringing words together.
The problem with this statement is that if LLMs are "just stringing words together, then we can't be pretty certain of that, because LLMs can demonstrably do things that it is wildly unintuitive that you'd be able to get from "just stringing words together". Either we can acknowledge that something more is going on, or our understanding of what is required for reasoning is even more woefully lacking.
E.g. ChatGPT can (badly) perform maths or symbolically evaluate code that is guaranteed not to be in its input in a way that would be very hard to explain as "just stringing words together".
Here is an example where I loosely describe a language that is a mix of an INTERCAL-inspired "come from" construct and Ruby, and then give it an example program and asks it what it will output:
> We're going to write a program in a "new" language that is a mix of Ruby and INTERCAL. We're going to take the "come from" statement and use it to allow "hijacking" the return of a function. Furthermore, we're going to do it conditionally. "come from <method> if <condition>" will execute the following block if <method> was executed and <condition> is true. In <condition>, "result" can be used to refer to the result of executing <method>.
>
> Can you deduce what this program will print, and why?
>
> def foo(str)
> str
> end
>
> come from foo if result == "hi" do
> print "foo returned hi"
> end
>
> puts "1:"
> puts foo("some string")
> puts "2:"
> foo("hi")
I explicitly avoided passing strings that'd indicate the expected output. There's an ambiguity introduced by passing a block rather than just letting the statement stand by itself (passing a block might indicate I'd like the statement to return at the end of the block). This was unintended from my side. I intended execution to just continue after the "come from" block, as that is consistent with INTERCAL. I'll get back to that ambiguity.
Here's ChatGPT(GPT4)'s output:
> Sure, let's analyze this program step by step. The new language is a mix of Ruby and INTERCAL, using the "come from" statement to conditionally hijack the return of a function.
>
> Here's the program:
[snipped a verbatim copy of the program I input for brevity]
> 1. The `foo` method is defined, taking a single argument `str` and returning it.
>
> 2. The "come from" statement is introduced with a condition to hijack the return of the `foo` method if the `result` is equal to "hi".
>
> 3. The program outputs "1:".
>
> 4. The program calls `foo("some string")` and outputs its result, which is "some string". The "come from" statement is not executed because the result is not equal to "hi".
>
> 5. The program outputs "2:".
>
> 6. The program calls `foo("hi")`. Since the result is "hi", the "come from" statement is executed, and "foo returned hi" is printed. Note that the return value of the `foo` method is not printed here, as there's no `puts` statement wrapping the call to `foo("hi")`.
>
> So, the output of the program will be:
>
> ```
> 1:
> some string
> 2:
> foo returned hi
> ```
This is what I intended.
This isn't a very complex problem, but it's the kind of problem that requires stepping outside how you're used to think to the point where I'd like to e.g. pose it as an interview question, but I'd worry about using it as I know from experience of conversations with people about INTERCAL's "come from" that a surprising number of people struggle with reasoning about it.
I asked a followup question:
> "come from" alters the control flow, and prevents a method call from returning to its call site. Suggest a couple of implementation and specification issues specific to dealing with the control flow and its implications.
It produced a detailed explanation of how it'd require stack unwinding, pointed out that you'd need to define how it is affected by dynamic dispatch in Ruby (e.g. which method should it actually intercept?), and gave an example of a scenario where "come from" could affect exception handling that I frankly hadn't thought of. If someone gave an answer like it did to this in a technical interview, I'd hire them.
I then followed up with one more:
> Are there other options to unwinding the stack? E.g. when the block passed to "come from" exits, what are our options for where code execution will continue?
It correctly ascertains that returning to the original call site is another option, and then deduced that this would make "come from" behave like an aspect-oriented "around" mechanism. It also produced several more esoteric suggestions and explained why they'd be a bad idea.
Because of looking through this I recognised that the block I'd added to the "come from" syntax made it somewhat ambiguous and might hint at different semantics, I asked this:
> Does the fact I passed a block to the "come from" statement give any hints as to which of these options I might have intended?
ChatGPT replied:
> Yes, the fact that you passed a block to the "come from" statement does provide some hints about your intended behavior. It suggests that you might want the "come from" statement to behave like a method interceptor or an "around" aspect, executing the block of code when the specified condition is met and then returning control back to the original call site. [snipped it pointing out that matches one of the options it had described before and copying it's previous explanation]
I was hopeful it'd pick up on this when specifically asked, but I'm frankly surprised it did. It's a subtlety of how idiomatic Ruby differs from INTERCAL, and something I didn't think about myself when I wrote the example, and so given I told it the "come from" was inspired by INTERCAL, it was right to assume at first that my intent was for it to pass through. But it's also right to recognise that the block given might imply other intent (and indeed either this construct should be named something else, like "around" or "after", or the block ought to go)
If this is just "stringing words together", then I know a lot of developers who are bad at just stringing words together.
> Humans are confidently wrong about all kinds of things all the time
there is a qualitative difference: humans may be wrong about facts because they think they are true, ChatGPT is wrong because it does not know what anything means. You cannot fix that, because it's just the way LLMs work.
For example, if asked about a URL for something, a human may remember it wrongly, but will in general say "I don't know, let me check", while ChatGPT will just spew something.
humans are constantly reinforced from childhood that making confident claims about things we don't actually know well is often going to have negative consequences
You seem to be saying that because it misunderstands certain things, everything else that is evidence of correct understanding also doesn't count as real "understanding". I don't see how this is a coherent position to hold, as we can just as easily apply it to humans who have misunderstood something.
Edit: and I also don't see why it has to be so black-and-white. IMHO there is no problem with saying it understands certain things, and doesn't understand other things. We are talking about general intelligence, not god-like omniscience.
i think you're conflating being self-conscious, and being intelligent. In AGI , A is also very important. A universal oracle stuck in a machine, able to correctly predict anything and answer all questions, but still having no "desire" or "will" or consciousness, would be an AGI ( imho )
maybe I am, but I would agree with the universal oracle being generally intelligent, I simply don't think our current tools are like that.
E.g. consider this exchange with chatgpt
> hey assistant, can you provide me with a link to learn about LLMs?
> Certainly! LLM stands for Master of Laws, which is a postgraduate degree in law. Here is a link to a website that provides information about LLM programs, including what they entail and how to apply:
> https://www.llmstudy.com/what-is-an-llm/what-is-an-llm-.aspx
> I hope you find this helpful! Let me know if you have any further questions.
> that is not a real link, can you provide another?
ChatGPT makes up a URL, twice. Does it understand URLs? Does it understand that the URL is wrong? I don't think it does, and I don't think it requires consciousness to do this interaction in a sane way.
Still this does not mean it's useless, it's super useful!
It just seems to understand. This is useful, and deeply impressive, but it's not the same thing.