I've been spending some time trying to get a sense of how it works by exploring where it fails. When it makes a mistake, you can ask questions in a socratic method until it says the true counterpart to its mistake. It doesn't comment on noticing a discrepancy even if you try to get it to reconcile its previous answer with the corrected version that you guided it to. If you ask specifically about the discrepancy it will usually deny the discrepancy entirely or double-down on the mistake. In the cases where it eventually states the truth through this process, asking the original question that you started with will cause it to state the false version again despite obviously contradicting what it said in the immediately previous answer.
ChatGPT is immune to the socratic method. It's like it has a model of the world that was developed by processing its training data but it is unable to improve its conceptual model over the course of a conversation.
These are not the kinds of logical failures that a human would make. It may be the most naturalistic computing system we've ever seen but when pushed to its limits it does not "think" like a human at all.
> If you ask specifically about the discrepancy it will usually deny the discrepancy entirely or double-down on the mistake.
I have had the exact opposite experience. I pasted error messages from code it generated, I corrected its Latin grammar, and I pointed out contradictions in its factual statements in a variety of ways. Every time, it responded with a correction and (the same) apology.
This makes me wonder if we got different paths in an AB test.
I pasted error messages from code it generated. It kept generating the same compiler error eventually.
When I applied the "socratic method" and explained to it the answer based on stack overflow answers. It would at first pretend to understand by transforming the relevant documentation I inserted into it, but once I asked it the original question, it basically ignored all the progress and kept creating the same code with the same compiler errors.
ChatGPT is immune to the socratic method. It's like it has a model of the world that was developed by processing its training data but it is unable to improve its conceptual model over the course of a conversation.
These are not the kinds of logical failures that a human would make. It may be the most naturalistic computing system we've ever seen but when pushed to its limits it does not "think" like a human at all.