OpenAI actually thinking about this too. It’s buried in their open source repo and not clear the exact mechanism that ChatGPT knows to make use of it. But we’re already here evidently.
> how to work with a memory module that remembers things about specific entities. It extracts information on entities (using LLMs) and builds up its knowledge about that entity over time (also using LLMs).
There are attempts via langchains [0] depending on how much context is required I could see a summary step where the history to compressed and used to carry forward progress.
An alternative could be a vector store, injecting small snippets of relative text as a step.
On a more serious note, I do agree with you that memory and self-excitation seem like they are the last push thats needed to get to something more akin to "AGI". But I don't think that Rubicon will be crossed with plugins.
>I do agree with you that memory and self-excitation seem like they are the last push thats needed to get to something more akin to "AGI"
"We show that transformer-based large language models are computationally universal when augmented with an external memory. Any deterministic language model that conditions on strings of bounded length is equivalent to a finite automaton, hence computationally limited. However, augmenting such models with a read-write memory creates the possibility of processing arbitrarily large inputs and, potentially, simulating any algorithm."
From "Memory Augmented Large Language Models are Computationally Universal"
I suspect with a 'window' of 32k tokens, OpenAI has already done similar memory tricks.
I suspect that if you filled the context window with "1 1 1 1 1 1 1 1 1 1", and then asked "How many 1's did I just show you?", it probably wouldn't know, simply because whatever tricks they use to have such an apparently large context window don't allow it to 'see' all of it at any given moment.
