The problem with Claude is that it is quite literally impossible to get off the waiting list to use it. To OpenAI’s credit they actually ship the product in an accessible way to developers.
I just checked out poe.com. Seems you can only buy a subscription if you own Apple hardware (first time I've ever heard that).
It's $20 a month and comes with 300 GPT-4 messages and 1000 Claude 1.2 messages.
By comparison, ChatGPT Plus gives gives you up to 6000 GPT-4 messages a month for the same price (admittedly it would be hard to use that many as they are given in 3 hour blocks).
First some context: llm "prompts" are actually the whole conversation + initial context. They learn nothing, hence the whole conversation gets fed into them every time, but the instruction following ones are trained to answer your most recent chat response.
In a nutshell, part of your llm prompt (usually your most recent question?) gets fed as a query for the embedding/vector database. It retrieves the most "similar" entries to your question (which is what an embedding database does), and that information is pasted into the context of the llm. Its kinda like pasting the first entry from a local Google search into the beginning of your question as "background."
Some implementations insert your old conversations (that are too big to fit into the llm's context window) into the database as they are pushed out.
This is what I have seen, anyway. Maybe some other implementations do things better.
> part of your llm prompt (usually your most recent question?) gets fed as a query for the embedding/vector database
How is it embedded? Using a separere embedding model, like Bert or something? Or do you use the LLM itself somehow? Also, how do you create content for the vector database keys themselves? Also just some arbitrary off the shelf embedding? Or do you train it as part of training the LLM?
Yeah its completely seperate. The LLM just gets some extra text in the prompt, that is all. The text you want to insert is "encoded" into the database which is not particularly compute expensive. You can read about one such implementation here: https://github.com/chroma-core/chroma
One thing I don't understand is how feeding the entire conversation back as a prefix for every prompt doesn't waste the entire 4K-token context almost immediately. I'd swear that a given ChatGPT window is stateful, somehow, just for that reason alone... but everything I've read suggests that it's not.
Have you tried something like Memory Transformers https://arxiv.org/abs/2006.11527 where you move the k/v pairs that don't fit in the context window to a vector db? Seems like a more general approach, but I have tested then against each other.
Any database can be used to extend the memory of LLMs. What a database does is store stuff and lets you search/retrieve stuff. Embeddings are differet form of data that are in many (but not all) cases superior to searching through text.
You do not need a fancy cloud hosted service to use an embeddings database like you do not need one to use a regular databse (although you could).
Check https://github.com/kagisearch/vectordb for a simple implementation of a vector search database that uses local, on-premise open source tools and lets you use an embeddings database in 3 lines of code.
Instruction tuned LLaMA 65B/Falcon 40B are good, especially with an embeddings database.
...But OpenAI has all the name recognition and ease of use now, so it might not even matter if others ambiguously surpass OpenAI models.