If you're interested in the SQL component of this, we're building a product strictly focused on that at https://www.definite.app/. We let non-technical users ask questions of their SQL database. We do this by:
1. Pulling in your schema information and structuring it in a way LLM's can reason about it
2. Pulling in your prior query history against the database to understand how you actually use your data (e.g. what JOIN's are common, what tables are used most frequently, etc.)
3. Adding context from other tools you may be using (e.g. we can pull in metadata and tests from your dbt project)
We also have a Slackbot you can add to your #urgent-data-requests channel. If you @Definite in a thread, it'll parse out messages that can be converted to SQL tasks and return the answer from your database.
You could certainly build this yourself with (or without) LlamaIndex, but it's still quite a bit of work to set up.
Just to throw it out there, an alternative approach is to have the AI talk to a semantic layer. There are pros and cons to this and in practice you'd probably want an agent that can intelligently decide whether to call your semantic layer or go through some other process that may involve multiple queries, function calls, etc.
IMO a semantic layer makes it so easy to query that natural language querying is often not beneficial, even for business users. I take a hybrid approach of the semantic layer solving >95% of use cases and custom SQL as needed in backend processing.
I like the concept of a semantic layer, but I haven't seen strong adoption of any of them (outside of LookML within Looker). Most of the companies I've talked to that are heavy dbt users don't use their semantic layer, but maybe that will change as the products mature!
We are seeing similar for Louie.ai . Data catalog yes, semantic no. We are also doing a lot of non-SQL like logs DBs (SIEMs) and graph DBs, which adds even more fun to that kind of question. My hypothesis here is that "semantic layer is to semantic web as LLM + vector search over activity is to Google/pagerank": easier to let the machines figure it out. We use knowledge graphs in some of our work, so definitely not disputing the value when people do put in the effort.
Wrt llmindex, the new tools like vector dbs & llmindex have been interesting. Fine-ish for our self-hosted, but most start getting tricky when a multi-tenant SaaS where we lower costs for teams and users by sharing infra yet need sharing boundaries.
I'm working on something that I believe has a similar to approach to what you're describing, particularly with the use of the semantic layer that AI talks to. Would love to connect sometime if you're open to trading notes? See profile for contact info, thx!
Do you have your source and model open? Its hard to use a AI service to give any type of access to prod DB without understanding the underlying code and models
Technically, one can say the same about any devops services out there, for instance. Makes more sense to opensource it.
Still, would you mind sharing the use case and the outcome? I have a couple of clients interested in similar solutions, but so far the potential outcome of this approach doesn't look promising.
It's not really. It was unfortunate timing. This used to be called GPT-Index, and then they changed names before Meta released their LLM. So their use predated it.
I feel sorry for the amazing team behind this great library. Changing names is hard.
OpenAI has started sending out threatening letters to people who use GPT in their project name. If they don't want to be the test case for that (pending) trademark, I don't blame them.
Meta's naming is clearly also a joke. It's a backronym, not an acronym— it's pretty obvious they started with Llama and came up with a cute acronym to justify it.
Large LAnguage Model Meta AI
Not saying they're not entitled to use the name too, but laying the blame on the developers of LlamaIndex when they had no idea that LLaMA was coming isn't fair to them.
We love the feedback, and one main point especially seems to be around making the docs better:
- Improve the organization to better expose both our basic and our advanced capabilities
- Improve the documentation around customization (from LLM's to retrievers etc.)
- Improve the clarity of our examples/notebooks.
I gave this a shot a while back and found plenty of examples but little documentation.
For instance, there is a tree structure for storing the embeddings and the library is able to construct it with a single line. However, I couldn’t find an clear explanation of how that tree is constructed and how to take advantage of it.
Yea, this was my experience too when I tried it out last week for my side project. It's easy to get started, but it's quite complex and disorganized and poorly documented. There are usually several ways to do things (which is by design, since it's meant to give you flexibility of either going with the default or customizing).
The main problem is the documentation is too disorganized, it's hard to figure out what even is the default and what are the configuration options, documentation is spread over a bunch of tutorials, reference pages, and blog posts by the founder. Sometimes the example code doesn't quite work because the library is changing so quickly.
We'll see if the community can figure out the best set of useful abstractions for this domain -- right now LlamaIndex is a mess and makes building things harder instead of easier and it's probably simpler to roll your own solution from scratch. However, the founders seem pretty smart, so hopefully with some time, they'll improve it and make it more usable.
If you’re doing legitimate retrieval rerank in the commercial enterprise setting, then I doubt this is a library that can support you beyond prototyping.
Retrieval involves complex integration (not just data connectors and open API wrappers), and meaningful rerank requires domain/context-specific trained models (that you can deploy performantly and cost effectively). If you’re doing these things, you’re well beyond the capability at platform scale vs what a python library provides
It’s just interesting to see the VC money pouring into these tools. My argument is serious integration/scale doesn’t involve a library like these (honestly prototyping doesn’t really need to either).
Id be more bullish on paradigm (platform/cloud level) shifts vs connectors, wrappers, and utility functions
Ymmv and to be fair I haven’t tried to scale these tools. I have worked on scaled platforms around embedding retrieval and rerank (including LLMs) so it’s just my take.
I would argue the level of abstraction it provides lowers the barrier to entry for most average programmers, myself included. LllamaIndex was my entrance to programatically utilizing LLMs. I have since moved to LangChain, with some documents loaded via LlamaIndex, but it has been a blast.
Yes but this is just ETL - LlamaIndex and LangChain are re-inventing it - why use them when you have robust technology already?
1. You ETL your documents into a vector database - you run this pipeline everyday to keep it up to date. You can run scalable, robust pipelines on Spark for this.
2. You have a streaming inference pipeline that has components that make API calls (agents) and between them transform data. This is Spark streaming.
Prophecy is working with large enterprises to implement generative AI use cases, but they don’t talk so much on HN. Here’s our talk from Data+AI Summit:
Build a Generative AI App on Enterprise Data in 13 Minutes
Cool! Lets say I have thousands of documents that I want questions and answers for. Would your solution work for this? I wouldn’t know which documents to send with the prompts though as I want info on the aggregate (like trends and most mentioned phrases or words).
It's an alternative, does a similar job, depends on/abstracts over langchain for some things. It's easier to use than langchain and you'll probably get moving much faster.
They've aimed to make a framework that starts concise and simple, has useful defaults, then lets you adjust or replace specific parts of the overall "answer questions based on a vectorized document collection" workflow as needed.
This works well overall, but some bits have kept me scratching my head for hours. Partly due to huge holes in the documentation when it comes to specifics ("plenty of examples but little documentation" another commenter wrote and I agree). Partly due to the frenetic release schedule, this project is highly active even by frothy LLM craze standards and interfaces change rapidly.
Overall recommend, LlamaIndex has helped me make good progress on my project
This stuff is like redis. Great for the infra folks
Langchain is like ruby on rails. Great for the AI app devs
There is overlap, where you might start and stay on langchain as a kitchen sink... But as you do full app dev, esp SaaS, the bottom starts falling out for more data infra centric tooling.
This is a layer of abstraction above that. So it handles ingestion of the data (possibly as embeddings), storing it (possibly in a vector database), and then querying of that data for LLM queries. You can swap in different embedding models, different storage databases, different retrieval methods, etc.
Not everything should persist, such as in a classic web architecture, analogous to redis vs postgres
Ex: For some of our use cases for more-text data sources and APIs we are helping folks index for Louie.ai, we do ingest-time embedding into persistent vector DBs for the raw, and are looking at llmindex and others for more ephemeral run-time caching of post-proccessed results. Ex: Pre-embed 1GB of text for fast search, but then only cache ad-hoc genAI task results on those, like classifications. We may still want to persist the embedded task results for other reasons, so not obvious.
If figuring out that kind of thing for use cases like analysts talking to DBs for cyber investigations, supply chain, emergencies, etc, we are actively hiring in backend engineering (remote): https://www.graphistry.com/careers, who build louie.ai
1. Pulling in your schema information and structuring it in a way LLM's can reason about it
2. Pulling in your prior query history against the database to understand how you actually use your data (e.g. what JOIN's are common, what tables are used most frequently, etc.)
3. Adding context from other tools you may be using (e.g. we can pull in metadata and tests from your dbt project)
We also have a Slackbot you can add to your #urgent-data-requests channel. If you @Definite in a thread, it'll parse out messages that can be converted to SQL tasks and return the answer from your database.
You could certainly build this yourself with (or without) LlamaIndex, but it's still quite a bit of work to set up.