Using RAG to enumerate all entries matching a condition

Hi, I have a Qdrant instance with almost 2000 documents. Each document describe a Product.

I would like search products with query like: “Enumerate all Products made after year 2021”.

But with this type of question I only get few products.

Is RAG the correct way to achieve my intent?

It seems to me that RAG works well searching for ONE specific piece of text.
But when I have to perform “business intellicenge” query (i.e. “Enumerate all red products. Calculate the average price of all my products”), the result are completely wrong.

I’m using OpenAI embeddings

Thanks a lot guys!

It sounds like your 2000 documents are currently organized in a database? If so, why not try using Natural Language to SQL https://platform.openai.com/examples/default-sql-translate

This, of course, depends upon the type of questions you intend to ask and the responses you want. But

could definitely be handled as an SQL query.

What RAG and vector searches are great at is finding context and ideas in document texts. This process isn’t all that great for is finding specific numbers and dates.

1 Like

For direct value queries, you’ll want to use a standard query language.
You could describe the schema of your table, and ask the model to generate SQL (or python) code to extract the appropriate data.
Then run that query.
If you then want to summarize, or filter based on sentiment or something, then you can apply the model directly to the text that comes out of this.