Openai RAG model not giving correct output even data is available in vector db

API
,
1

I am developing a POC on RAG model using OpenAI ChatGPT 4 LLM. I am using my multiple bank statements for last 1 year [removed PI] and created embeddings using langchain framework. But when i ask the question to model like "Get all transacttions for last 3 months from xyz bank then it is not giving proper results.

When i upload the bank data to assistant and use same context and prompt to chatgpt 4 then i am getting correct result. I need to understand where im going wrong.

2

Unfortunately for many cases like this it’s just not that simple.

RAG is best suited to cases where you have a very large amount of information in a consistent or structured text format, and you need it to automatically pull the most relevant bit of text into the context window.

If you’re asking it to check all 3 statements and accurately collate the information from separate documents you probably won’t have a ton of success.

For a use-case like that you might be better off just trying to put the information/documents directly into the model’s context window, or otherwise extracting the important information from the statements first and querying with the extracted info.

2 Likes
3

In addition to this, it is also worth recognizing how the file search under the Assistant works:

How it works

The file_search tool implements several retrieval best practices out of the box to help you extract the right data from your files and augment the model’s responses. The file_search tool:

  • Rewrites user queries to optimize them for search.
  • Breaks down complex user queries into multiple searches it can run in parallel.
  • Runs both keyword and semantic searches across both assistant and thread vector stores.
  • Reranks search results to pick the most relevant ones before generating the final response.

What’s particularly important to highlight in your case is that it inherently difficult to get good search results when you are looking for information from three different months in a single search.

You’d be better advised to break down this query into multiple queries, one for each month. This would then increase the likelihood of the search returning the chunks with the relevant information for each month. Under this approach, you would combine the chunks from the three individual searches and then include them in the model’s context to generate a final response.

2 Likes