Vector Storage RAG - AI Agent Not Retrieving Data Beyond Midpoint of Uploaded CSV

API
,
1

Hello everyone,

I’m encountering an issue reported by a customer and was able to reproduce it myself.

The scenario is as follows:

  • A CSV file (~1MB) with ~500 rows and 2 simple columns (SKU, Inventory) is uploaded.

  • When querying the AI agent to check stock for specific SKUs, it successfully retrieves results from the beginning of the file.

  • However, from approximately the middle of the dataset onwards, the AI consistently fails to return results.

To validate this further, I tested with a different dataset (a list of shops in Mall of the Emirates). I queried the AI sequentially:

  • The first ~1–1.5 pages of entries are retrieved correctly.

  • After that point, the AI stops returning results for the remaining entries.

This behavior appears very consistent and predictable, which makes me suspect a limitation or misconfiguration rather than random failure.

Has anyone encountered something similar or can provide insight into what might be causing this? At the moment, this significantly limits the usefulness of the feature.

Any guidance would be greatly appreciated.

Thanks in advance – God bless

2

Your post does not describe which API or product or software that you are using. Indeed, you could even be discussing a non-OpenAI solution when you only provide us “querying the AI agent” and “vector storage RAG”.

The chief concern is that CSV is not a supported file type for OpenAI vector stores. File search | OpenAI API

The reason for the failing on CSV as a data format for vector stores is that if it were supplied as a plain text file so it could be ingested, a CSV typically has a header row with the column labels. The document extraction then splits this text into chunks, and the splits of the file after the first chunk have no header of the data fields describing the purpose.

Embeddings-based semantic search works best where the document, when split, has natural language information about a topic and subject in every chunk, that can stand alone as knowledge, to then return a score for how similar the query is to each chunk being individually evaluated.

A SKU + quantity has no information with useful semantic correlation:
58826682, 3
85388662, 10

There are not learned embedding layers that can match token 588 to a query how many 5838666 do we have on hand?, as there is little distinguishing of meaning. Then, the embeddings has been provided many rows of data in a chunk, essentially making any search (beyond the first chunk with a legend with the word “SKU”) meaningless.

If you need database-like search, you will likely need to use a different technique, or even prepare the database for query by field. This is not a solved AI problem for arbitrary user-provided data - except for providing everything as input to a large-context AI model.

3

I’d recommend using File Inputs on the Responses API, Code Interpreter, or the Shell tool to work with structured formats such as CSV. CSV isn’t currently a supported format for retrieval, so one of those approaches would be the best fit.

4

Hi guys thanks so much for your reply.

I forgot to mention the api using forgive me.

We are using realtime api (1.5 and mini), and gpt 5.4 for chat completions. + OpenAI Python SDK’s Vector Stores API.

And the embedding model of openai. (Which as I see we have no control of) but for sure it is one of the best out there because that’s what our research showed.

CSV not supported okay. So I tried with PDF which is supported.

It also happens with a PDF

Lets put aside the SKU example because I agree - thats the work of a Database tool. Lets make it simpler.

Example: I have a pdf with a list of participating brands at the mall of the Emirates. This is page 1.

image
image818×915 43.5 KB

If I ask simple questions like:

Is Adidas present? model says “yes”

I ask for Chanel? It says “I cant find Chanel”

It’s a one page long PDF. I must be doing something very wrong.

Any help would be greatly appreciated

5

Look how strange - Im going to stick to page 1 only

I asked for “participating outlets at the Mall of the Emirates”
2026-04-11 18:41:13.516|HIGHLIGHT|{‘call’: 115, ‘dir’: ‘ai’} FUNCTION CALL: search_information({“query”: “participating outlets at Mall of the Emirates”, “top_k”: 2})

openAI Gave me
2026-04-11 18:41:15.958|HIGHLIGHT|{‘call’: 115, ‘dir’: ‘ai’, ‘detail’: True} FUNCTION CALL RESULT: Top 2 result(s) for: participating outlets at Mall of the Emirates

- 1. score=0.891

excerpt: Annex A Participating Outlets at Mall of the Emirates 7 For All Mankind Abercrombie & Fitch Accessorize ADIDAS ADIDAS Originals Aeropostale Agatha Aizone AL FARDAN JEWELLERS Al Futtaim Jewellery Al Futtaim Watches Al Liali AL MOTAHAJIBA Aldo Aldo Accessories Alexander Mcqueen ALL SAINTS ALMADANI TAILORING ALZAIN JEWELLERS American Eagle American Girl Andrews Tie ANTONIO AQUA Armani Exchange…

- 2. score=0.828

excerpt: 12 new dining options on Level 2 including Omina Baharat, 800 Degrees Neapolitana Pizzeria, Dean & Deluca, Eat Greek Kouzina, Din Tai Fung, Azkadenya, Texas De Brazil & Common Grounds. The Cheesecake Factory, American restaurant Al Halabi – Lebanese restaurant P.F Chang’s – American Chinese restaurant Miu Shanghai Tea House and Restaurant offers nosh from Japan, China, Thailand and Indonesia. YO!

Then I asked if Cocoon is part
2026-04-11 18:41:44.627|HIGHLIGHT|{‘call’: 115, ‘dir’: ‘ai’} FUNCTION CALL: search_information({“query”:“Is Cocoon store part of Mall of the Emirates participating outlets”,“top_k”:2})

Openai reply
2026-04-11 18:41:50.654|INFO|{‘call’: 115, ‘dir’: ‘ai’, ‘detail’: True} Agent transcription content > item_DTUksOxresS4a5JevAJau text=“I wasn’t able to find Cocoon listed among the participating outlets. It’s possible the store isn’t part of the mall, or it may go by another name. Would you like to ask about a different store or any other details?”

I try again
2026-04-11 18:42:43.195|HIGHLIGHT|{‘call’: 115, ‘dir’: ‘ai’} FUNCTION CALL: search_information({“query”:“Is Chanel store part of Mall of the Emirates participating outlets”,“top_k”:2})

2026-04-11 18:42:48.974|INFO|{‘call’: 115, ‘dir’: ‘ai’, ‘detail’: True} Agent transcription content > item_DTUlpPnKXqHtv06a6rXnm text=“I checked again, but I didn’t find Chanel listed among the participating outlets either. It may not be included in that particular list, or it could be listed under a different section. If there’s anything else you’d like to know, feel free to ask.”

Could it be we cannot use this example? Because maybe it doesn’t work well with single words?

I made a search online - it seems that this type of document is problematic. Its seems to be a chunking issue but I do not know how to control the chunking.. Can I control the chunking or is this handled by openAI Internally?

6

Why not just send a base64 PDF and use the response API? What you are doing may be unnecessarily complex?

Here is specific analysis example of a PDF from DELL:

image
image949×952 32.1 KB

Actually, you can do something similar with a CSV file.

7

The answer is “one megabyte” file.

If you note further what you directly reply to: “We are using realtime api (1.5 and mini)”

The context window of those models is 32k, from which the possible 4k output is subtracted from the input capability.

That makes the typical vector result of 20 chunks x 800 tokens fit in context - if you are willing to discard a huge chunk of chat per vector store search result.

Limiting the results to top-k:2 is simply not good practice, because the embeddings search and ranking is not that good.

If you were to reduce the chunk size of the data when uploading and attaching a PDF or were to use that Mall PDF shown, a chunk that is placed into AI context from the example above might be, “CHOPARD Ecco American Girl …” - but still that chunk of language provides no context of why there is a stream of brand names being spewed at the AI if the “participating outlets” is not in that particular chunk.

The RAG file search tool results are given in the sorted ranked order, not in a reconstructed elided document that still makes sense and from which questions can be answered.

Proprietary hint:

8

Yeah, that makes a difference…