I have 5 files uploaded to a vector store, all of which have the status completed. However when prompted to return all file names, it misses out one. Then after some debugging, I found out that the file has the usage_bytes=0, which I assume means the model skips over that file as it sees an empty file which is no use to it. But I cannot upload the file in a way which causes the usage bytes to not be 0. When I upload the file separately within the openAI playground, it has no problem at all reading it. The file size is relatively much larger than the others, at ~200KB compared to the second biggest at 35KB. The file in question is the attached image as a pdf
I expect the issue you are running into is that the PDF contains an image only. OpenAI’s embedding models do not support image files. Wrapping an image inside a PDF does not return an error when the file is added to the vector store, but there is also nothing from the image that can be processed and retrieved through file_search later.
You can either send these files directly to a vision-capable model or extract the text first and then store that output in a vector store for file_search.
That makes sense and yes, the file is an image that is saved as a pdf. But how come if I upload the same file direct to the playground api does it then work as expected?
Using the playground chat pdf files are processed differently because on models with vision capabilities, the API extracts both text and page images and sends both to the model.
This file_inputs process is described here:
Do you have a method to distinguish pdf file contents before creating vector stores or how do you plan to work with this going forward?
Okay, I shifted my approach at this. I will just upload the files via the python script I’m making, rather than use the vector stores, as I don’t actually need it to remember the data. Thank you for your help
Hey DanTheManIsBack, glad you found VeitB's suggestion helpful!
One small clarification for anyone who runs into this later: usage_bytes=0 on the vector store file doesn't necessarily mean the original PDF itself is empty. usage_bytes refers to the total vector store usage for that file, which may differ from the original file size.