Issues with OpenAI API – Vector Store Listing and File Upload Limit

[buge16888]

Dear OpenAI Support Team,

I am encountering some issues while using the OpenAI API for vector stores and would appreciate your assistance.

1. Vector Store Creation and Listing Issue

I created a vector store using the following method:

client.beta.vector_stores.create(name=vector_store_name)  

However, after creation, when I attempt to list the vector stores using:

client.beta.vector_stores.list()  

The newly created vector store does not appear in the results. That said, I am able to retrieve it directly using:

client.beta.vector_stores.retrieve(vector_store_id=vector_store_id)  

For reference, one of the vector store IDs that exhibits this behavior is vs_67c9a9d620f88191841decc6f4192278. However, this issue occurs with all newly created vector stores.

Additionally, this does not seem to be a temporary indexing delay. Some vector stores, such as vs_67c5d1a3584881918b9f1eb09c386eaa, were created several days ago yet still do not appear in the list. This suggests the issue is persistent rather than a temporary delay.

2. File Upload Limit Issue

I am also facing an issue where I can no longer upload new files to an existing vector store with ID vs_67c5d1a3584881918b9f1eb09c386eaa. The error message returned is:

Error code: 400 - {'error': {'message': 'Vector store size limit reached.', 'type': 'invalid_request_error', 'param': 'file_ids', 'code': None}}  

According to the API documentation, the file storage limit for each project is 100GB. However, my vector store currently only occupies 91MB, which is far below the stated limit—less than a thousandth of the quota. Given this, the file storage limit error seems unexpected.

Additionally:

• The file I attempted to upload is less than 1MB, which is significantly smaller than any reasonable per-file size limit.
• I have successfully uploaded this exact same file to other vector stores without any issues.

Questions:

1. Why are newly created vector stores not appearing in the list?
2. How can I resolve the file upload limit issue?

I appreciate your support and look forward to your guidance.

Best regards

[buge16888]

[_j]

Here’s what I suspect for you not seeing more newly-created vector stores:

• there is a default limit of 20 vector stores returned per call

With the API parameter limit, you can increase this to 100 returned per API call.

If you have more still, you will need to paginate through them, which is not by position number, but by specifying the last ID so that you can continue getting the next. You can switch the order so you get the newest first instead.

https://platform.openai.com/docs/api-reference/vector-stores/list

---

Then, for the size limit, that is hard to decipher. A 1MB file would not exceed the 5 million tokens per file limit, and it is unlikely that you have the stated maximum of 10,000 files. It could be that there is some other cap on the total to bring the message about “vector store size.” This may refer to the total billed vector storage, of vectors plus extracted file data chunks.

[buge16888]

The main point is that it didn’t return a single one, which means I created many vector stores, but there isn’t a single vector store under my account! So this isn’t due to the limit.

[jpaterson]

I am also seeing that first issue. It’s happening in the Open AI website’s storage dashboard as well, I suspect because it’s also using vectorStores.list. I don’t see created vector stores appearing in the storage UI, but if I put the id directly in the URL I can see the individual store. I can also access the vector store directly via the node API vectorStores.retrieve.

Seems somewhat intermittent though, sometimes I can create a vector store and it appears in the list, sometimes it doesn’t.

[zbrunson]

I am digging in with the team on why vector stores aren’t appearing in the list endpoint, we will work on getting this resolved.

For the file upload limit, the 100GB limit is for all files in your account. The error you are seeing indicates that you have hit the 10,000 file limit in the vector store you are working with.

[pawl.09bumpers]

I am having the same issues. Sometimes it creates the new vector store and it shows up in the list, sometimes it does not show up in the list although I am getting an ID back when it is created. It’s not consistent at all. Sometimes it works, other times it doesn’t. No idea why.

[jamesAmckinnon]

I am having the same issue. It works sometimes, sometimes it doesn’t. I also delete the file each time and always have since I started using the API:

for _ in range(2):
    message_file = client.files.create(
         file=open(the_pdf, "rb"), 
         purpose="assistants"
    )

     ...

    client.files.delete(message_file.id)