Deleting vector store files does not delete them

API
,
1

After adding a file to the vector store via

https://platform.openai.com/docs/api-reference/vector-stores-files/createFile

it sometimes ends up in the status “failed”.

When I then try to delete it via

https://platform.openai.com/docs/api-reference/vector-stores-files/deleteFile

I get a response saying

{
  id: "file-...",
  object: "vector_store.file.deleted",
  deleted: true
}

But when I then list the vector store files again via

https://platform.openai.com/docs/api-reference/vector-stores-files/listFiles

I still see it (still with status “failed”) - also a day after deleting it!

Is there a general issue with deleting failed vector store files?

3 Likes
2

I now deleted the vector store and created a new one.

After re-adding all my files I now have two vector store entries that are in status “in-progress” - for many hours now, while all 500 others are long completed, so highly unlikely that they will finish…

After deleting those they now have status “canceled”, but are still there and I cannot get rid of them and also cannot re-add the corresponding files to the store.

It seems the only way I can get the API to really retry indexing those failed entries is by either deleting the whole vector store or by also deleting the corresponding file, re-uploading it and then re-adding it to the vector store (with its new file id).

The latter of course leaves a whole bunch of dead vector store entries lying around…

3

I’m dealing with the same issue. I can delete it in the regular storage and when it says deleted = true, it’s actually gone. But when I delete from the vector store ‘deleted = true’ is not in fact true. The same file_id’s show up every time.

4

So I just deleted all from the vector store and then deleted all from the regular file storage. The vector store files didn’t disappear until after deleting the file from the file storage.

The documentation for Delete Vector Store File says “This will remove the file from the vector store but the file itself will not be deleted. To delete the file, use the [delete file] endpoint.”

However, it appears that this does NOT “remove the file from the vector store” unless you also delete it using the delete file endpoint. Perhaps further testing is required but this doesn’t appear to be consistent with the documentation.

edit:
I’ve also limited my uploads to 100 files at a time. Then I attach them to the vector store by batching those 100 files. Check the statuses within the vstore until all are ‘completed’. Then repeat. I’ve only had 1 hiccup so far doing it this way but the success rate is significantly higher.

My files are pretty small so I’m well within the limits here:
https://platform.openai.com/docs/api-reference/files/create

And I’m not seeing any limits on creating Vector Store File Batches:
platform.openai.com/docs/api-reference/vector-stores-file-batches/createBatch

5

The vector store items that were in the “canceled” state now finally disappeared. So it takes a long time, but at some point they will go away also without removing the corresponding file item.

I did not see the same thing with vector store items that were in the “failed” state – but maybe waiting two days before trying the stuff described above to get rid of them was not enough and they would at some point have gone away by themselves also…

The main issue seems to me that when the embedding fails I cannot easily retry just the embedding (create vector item) part. I either have to wait a long time or delete the uploaded file too and start all over.

6

Having the same issue here. I thought there was something wrong with my code but I’m seeing the same thing as you - Vector Store Filed that had a status of “failed” stick around for a long time after a successful call to the DELETE endpoint for that file ID.

Would be helpful to get official guidance on this. I seem to have noticed the vector store will show “processed” as its status, even if it is storing Vector Store Files with the status “failed”

7

I’m facing the same issue even when I try to delete the file manually through the platform it just spins as if it’s deleting but even after visiting it after an hour it still exists. If I refresh the page and go back it’s still there as well but it’s no longer attempting to delete the file.

8

I had little success via the UI. The only way for my was to use the API and not only delete the vector store entry but also the corresponding file and then re-upload the file (also via API) and re-add to the vector store…