We identified the following issue/bug firstly using the API and were then also able to reproduce it in the Playground.
Files and their Contents:
file1.txt → “Mary is the accountant”
file2.txt → “Joe is the client”
Steps to reproduce:
-
Create a new thread, attach file1.txt and add the message “Who is the accountant?”. A vector store is created, the file is attached and the reply is correct.
-
Open the vector store and detach it from the thread.
-
Create a new message with the file2.txt attachment and prompt it with “who is the client?”. The image also illustrates that there are no attached vector stores.
-
A new vector store is created, the file is attached as expected, but the file contents cannot be accessed by the file_search tool.
From there on any files attached to the new vector store will be ignored by the thread.
- BUT: if I attach file2.txt to the original (i.e. old, first, detached) vector store, the thread, although it only has the new vector store attached can access it.
This concludes that:
- any vector store apart from the first one attached is not accessible by the thread. This causes great headache in cases were you work with expired vector stores but need to add new files to the “replacement” vector stores.
- the original vector store, even if detached, still seems to be accessible by the thread. This causes security concerns as it seems that detaching a vector store does not guarantee that information in that vs is no longer accessible. From our tests we found that only when deleting the vector store, it is no longer accessible by the thread.
We found this similar issue which was never answered although it differs in that it involves assistant vs thread vector stores, but it might be the same underlying issue