Recently, I finally found the time to take a closer look at the file_search tool and the vector stores, as I find this option quite powerful.
My use case is to answer a question by forcing the assistant to use both the file_search and my custom function. The information that my assistant should use to answer my input question has to consist of chunks from the file_search and externally retrieved data.
In doing so, I encountered a few difficulties. file-search v2 is marked as beta, so I am aware of possible problems. Therefore, I would like to share my observations here in the hope that it may help one or the other to fix these problems.
(FYI: I always wrote incoming test message in such a way that a file_search should be triggered. The following observations mainly refer to the assistants API usage)
1. Occurrence of 'msearch'
I noticed that yesterday (when I was still using the openai npm module version 4.57.0) a tool call (requires action) of 'msearch' occurred repeatedly when forcing file_search (tool_choice: { type: 'file_search' } or 'required'), but basically file_search was not used here anyway. Since today (including the update to 4.58.1), this no longer seems to occur.
2. Calling several tool functions in a forced manner
Obviously it is not possible to call both tools (file_search and custom function) in a forced (consistent) manner. Neither via explicit instructions nor via the tool_choice: 'required' field. As described in the docs, the 'required' field only ever calls at least one function, but as mentioned, it is not possible to ALWAYS call EXACTLY function X and Y etc…
3. Correct behavior when thread is empty
If I set tool_choice: { type: 'file_search' }, my custom function is still (correctly) called in an empty thread. But as soon as there are recent messages in the thread, only file_search is called.
4. Missing chunks logs in step data
file_search is, as indicated, correctly called in principle and in the logged runsteps, score thresholds with corresponding found files are also correctly displayed. Unfortunately, the chunks used from the search are still missing here (see Improve file search result relevance with chunk ranking).
5. Duplicate calls of the same functions
In addition to point 4, I noticed that the same custom function occurs twice (but with different call IDs) in the requires action payload, which wasn’t the case before.
6. Irregular timeouts
As soon as I experiment with the above attempts, it happens from time to time that the request runs into a timeout.
In my case, a workaround for the 'forced' use of several tool functions would be to abort the run directly after the forced call of the custom function and to start a new one, with forced file_search and previously retrieved information as additional instructions attached. This costs more tokens and is certainly not in the sense of the matter.
A workaround for the problem mentioned in point 5, I have already implemented a set so that only a tool call with the same name gets the data. That works so far without any loss of quality.
I am referring here, among other things, to the following existing thread: Assistant + function calling + file search
Hope this helps. Thank you for your attention and have a nice day!
Cheers