The ‘file_search’ tool will only retrieve information when gpt-4o is set as the Assistant model. Switching to any other model results in either a failed run or a response indicating that the file(s) could not be accessed.
How might this be resolved?
From my experience and a re-test just now it is only the non-turbo models such as GPT-3, GPT-3.5, or GPT-4. All other turbo models work correctly with the file_search tool from 3.5 up.
1 Like
Unfortunately, this is not my experience. All the models successfully use the ‘file_search’ tool in a fresh thread before throwing errors at some point thereafter (save for gpt-4o).
Perhaps a reminder - and a reminder to the AI - is necessary.
file_search is the name of the internal vector store search tool only when gpt-4o is selected.
When employing other models, it has the same name as the retired “retrieval” tool had, myfiles_browser.
So you need to not only understand that
- you can’t talk directly to the AI about “files” (for there are no files for it to observe or read, only a search that can return ranked document chunks), but you also
- must instruct the name of the proper search tool that assistant must call upon within its tools section, when you do instruct the AI about how to find new knowledge that it must employ by writing search queries. The wrong tool name will produce an error return the AI may try to disguise.
- assistant still has file_search enabled and you are using beta:v2 headers
The tool also has misinformation for the AI about where the files come from and “automatically included”:
Parts of the documents uploaded by users will be automatically included in the conversation. Only use this tool, when the relevant parts don’t contain the necessary information to fulfill the user’s request.
I also verified that despite specifying "max_num_results": 6 and it appearing in the run steps for assistant, that 19 chunks were returned, the assistants endpoint blowing through the gpt-4-turbo budget I wanted to spend to replicate any issues. Avoid the malfunctioning Assistants endpoint entirely.
2 Likes
Thanks! Listing myfiles_browser as an available tool in the system prompt seems to have fixed the problem.
2 Likes
great breakdown @_j
For my use case I have been implementing the prompting strategy similar to how you outlined hence not seeing any problems.
Also for my use case returning maximum results is prefered regardless of total use.
Based on documentation, file_search tool supports gpt-4* and also gpt-3.5-turbo models, if they do not then it is a bug or something OpenAI should fix. First time heard about “myfiles_browser”, I guess it is not officially documented
It’s been not working right or as expected, by name, since days after the DevDay release thirteen months ago.
And here a dump of the full text of both tools from a few weeks ago when again it is a failure:
The tool text should be under your control as a developer, instead of attempts to keep how it (doesn’t) operate some sort of secret without documentation.