LLM-powered functions that I could rely on this morning and afternoon seem to be giving mixed performance. These functions involve parsing a couple of JSONs and responding in a structured output. Some of the values in the structured response require a little bit of reasoning (finding items in the input JSONs) and they are usually quite good and finding the things, except for the past couple of hours.
Iâm getting the famous generic error that tells you absolutely nothing when calling simple functions.
'OpenAI Run Failed. Error: ', âSorry, something went wrong.â
This is getting so freaking ridiculous. Plus the stupid msearch issue where the assistant tries to use a tool called msearch that doesnât exist when itâs supposed to be using file_search. Iâm about to cancel all my OpenAI accounts. I literally canât get anything done recently becuase of these errors and nothing but crickets from OpenAI. What am I paying for here?
That is exactly how it DOES work. The name of the tool method given to AI hasnât changed since it was called retrieval in v1, with many more methods.
To enhance your understanding, here is a complete reproduction of what OpenAI sends when you enable just file search on assistants and use gpt-4o, with text you control in the curly brackets:
Full assistant context with just file search, before any file search results
system
{instructions}
{additional_instructions_probably}
Image input capabilities: Enabled
# Tools
## myfiles_browser
You have the tool `myfiles_browser` with these functions:
`msearch(queries: list[str])` Issues multiple queries to a search over the file(s) uploaded in the current conversation and displays the results.
please render in this format: `ă{message idx}â {link text}ă`
Tool for browsing the files uploaded by the user.
Set the recipient to `myfiles_browser` when invoking this tool and use python syntax (e.g. msearch(['query'])). "Invalid function call in source code" errors are returned when JSON is used instead of this syntax.
Parts of the documents uploaded by users will be automatically included in the conversation. Only use this tool, when the relevant parts don't contain the necessary information to fulfill the user's request.
Think carefully about how the information you find relates to the user's request. Respond as soon as you find information that clearly answers the request.
You can issue up to five queries to the msearch command at a time. However, you should only issue multiple queries when the user's question needs to be decomposed to find different facts. In other scenarios, prefer providing a single, well-designed query. Avoid single word queries that are extremely broad and will return unrelated results.
Here are some examples of how to use the msearch command:
User: What was the GDP of France and Italy in the 1970s? => msearch(["france gdp 1970", "italy gdp 1970"])
User: What does the report say about the GPT4 performance on MMLU? => msearch(["GPT4 MMLU performance"])
User: How can I integrate customer relationship management system with third-party email marketing tools? => msearch(["customer management system marketing integration"])
User: What are the best practices for data security and privacy for our cloud storage services? => msearch(["cloud storage security and privacy"])
Please provide citations for your answers and render them in the following format: `ă{message idx}:{search idxâ {link text}ă`.
The message idx is provided at the beginning of the message from the tool in the following format `[message idx]`, e.g. [3].
The search index should be extracted from the search results, e.g. # ă13â Parisâ 4f4915f6-2a0b-4eb5-85d1-352e00c125bbărefers to the 13th search result, which comes from a document titled "Paris" with ID 4f4915f6-2a0b-4eb5-85d1-352e00c125bb.
For this example, a valid citation would be ` `.
All 3 parts of the citation are REQUIRED.
system
User has uploaded files. These are available with `msearch` using the tool you have for searching files.
user
{first user message from thread}
To get a full appreciation, you might want to paste that into a text editor with word wrap.
You can see how it is bound to misbehave when you use vector store as an assistant knowledge skill, because of files uploaded by the user.
(Note, the role messages are contained within tokens that canât be reproduced unless the AI is cleverly taught their string encoding, here replaced by linefeeds.)
To answer this topic: while old threads are just impossible to get loaded via UI, this new API request to show you todayâs assistants operation was fulfilled in a timely manner.
1 Like
Thanks for taking the time to explain that the tool exists, never heard about it before and itâs no where in the docs that I could find. That doesnât help me on my issue, but at least I understand how it works now. My problem is Iâm just asking regular questions that would normally be answered by RAG, and works fine in the playground (most of the time) with the same agent where it does not work through the API, but itâs trying to call msearch and canât find it
1 Like
It seems the AI is confused by all the functions you have added.
Functions have to be sent to a âfunctionâ tool recipient for you to get them, whereas the file search uses a âmyfiles_browserâ tool recipient.
tools:
â python
â myfiles_browser
\ ---- msearch()
â functions
\ ---- AddMemoryTool()
\ ---- ChangeFileTool()
\ ---- âŚ
If you are using a poor model (like one with âoâ in the name), a fine-tuning that cannot be trained on examples searching with the file search (as OpenAI doesnât allow you any true creation of your own âtoolâ level in this hierarchy) but might have been trained on functions, or are flooding the AI with other functions and confusing system messages:
- The AI may emit the functions tool name instead of the myfiles_browser tool name, change its mind and try to then generate the msearch within functions, and then you receive it instead of it being processed internally by assistants and the tool recipient handler.
You can correctly describe in the system prompt what information will be returned when myfiles_browser is used, and that will make it more likely to employ the tool correctly and not needlessly.
4 Likes
Thanks. I made some adjustments yesterday and I havenât encountered the issue since, so maybe I accidently fixed it 
fingers crossed. And you guessed correctly, Iâm using the âoâ to save on costs during my dev journey. Anyway, thanks for sharing the knowledge!
Edit: Oh yeah, one thing that may have helped is I just remembered last night I got rid of the tools that were just using commands anyway and the agents just uses the command tool to handle the file operations and navigation and all of that fun stuff. That may have lessened the confusion like you said.
I will add that this particular message recently placed by OpenAI that you can see in my context transcription is a complete distraction from the proper use of myfiles_browser, and absolutely could be the reason why assistants doesnât function right.
Read closely what it says
using the tool you have for searching files â
- uploaded files are available with
msearch - uploaded by the user
If you have other functions with various file abilities, like many in the screenshot, the AI SHOULD be befuddled by this poor prompt engineering injecting ambiguous anaphoric reference.
(Real RAG is not a search done by a chatbot.)