I’m using the OpenAI Platform’s Evals to evaluate tool usage, specifically with the file_search tool. When I evaluate the arguments the model passes to the tool, to make sure they are accurate, etc many of the responses return empty. Specifically as “Empty assistant message.”
(i’ve been using generated car data to test)
Now, additional details:
The token usage for the runs that fail are about the same for the ones that succeed,
Success:
Input 5,300t | Output 1,759t | 7,059 Total
Failure:
Input 5,112t | Output 1,984t | 7,096 Total
Also, because I pass the function.name and function.arguments to the model scorer, I know that the ones that fail still make tool calls, similarly to the ones that succeed.
I have used the file_search tool on OpenAI Platform vector stores & single files, the results are the same.
Models this seems to be an issue with:
gpt-5
gpt-5-mini
gpt-5-nano
Models I’ve tested and found no issue:
gpt-4.1
gpt-4o
o3
o4-mini
So TLDR, there seems to be an issue with the gpt-5 models in the OpenAI Platform Evals where when using tools (specifically file_search) leads to many of the requests coming back empty.