When attempting to use multiple Chat Completions with tools, it’s possible to run into the following situation:
u: Add 5lb of apple to inventory and 10lb of banana to inventory.
a: [tool calls: add_inventory ...]
So far so good. The tools are executed and two messages with the role of tool are appended to the list of messages:
tool_call_a: {'result':'failure', 'additional_instructions':'Apples do not exist in inventory, but there are similar items gala apple and fuji apple. Ask the user for clarification.'}
tool_call_b: {'result':'success', 'new_quantity':20}
t: ...
One of the tools succeeded and the other did not. The program wants to try calling the tool that failed again, after getting clarification from the user.
a: I've added 10lb of bananas, bringing the total to 20lb, but I had an error adding the apples. It seems that apples don't exist in inventory. Did you mean a similar item like gala apples or fuji apples?
What’s tricky here is that the Completion no longer has any tool calls set in the Choice object. It only has text content for the message, the same as it does for a fully successful tool call, or a non-tool response. There is nothing distinguishing this response from a fully successful response.
This is a problem because if you are trying to have a conversation with ongoing context, you need to know where you can safely cut off and summarize old messages. If your cutoff orphans a tool response in your messages list, the Chat Completion will throw an exception. And the tricky part - that partially successful tool calls get the same response format as a fully successful call - means you have to use fuzzy logic to try and determine if an assistant’s response is asking for clarification or not, in order to retain that context for the next user message.
The problem would be solved if the Completion had a property like “tool_status” that confirms the state of each tool_call (success, failure, needs_clarification, etc). But we can take it a step further with a general metadata field that can be used to maintain a small amount of stateful data between the user’s message, any tool_call messages, and the response. So as not to overlap with the Assistant API, it can expire with the response.