I ran into a problem with the responses API with 5.6. Our system uses function tools extensively. We have a large stable input context, and then we see a series of function tool calls, each appending tool results onto the end of that context until a final cycle where the response text is generated. In our system we require that at least one tool is called before the final generation happens.
The problem we’re running into is that we effectively need tool_choice=“required” on the first several calls while we assemble the evidence for the answer, and then we need tool_choice=“none” on the final call.
But, it appears, that tool_choice is rendered into the input context. And therefore if you change the tool_choice on the last cycle, you are effectively losing the entire cacheable prefix and we have pay write tokens for the entire size of the last cycle. That’s very expensive.
If we just leave tool_choice=“auto” (or not specify it at all), it normally works. But the model occasionally decides that it doesn’t really need to call a tool so it won’t.
Anyone have any clever ideas on how to change parameters like this without losing the entire cache? It’s easier to notice this problem now that OpenAI has made explicit caching effectively mandatory. You can still use implicit caching if you want – but it is never a better choice, just a lazy one.
There’s no good reason to turn off the “implicit” caching by using the top-level explicit write parameter in a growing chat context, unless you expect to never reuse the API call “write”. That marks the final turn part as a breakpoint automatically.
The unexpected symptom, that breaks some use-cases because of the limit of 4 effective write points and 50 total retained (per prompt_cache_key?), is that in “explicit” every input turn that you want to potentially match a previous cache write still has to be marked as a breakpoint, otherwise there is no hit. This is either undocumented or a flaw in the implementation.
It may be that the default with no additional parameters is the equivalent of all turns being marked “implicit” with the effect of four potential writes at various lengths. Marking your own four (or more with the oldest ignored for writes) does not cost 4x.
It sounds like what you need to do is:
not use prompt_cache_options
and/or additionally mark the tool return turn before the modified turn as a breakpoint, even when it is in the past, so you can “branch” a cache match from there.
use unique prompt_cache_key per chat.
As I described before, an oversight (or simply disregard of developers) is that only the Responses API endpoint supports explicit cache breakpoint on tool output return, not Chat Completions and its “tool” role. And functions are blocked anyway unless you use “none” for reasoning effort.