I orchestrate workloads that fan out N parallel workers on the same model, all sharing a large identical prompt prefix (shared instructions plus a big shared context blob) with only a small per-worker suffix. I set the same prompt_cache_key on all of them, which per the docs and this thread is the intended pattern: same key + same prefix routes to the same cache instance, the first request computes the prefix, the rest read it at the cached rate.
The problem is the warm-up. There is currently no way to get the prefix into the cache without running a real generation:
max_output_tokens has a floor of 16 on the Responses API, so a “warm” request always pays for a throwaway generation on top of the input.
More importantly, there is no signal for when the cache entry is actually committed, and no way to verify a write happened other than re-running requests and inspecting usage.cached_tokens after the fact. Threads like this one show people with byte-identical prefixes and correct keys getting cached_tokens: 0 with no way to tell whether the write failed, the routing missed, or the entry expired. If I fan out immediately after warming, the workers race and most of them miss, and each miss is a full-price 100k+ token prefill.
The ask, which I think is small given how the cache already works:
A prefill-only mode on the Responses API: either allow max_output_tokens: 0 or add a flag like warm_cache: true. It runs the prefill for the given input under the given prompt_cache_key, writes the cache entry, samples nothing, and returns only usage. Billed as normal input.
Make its completion mean something: the response returns once the cache entry is visible to subsequent requests on that key, so an orchestrator can warm once and then fan out deterministically instead of guessing.
Since the routing guidance is roughly 15 requests/minute per key before spillover, fan-out workloads are exactly the case where you want one deterministic warm followed by a burst of readers, rather than a priming loop that itself eats into the per-key budget.
For comparison, Anthropic supports exactly this: a max_tokens: 0 request runs prefill only, writes the cache at the breakpoint, and returns immediately with just usage. Gemini goes further with explicit cache resources. And with GPT-5.6 introducing explicit cache breakpoints and billed cache writes, a prefill-only request with a commit signal seems like the natural completion of that design.
It sounds like everything you wish for is in your ability.
Make a first real call that can also do work, with an explicit store on a message where the commonality between calls will end.
Make remaining calls with:
mode:explicit but no marked message, so that no write to cache can occur.
Make calls at a rate of maximum queue depth 5-10 so that the servicing server with cache will not saturate and roll over
Ensure another call is placed under the 30 minute minimum expiry again with explicit message store. Also force this message explicit parameter on another if you have a threshold of sequential calls that have no cache hit.
{
"model": "gpt-5.6-terra",
"prompt_cache_key": "jobjobjob", # always
"prompt_cache_options": {
"mode": "explicit", # always
"ttl": "30m",
},
"input": [
{
"type": "message",
"role": "developer",
"content": [
{
"type": "input_text",
"text": "You are an OpenAI API problem solver",
"prompt_cache_breakpoint": {
"mode": "explicit" # for first call, and wait for completion
}
},
{
"type": "input_text", # varying input
"text": "Find an API call pattern to meet my needs..."
}
]
}
]
}
An explicit store will not work without meeting the 1024+ write token threshold, and you should verify enough input beyond that in practice via tiktoken to receive a success of “cache_write_tokens” in the first usage, or block further calls until a refined input length will write:
My understanding is that the goal is to write to the cache first without making the actual request.
Since this is not yet implemented on the API side, what about sending a minimal request containing only the part that needs to be cached before starting the actual work?
If the context blob is very large, this should already reduce the cost.
That would result in overbilling from actually making a productive call as the “warm up”. Even if we consider the 16 token minimum for output a negligible cost.
100k input discarded with a write = $0.62
100k commonality with a hit = $0.05 that didn’t need to be paid
The magnitude can be much higher if we consider crossing the 272k threshold as a doubling in total token price besides the lengthier cost itself.
Plus, then you are behind one job in your queue before expiry.
I provided a technique aligned with the newest models’ API parameters.
Sending with “in-memory” cache (instead of the now-default 24h) on 5.4 and before would have given any length of previous input a cache match, which I have not verified the quality as of today to offer a guarantee this would give you expected 5+ minutes or a constant stream of calls a cache discount of arbitrary branch points…given that OpenAI already broke a promise silently.
Thanks for the reply! I wasn’t aware of prompt_cache_options: {mode: "explicit"} and prompt_cache_breakpoint, and being able to use them like this. I agree that a dedicated throwaway warming call is worse than using the first real call to warm.
I still have two problems even with this approach
I have to wait for the first call’s full response before starting workers 2..N, because there’s no guarantee the cache is usable the moment it starts streaming. On a reasoning model that can be minutes of waiting, when computing the shared prefix itself would take seconds.
From what I’ve seen, cache_write_tokens only confirms that it was written, not that I can use it immediately, and there are a few threads on the community that talk about this as an issue.
And if I don’t want to wait and fan out earlier (at the first streamed token), I’m just taking a guess on whether the cache will actually be there.
Your suggestion is an improvement over blind priming but it’s still a workaround. I still need the feature request as I described it, which is a prefill-only request under a prompt_cache_key with the guarantee that once it returns, the entry is available and readable for the next request on that key.
Let me know if there’s anything I’m mistaken about in my reply!