We’re working to improve our caching performance with gpt-5.6-luna. We are using implicit caching, but with explicit cache breakpoints. Given that we’re doing a lot of function tool calling, this is the only pattern that makes sense.
However, I came across this: Prompt caching | OpenAI API
It talks about splitting your cache key if you are getting above 15 requests per minute. That’s a bit scary to us. We serve many different customers, and we’ll use a different cache key for each of them because we will have different content that is relevant to each one. But 15 requests per minute is not very many! Think about the fact that we might use 5 steps in a single response generation because of a sequence of tool calls. Each one of those is a call to response.create and that means for one question, we have already used up 5 requests in that minute. If we’re handling a few users within the same minute, it would be easy to go over 15 RPM. And they say that at that point, we will start seeing cache misses. Egads! Misses are very expensive both in time and money.
The solution proposed is to shard the cache keys. But that’s a problem because it means that cache performance will degrade badly for the case where we’re at lower request rates – because we will effectively be failing to use a perfectly good cached version because it is behind a different key.
I encourage the OpenAI folks to give this a little more thought. Caching is a win-win and we want to be “good citizens” to make it as effective as possible. But this puts us in a bit of a bind.