Yeah, this is just absurd. They are not delivering the service that I agreed to when I signed up. I was told that I’d be getting prompt cache, and they silently changed or broke that, and now I’m hemorrhaging money.
I’ve spent ten times as much diagnosing and redesigning my project architecture to work around this problem than I’ve saved with the mitigations I’ve put in place.
Still radio silence from their support team as well. Supposedly it’s been escalated multiple times, but beyond some “very reassuring prose” from their chatbot support, absolutely nothing.
Further incremental update: I’ve achieved an additional increase in cache stability at the expense of unrestrained tail growth.
By sending “store”:“true” and using 24-hour retention with foreground synchronous responses, the cache hit rate seems almost perfect, as long as I maintain the body stability.
However, it gets frozen in place. It never rolls me to a new cache, and so the tail starts growing very rapidly, sometimes exceeding 80,000 or even 150,000 un-cached tokens.
For those following along at home: I am still hemorrhaging money. For science! (and the sunk-cost fallacy)
You want to know what has been completely wiped from existence in documentation?
Any mention of 128 token increments of matching.
There now:
In-memory prompt cache retention (ed: not the 24 hour version that has now been switched on retroactively as a default) is available for all models that support Prompt Caching, except for
gpt-5.5,gpt-5.5-pro, and all future models.
So, have they cheapened out in preload by not producing incremental k-v states?
Is there only one “hash match” for an input, the latest, or the longest? It says:
Extended Prompt Caching works by offloading the key/value tensors to GPU-local storage
You send 272k of input - Is OpenAI now avoiding storing 2100 cache window state increments, replacing that with just one possible match?
The documentation has been altered. The methodology has been obfuscated. The models have been changed. The expected pricing is not being delivered in branches, in parallel calls, in partials that should match.
OpenAI knows what they did. The silence, nothing to diagnose, is evidence.
The only thing that would seem to work would be continuously growing on a prior exact input. It’s not really breaking a promise when the prior operation method and its documentation has been eradicated.
For a prior model than gpt-5.5, you have the option of sending “in_memory”, and must.
This pattern seems to allow and anticipate and only be built for ChatGPT’s use, disregarding API developer’s potential reuses.
There is no good method of making a “mid-state” with a prompt cache key API parameter. That can only serve to break a cache - you can’t change it without a 0 hit being made.
The pattern fits: single growing converations unique to a user. Which cannot be rejoined from an earlier state.
Perhaps the reasoning there are multple feature gates being A/B on ChatGPT users, to turn off message editing and branching in ChatGPT in different ways.
Even after the initial thrash and redesign period (marked by the yellow bars from “Holy Sh!t where is my money going?!” to the point where I optimized my application around stable tail growth… I’m still only able to get a 1:5 ratio of cache hits by cost, at best.
Every scheduled cache write still takes one to three full misses to stabilise, and every unscheduled cache write takes two to four to stabilise.
It’s also the only possible explanation - that this was intentional - that would make this behaviour consistent across both OpenAI and OpenRouter access points.
I feel nauseated. I’m not a business. This is my own money, and they still haven’t communicated with me.
I don’t honestly know what my playbook is from this position:
It would take a week to redesign my architecture at the cost of several thousand dollars.
So far, any requests for assistance have just been met with the standard chatbot Circle of Hell.
If I recharge the credits paid, they ban my account.
I don’t have a strong enough social media presence to… get noticed… through side channels.
I welcome any ideas or assistance in finding a strategy to get anything other than silence out of this incident.
Just figured I’d record this one here as part of the cluster.
I have a significant incremental update. It turns out that a few days ago, I collapsed from exhaustion just before the finish line.
A new architectural mode I designed; In which all of the potential volatile tail content that was previously poisoning my cache is delivered as a static tool result during a known cache-busting maintenance window, I finally got it to an operational state and tested it.
This mode guarantees that as much as the other systems keep the body content stable, the tail is also kept append-only and stable, and all of the live-churn, high-volatility, heads-up display content that the agent would normally rely on is instead delivered in heavy blocks during these maintenance windows.
For anything operationally critical, like access tokens for write operations to disk, these still get injected in-band, but their content mass is relatively small, and so it’s tolerable for them to accumulate.
tokens=330,628 cached=0 tail=330,628 ratio=0.00%
tokens=331,078 cached=330,496 tail=582 ratio=99.82%
tokens=331,740 cached=330,496 tail=1,244 ratio=99.63%
tokens=332,381 cached=331,520 tail=861 ratio=99.74%
tokens=341,628 cached=332,032 tail=9,596 ratio=97.19%
tokens=342,859 cached=341,248 tail=1,611 ratio=99.53%
tokens=345,898 cached=342,272 tail=3,626 ratio=98.95%
tokens=346,422 cached=345,344 tail=1,078 ratio=99.69%
tokens=349,013 cached=345,856 tail=3,157 ratio=99.10%
tokens=349,911 cached=348,416 tail=1,495 ratio=99.57%
tokens=350,748 cached=349,440 tail=1,308 ratio=99.63%
tokens=351,614 cached=350,464 tail=1,150 ratio=99.67%
tokens=354,257 cached=351,488 tail=2,769 ratio=99.22%
tokens=355,829 cached=354,048 tail=1,781 ratio=99.50%
tokens=361,141 cached=355,584 tail=5,557 ratio=98.46%
Look at that: Incremental, adaptive, continuous, cache prefix tail growth.
This breaks a bunch of my systems and workflows and violates a lot of expectations in my application, but it’s a promising first look. There may actually be a way for me to continue to use the API in my application without suffering the previous level of financial damages in the future.
I wish this was a concession I could make. Unfortunately, my application has nothing to do with conversational chatting and everything to do with extremely precise and complicated engineering and code work.
At the moment, 5.5 is the only trustworthy model when it comes to making any kind of meaningful contributions to it. Even Opus 4.6 is unable to keep up. Opus 4.7 and Opus 4.8 are fundamentally unsuitable for this kind of work as well.
GLM 5.2 can handle low-stakes tasks, but isn’t quite there in terms of keeping the coherence and alignment with the objectives.
Most of these models are suitable only for the initial reconnaissance fan-out, learning the codebase, mapping the relationships between modules and things like that, but when it comes to actual implementation work, 5.5 is the only one that I can trust to actually see it through.
The unfortunate reality is I’m pushing these models to the absolute limit of their capabilities, and in many cases well beyond the good sense of agentic code work, so anything less capable than 5.5 simply cannot deliver. It’s not a matter of underperforming, it’s a matter of failure to deliver anything usable at all.
If it’s not clear by now, the primary development scope is the self-modification of their own development harness. Though there are many, many other projects that it externally manages as well.
Here’s switches to apparently save you hundreds a day, right off:
Then
I’d look at what the AI really needs to know, how you are loading the initial input so aggressively (where you might also have bigger wins in attention quality with more focus and less distraction as a benefit). We can do stuff with context understanding inconceivable a year ago, at prices that took long conditioning to pay, but Shakespeare’s Hamlet to ask about is 61267 tokens - and you are doing 5x that at a go.
It must look unreasonable from the outside, the way I’m using these models. And I think the reality is, that is true. Though part of the purpose of this architecture and the overall project is to push them past the limits of reasonable usage and…
Honestly I would have to try and describe the entire project and essentially onboard you into everything to try and justify the position and how it looks. All I can really say is that I genuinely believe that this is as little context as the agents require to competently perform their work.
Here’s a bit of an indication of what a standard context window actually looks like after it’s been properly compiled from all data sources that constitute the final (now cache-stable!) payload.
I don’t think more prudent management of the context window contents themselves is a viable direction for me to pursue at the moment, although I’ll have a look into the service tier flex. Batch mode also won’t work for me because I can’t afford to wait 24 hours for a response. This is high-paced serialized work.
