Many apps pass the same context over and over again to the model, for instance a codebase, product information, a bunch of function calls, or a long multi-turn conversation history. Starting today, we will give you a 50% discount for input tokens that the model has seen recently, no action required.…

thank you so much for the community post on this!! I’ve just tried using it, unfortunately it didn’t work, so is this something that will be added later today, perhaps along the week or as we go? I used gpt-4o for my tests. Edit: I’ve also noticed the python api hasn’t been updated in 6h, so maybe…

I think this is a great idea! It really makes “Prompt Engineering” more engineering-ish. There are so many nuanced and exacting, yet regularized tasks that this will make accessible to small businesses.

Will the discount stack with a batch job if there are cachable prompts in the batch?

It’s pretty awesome. It’ll be even more awesome when it’s more than 1024 tokens, then multi-turn chat sessions will come with ~50% off input tokens.

I believe there is an “automated prompt engineering” entry coming to the cookbook soon.

It’s working, although the 5 minutes are not guaranteed - it’s more like 5 seconds(!). More often than not, after a few seconds the cache is cold already. Or maybe it’s a CDN issue, and we get fresh machines 50% of the times. [image] o1-mini seems the most predictable at the moment with ~80% hit…

From the docs : Discounting for Prompt Caching is not available on the Batch API

Ahhh! Thanks. That’s exactly what I expected!

interesting, thanks for sharing this! I’ll give it another try tomorrow

Prompt caching (automatic!)

Announcements

stevenic October 1, 2024, 11:45pm 2

By far the most immediately useful feature you launched today. Love that it’s automatic.

Topic		Replies	Views
Prompt Caching for o3-mini? API o3-mini	3	731	February 4, 2025
How can I reduce API costs with repeated prompts? API api , assistants-api	10	2273	May 7, 2025
How Prompt caching works? API assistants-api , prompt-caching	17	11363	February 4, 2025
Realtime API pricing is wrong, will overcharge API realtime	36	5688	January 15, 2025
Prompt caching with multiple agents API	1	1398	October 9, 2024

Prompt caching (automatic!)

Related topics