Prompt Caching Not Applied When Schema Changes

API Bugs
2

Welcome to the community @dyeoman2

This a very good question.

From the docs:

What can be cached

  • Structured outputs: The structured output schema serves as a prefix to the system message and can be cached.

This means that changing the structured output schema which itself serves as a prefix to the system message would result in a cache miss because the cached prefix is changed.

The image illustrates a cache system with an "Original prompt" leading to outcomes of "Cache hit" or "Cache miss" represented by different colored shapes. (Captioned by AI)
The image illustrates a cache system with an "Original prompt" leading to outcomes of "Cache hit" or "Cache miss" represented by different colored shapes. (Captioned by AI)1681×878 92.5 KB

Structuring Prompts

Cache hits are only possible for exact prefix matches within a prompt. To realize caching benefits, place static content like instructions and examples at the beginning of your prompt, and put variable content, such as user-specific information, at the end. This also applies to images and tools, which must be identical between requests.

5 Likes