How does Prompt Caching work?

I honestly don’t understand ALL of the linked thread on prompt caching either :smiley: but I provided it to Claude with your questions…

Scenario 1: Fewer cached tokens than total

  • Caching starts at 1024 tokens and increases in 128-token blocks
  • Maximum cached tokens will be the largest multiple of 128 that fits your total
  • Example: With 5672 total tokens, you’ll see 5432 cached (42 blocks of 128 + 1024)

Scenario 2: Large cache drop with small changes

  • KV cache requires exact prefix matches
  • Even small changes early in the sequence break prefix matching
  • System must find next valid cache point after the change, potentially invalidating large sections

Scenario 3: Token sequence priority

  • Caching prioritizes prefix of input (system prompts → tools → user messages)
  • Put static content first (system prompts, tools) and variable content last (user messages)
  • Cache typically lasts 5-10 minutes (up to 1 hour in off-peak times)