# How Prompt caching works?

**URL:** <https://community.openai.com/t/how-prompt-caching-works/970979>\
**Category:** API\
**Tags:** assistants-api, prompt-caching\
**Created:** [October 8, 2024, 12:36am UTC](https://community.openai.com/t/how-prompt-caching-works/970979 "2024-10-08T00:36:13Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![sps](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/sps/32/7736_2.png) [@sps](https://community.openai.com/u/sps)\
**Post date:** [October 8, 2024, 1:17am UTC](https://community.openai.com/t/how-prompt-caching-works/970979/2 "2024-10-08T01:17:39Z")

</div>

Hi @shure.alpha

Great set of questions.

> [@shure.alpha](#):
>
> If I want to cache only the system prompt, how is it possible to do this by only providing the system prompt as input?

Here’s the key: caching works by storing exact prefixes of your input. So, if your system message is consistent (let’s say it’s the same instructions or context), you’re in luck. But it won’t cache only the system prompt alone. Caching kicks in when your entire input (system prompt + user input) reaches 1024 tokens or more. So, think of it as caching the start of your prompt – which will likely be the system prompt initially.

To maximize caching benefits, you’ll want to structure your prompt like this: put static content (like the system prompt) first, and dynamic content (like the user’s input) afterward. This makes it more likely for that static system prompt to be cached across multiple requests.

> [@shure.alpha](#):
>
> Specifically, what exactly is being saved during the caching process?

Caching stores the prompt prefix, meaning the first part of your entire prompt—everything leading up to the part that changes (like a user’s input). The system looks for identical prefixes from previous prompts to match and reuse them. So, it’s not just caching the system prompt by itself, but rather the combination of all messages (system, user, etc.) up to a certain token length.

> [@shure.alpha](#):
>
> Additionally, considering that LLMs predict one character at a time recursively, how does the concept of caching apply in this scenario?

LLMs do indeed predict tokens one by one (not characters, but tokens), but prompt caching doesn’t alter that process. When there’s a cache hit (i.e., a prefix match), the cached tokens aren’t reprocessed—they’re reused, which reduces latency. The model still generates the rest of the tokens based on the user input that follows. The caching just cuts down on the time it takes to process the repeated parts of your input.

> [@shure.alpha](#):
>
> What exactly is being stored in the cache during this process?

What’s stored is the prompt tokens—essentially, the tokenized version of your input (system + user + assistant messages). This includes the messages array (system prompts including tools, user inputs, assistant responses), any images included in the prompt. The stored prefixes stick around for about 5-10 minutes, and during off-peak times, maybe up to an hour.

tl;dr:

- Caching saves tokenized prefixes of prompts that are 1024 tokens or longer.

- It’s useful when you have repetitive content (like the same system prompt) across multiple requests.

- It helps reduce processing time, but the actual output tokens are always generated fresh each time.

---

_[View the full topic](https://community.openai.com/t/how-prompt-caching-works/970979)._
