How Prompt caching works?

API
,
6

Hiya there

Based on @sps response, my understanding is as follows:

When we have a combined input of (System Prompt + user input A), and the total token count reaches or exceeds 1024 tokens, the server-side will begin to cache any components that remain consistent throughout the conversation.

This assumes that user A maintains a consistent dialogue, meaning that the system prompt is sent with every new input, as illustrated below:

Example of Inputs:

  1. SEND: System Prompt + userA_input(1)
    RESPONSE: gptResponse(1)

    message_history = System Prompt + userA_input(1) + gptResponse(1)

  2. SEND: System Prompt + userA_input(1) + gptResponse(1) + userA_input(2)
    RESPONSE: gptResponse(2)

    message_history = System Prompt + userA_input(1) + gptResponse(1) + userA_input(2) + gptResponse(2)

As the conversation history grows, the inputs become increasingly lengthy. It is reasonable to conclude that the server will analyse the repeated elements within this input and begin caching them. Therefore, as the system prompt and prior user inputs are repeated in subsequent messages, the server-side caching is likely to minimize processing costs for users.