Prompt caching not working

API
1

Hello, i have a startup witch is basically a chatgpt wrapper and a very clever system-prompt

The problem is that for each user-request i use the same system-prompt which is 1500 input tokens and more 500 to 1000 tokens that are inputs from my user that changes(therefore cannot be cached)

There is any way i can cache my system-prompt so my startup can be more profitable? the profit margins are very bad today since chatgpt costs are 50% of my revenue and i really need to use gpt4o-latest since its the only one that gives me great results

2 Likes
2

for some reason nothing is being cached

image
image390×725 9.39 KB

3

Hi @jrabelo and welcome to the community!

Are you inserting any dynamic content in the beginning or middle of your system prompt? For example, dates/times, usernames, etc? Basically you need to have at least the first 1024 tokens to be constant for caching to take affect.

Also what is the frequency of API calls? If they are very infrequent, depending on the general API load, they may not be cached.

2 Likes
4

Welcome to the dev forum @jrabelo

Just to confirm, are you using the chatgpt-4o-latest model?

1 Like
5

hello, yes, im using “chatgpt-4o-latest” model

1 Like
6

Thanks for confirming. Prompt caching is not available for this model, at the moment.

Here’s directly from the guide:

Prompt Caching is enabled for the following models:

  • gpt-4o (excludes gpt-4o-2024-05-13 and chatgpt-4o-latest)
  • gpt-4o-mini
  • o1-preview
  • o1-mini
1 Like
7

to be more precise, my system-prompt is exactly 998 chars and NEVER change

image
image448×574 22 KB

8

thanks for your reply, whats the most recent model from gpt4-o i can use with prompt caching?

9

gpt-4o currently points the latest 4o model gpt-4o-2024-08-06 which supports prompt caching.

2 Likes
10

thanks you guys so much for the amazingly quick responses

3 Likes