Cachedtoken not working recently while using chatCompletion api

API
,
1

CachedToken is not working recently (since Dec.20 2024)

Is there anyone having similar problem or knowing how to solve this problem?

[fyi]

  • Prompt length is around 2000~2400 tokens
  • Most of prompt is not changing
  • Chaning part of prompt is after 1800 token length which means, first 0~1800 tokens are exactly same

image
image1700×512 35.7 KB

2 Likes
2

I am experiencing the same issues.

A pipeline that used to cache most of its prompt tokens between calls is not caching them any longer. We first identified this issue on December 29th.

1 Like