Is it possible to save a frequent used prompt on the api endpoint ?
for example I have only one prompt in my service, so I need to send every time the same prompt with the user input to the api which is kinda …
Idea would be to save the prompt per session so its available for like 1 hour or till the session expires
From what I have researched it is not possible. The best you can do is to use the best_of parameter=n. This uses the prompt and does n completions. If your prompt has token length |p| and your completions on average have token length of |c|, rather than costing n * |p| * |c|, it costs just |p| + n * |c|.
the “instruction” part is always the same. If it were possible to cache the “instruction” part it would greatly save token usage for me.