Hello everyone,
I read a forum that the input token will have a square growth due to new messages will append with the old message list and send the whole against the AI again. Therefore, I would like to know the exact cost calculation in OpenAI API.
I have make an example for calculation. The example and result are shown below:
When you preserve history for subsequent API calls, token consumption will naturally escalate. However, it’s important to note that each previously generated output tokens will transform into an input tokens once it’s included as part of the history. Therefore, only the responses from the API will be counted as output tokens.
Seems like your calculations are correct
Hi Eugen, thanks for your quick reply.
Actually we are trying to create a teaching application which pretends AI as a patient while user as a nurse.
Firstly, prompts are given to AI that his legs has pain. User will discuss with AI on what he is suffering.
Therefore i guess the previous generated output needed to be included as part of the history so that AI will be biased based on previouly discussion.
Am i described it correctly?
Or, put i ask in this way: What is the difference if we did not include the history at input tokens comparing included all history at all input.
Hi!
Looks correct!
If I’m understanding you correctly, if you didn’t include the history at all, the model probably wouldn’t know what you were talking about and possibly start making things up
If you don’t want to include the whole chat history, one method is to maintain a rolling summary of the chat, and include that (maybe as part of the first system message).
So yeah, overall it’s all about managing the context. the easiest is obviously to just include the last messages, but the underlying goal is to ensure that the model has enough information to process the next request.
