how to get totaltokens from a stream of CompletionCreateRequests
To see how many tokens are used by an API call, check the usage field in the API response (e.g., response['usage']['total_tokens'] ).
response[‘usage’] is null
how to get total_tokens prompt_tokens completion_tokens ?
Each response in the stream is one token. I have found that you also need to add one extra token as overhead.
There is no easy way to get the token count for the prompt part. You will have to use a tokenizer library. There are also overhead tokens for the start and end of each user, system, and assistant record.
Streams don’t pass the token count.
HI Raymond, are you sure about the each chunk is one token? Even if it is true, that only provides the complete_token, not the total token.
Just want to confirm it. I am seeking a way of finding the cost. The document mentions that the token will be sent via SSE, but from python, I do not the endpoint
I just double check with my code with stream and non-stream.
I can confirm that it is true that each chunk is one token, but quite close. The answer from stream and non-stream is not the same, but close, and their token cost (260, and 293) are also close.
This is only providing the complete_token.
Any one know how to find the promt_token?
The only way I have been able to get tokens for the prompt is by using a tokenizer in my code
There also appears to be an overhead of a few tokens for each role record
In the end, the difference is small when you consider the cost is per thousand tokens and not per token.
When you also consider the cost is a small percent of a cent per thousand tokens, the difference doesn’t make much difference at all.
Agree. At the end of day I used the length of my promotion string divided by 4 to estimate the number of prompt tokens and used your way of counting chunks to estimate the completion ones. Thanks