Where do you get the token count from when you use the stream option for completions?
The system streams the result as blocks of text - but none of the responses have token counts attached to them
The final [DONE] doesn’t have the token count either
You might consider writjng a method which estimates the token count by counting the words received and using the documented OpenAI word-token estimate.
I have a tokenizer but it is not accurate enough with foreign languages where some words are represented by several tokens
Ie a different ratio to English
The publically available gpt 2 function uses a different dictionary