How do you get token count when streaming

Where do you get the token count from when you use the stream option for completions?

The system streams the result as blocks of text - but none of the responses have token counts attached to them

The final [DONE] doesn’t have the token count either

1 Like

You might consider writjng a method which estimates the token count by counting the words received and using the documented OpenAI word-token estimate.

.

I have a tokenizer but it is not accurate enough with foreign languages where some words are represented by several tokens

Ie a different ratio to English

I have found the JavaScript function in the playground and tokenizer pages.

I guess I will have to reverse engineer the dictionary and regex expressions in the JavaScript given that openai don’t seem to pass the token count when they stream results

The publically available gpt 2 function uses a different dictionary

1 Like