The total number of tokens counted at playground is 209, and the bill shows 200, which is playground link https://platform.openai.com/playground/p/59GDMkeYd2zoiVauA2NaCvqQ?model=text-davinci-003 [账单] I tested it over and over again, and both the Tokenizer and playground calculated differently fro…

Yes, the code model seems to be correct, but the calculation of the text-davinci-003 model is wrong. I have been studying this problem for two days

No the calculation isn’t wrong. The codex series models use a different tokenizer than the text series.

Sorry I didn’t understand you, have you used the text-davinci-003 model? Is it consistent with the bill? I really don’t know how to accurately calculate the number of tokens used by text-davinci-003

It is working as expected in my case. Billing: [image] Here’s the tokenizer: [image] Here’s the playground: [image]

Thank you very much for your patient answer. I tested it again and found that the calculation of tokens is still inaccurate. It can be seen that the tokens of the question are accurate, and 14 tokens are consumed [例子1] But the answer is wrong, a total of 88 tokens, 88-14=74, the bill shows 14+60…

Exactly i experienced the same problem!

I conducted a second test, this time I set the maximum token to 1000, the tokens calculated by the question were correct, but the answer was wrong, and the number gap was very large [lizi1] [lizi2] [zhangdan]

I reverse engineered the actual javascript code and data files on the Tokenizer site and noticed a big difference as well The tokenizer page uses a different encoder.json and vocab.bpe file. The number of tokens are the same, but the order and the values are different when it comes to unicode char…

@raymonddavey I found a method, which can make the calculation error not exceed 10 tokens, I only tested Chinese and English [1] curl --location --request POST 'http://api.ppgjx.com/app/func/chatgpt/getTokens' \ --header 'Content-Type: application/x-www-form-urlencoded' \ --data-urlencode 'conte…

Yeah, I did a port of the tokenizer into Swift a while back, and noticed that there is no difference between Text and Code in OpenAI’s Tokenizer page - they both use identical encodings.

Tokenizer and playground calculated a mismatch between the number of tokens and the bill for text-Davinc-003

API

Foxalabs June 29, 2023, 2:48pm 22

this may be of potential interest?

Might not be it exactly, but it’s in the same ballpark.

Topic		Replies	Views
Do you get billed extra when echo=true API	6	2490	January 31, 2023
Official tokenizer has huge count difference from OpenAI tokenizer API	12	5746	October 1, 2023
What is the OpenAI algorithm to calculate tokens? API	34	34167	August 1, 2023
The GPT-4-1106-preview model keeps generating "\\n\\n\\n\\n\\n\\n\\n\\n" for an hour when using functions API chatgpt , api	9	2791	December 31, 2023
Explosion in the number of tokens / words generated API gpt-4 , api	13	5702	August 9, 2023

Tokenizer and playground calculated a mismatch between the number of tokens and the bill for text-Davinc-003

Related topics