according to OpenAI API , For languages other than English, the number of tokens tends to be overestimated. In fact, I don’t think GPT-3 will tokenize in this way, can I just consider this pricing?

[image] rluvu2: In fact, I don’t think GPT-3 will tokenize in this way, can I just consider this pricing? If I were you, I would experiment with different completions and then review your account to see how many tokens were used according to your account profile stats. Then, you can take …

Thank you. I am still in the process of comparing with models elsewhere. The billing format is important, but the output result is also important, so I checked it first before training quite a lot of data. Thank you for the good reply.

in our tests, Although it charges a bit more, it produced similar results to the token calculator. Doesn’t the actual GPT-3 have a separate tokenizer in languages other than English? 1200 characters and 2350 tokens is too much of a burden.

[image] rluvu2: Doesn’t the actual GPT-3 have a separate tokenizer in languages other than English? No, I believe the current models are tokenized in English only…(At this time?) I don’t think they recommend it for non-english queries. At least in the initial beta rollout… I believe it’s i…

Thank you for your advice. When using the prompt in the playground, I think it has performance worth considering, but I am burdened with the cost of tokenizing, so I plan to use it for free credits. Thanks again for the advice.

Stumbled upon this trying to answer my own question (“what’s the average number of characters per token in languages other than English?”) I’m still looking but I do have data to show that it’s widely variable depending on the language, just compare: Sentence 1: The early bird catches the worm. …

You can also get the token count by calling the completion API method. See: [image] ChatGPT Can't Count Characters? ChatGPT Hi @AiNewbie Just for you, I confirmed that the OpenAI completion API method returns the usage: "usage":{ "prompt_tokens":8, …

How does GPT-3 cost calculation for languages other than English?

API

ruby_coder February 20, 2023, 2:41pm 8

You can also get the token count by calling the completion API method.

See:

Topic		Replies	Views
How do I calculate the pricing for generation of text? API	11	7570	March 6, 2023
Counting tokens for chat API calls (gpt-3.5-turbo) Documentation	5	28422	December 13, 2023
Tokens counting for Hebrew response seems much higher API	5	1463	December 20, 2023
Explosion in the number of tokens / words generated API gpt-4 , api	13	5339	August 9, 2023
Understanding billing of usage API gpt-4 , api	7	2262	February 16, 2024

How does GPT-3 cost calculation for languages other than English?

Related topics