# Official tokenizer has huge count difference from OpenAI tokenizer **URL:** \ **Category:** API\ **Created:** [October 1, 2023, 5:30pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270 "2023-10-01T17:30:47Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![sanandataleyk](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/sanandataleyk/32/146743_2.png) [@sanandataleyk](https://community.openai.com/u/sanandataleyk)\ **Post date:** [October 1, 2023, 5:30pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/1 "2023-10-01T17:30:47Z")
The official tokenizer returns completely different number of tokens than API. I noticed that when I get error. Since the difference is huge I write the approximate number. I tried to do analyze for content of this article azpolitika.info/?p=730071 Official tokenizer value: ~12k Due to API: ~19k 7K is huge gap between API and Official tokenizer. It makes your service not transparent. There is no way to evaluate the price before running. ---
**Author:** ![\_j](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/_j/32/766292_2.png) [@\_j](https://community.openai.com/u/_j)\ **Post date:** [October 1, 2023, 5:33pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/2 "2023-10-01T17:33:58Z")
Use this one and select the model: [https://tiktokenizer.vercel.app/](https://tiktokenizer.vercel.app/) The OpenAI site needs to be taken down if they won’t update it for new models… ---
**Author:** ![anon22939549](https://avatars.discourse-cdn.com/v4/letter/a/f14d63/32.png) [@anon22939549](https://community.openai.com/u/anon22939549)\ **Post date:** [October 1, 2023, 5:35pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/3 "2023-10-01T17:35:32Z")
You’re using the wrong tokenizer. The OpenAI GUI tokenizer is for GPT-3 and uses `r50k_base`, the newer models use `cl100k_base`. ---
**Author:** ![sanandataleyk](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/sanandataleyk/32/146743_2.png) [@sanandataleyk](https://community.openai.com/u/sanandataleyk)\ **Post date:** [October 1, 2023, 5:35pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/4 "2023-10-01T17:35:47Z")
thanks for quick reply. this service which you provided shows even less 9k token for same content. ---
**Author:** ![sanandataleyk](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/sanandataleyk/32/146743_2.png) [@sanandataleyk](https://community.openai.com/u/sanandataleyk)\ **Post date:** [October 1, 2023, 5:38pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/5 "2023-10-01T17:38:15Z")
thanks for quick reply. I have counted manually with `cl100k_base` and also returns ~9k which is even less than offical tokenizer. None of the tokenizer returns ~19k. This is approximately 2factor more cost from openai side. And it would be nice to have someone from OpenAI to clarify this. ---
**Author:** ![\_j](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/_j/32/766292_2.png) [@\_j](https://community.openai.com/u/_j)\ **Post date:** [October 1, 2023, 5:44pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/7 "2023-10-01T17:44:45Z")
> [@sanandataleyk](#): > > None of the tokenizer returns ~19k None of the AI models that are public on the API accept **19k tokens** either. Where did you get that figure? If in the usage log, there may be multiple requests that are combined into one report per 5 minutes. The ultimate billed token count can be seen in the API response when you don’t use streaming. The tokenizer site preloads a template for you as if you were sending an input. Clear out all the data until the token count reads 0, then paste a response if you want to measure the text sent back to you. The price shown is for input, which is 75% the price of output on gpt-3.5-turbo. What I just started a GPT-4 conversation with (ironically, about tokens): ![image](https://us1.discourse-cdn.com/openai1/original/3X/0/c/0c63d1e7bd57548eb394ce5d363459955ff66f2e.png) ---
**Author:** ![sanandataleyk](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/sanandataleyk/32/146743_2.png) [@sanandataleyk](https://community.openai.com/u/sanandataleyk)\ **Post date:** [October 1, 2023, 5:47pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/8 "2023-10-01T17:47:54Z")
Exactly because of that reason I caught that. I notice in my logs error about the reason which you mentioned. Then it was interesting to me and I checked official tokenizer and different sources but non of them returned ~19k result. Error message from API `This model's maximum context length is 16385 tokens. However, your messages resulted in 18870 tokens` ---
**Author:** ![\_j](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/_j/32/766292_2.png) [@\_j](https://community.openai.com/u/_j)\ **Post date:** [October 1, 2023, 5:50pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/9 "2023-10-01T17:50:31Z")
You need the rest of the message, where it says _how much you sent_ PLUS the amount that you reserved for an output as `max_tokens`. > openai.error.InvalidRequestError: This model’s maximum context length is 4097 tokens. However, you requested 12625 tokens ( **280 in the messages** , 12345 in the completion). Please reduce the length of the messages or completion. That’s me getting rejected for trying to reserve too much of the context length for a response. And here’s the actual API response from -16k, so I think some software you’re using is re-writing the error: > openai.error.InvalidRequestError: This model’s maximum context length is 16385 tokens. However, you requested 23736 tokens (280 in the messages, 23456 in the completion). Please reduce the length of the messages or completion. ---
**Author:** ![sanandataleyk](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/sanandataleyk/32/146743_2.png) [@sanandataleyk](https://community.openai.com/u/sanandataleyk)\ **Post date:** [October 1, 2023, 5:52pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/10 "2023-10-01T17:52:59Z")
I have big texts that’s why I just reserve 256 tokens. So that’s not the case. There’re ~7k difference. ---
**Author:** ![\_j](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/_j/32/766292_2.png) [@\_j](https://community.openai.com/u/_j)\ **Post date:** [October 1, 2023, 5:55pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/11 "2023-10-01T17:55:13Z")
Are you using software not developed by you with a built-in token counting method that rejects you with a miscount before you even send to the API? The tokenizer site where you can select the gpt-3.5-turbo model is correct. Are you only considering the data? **Also counted:** system prompt message. Past conversation. Function specifications. ---
**Author:** ![sanandataleyk](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/sanandataleyk/32/146743_2.png) [@sanandataleyk](https://community.openai.com/u/sanandataleyk)\ **Post date:** [October 1, 2023, 6:04pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/12 "2023-10-01T18:04:48Z")
No software built by me. I am getting this error from OpenAI. I just using `github.com/tiktoken-go/tokenizer` library to count which returns ~9k which is probably same as the service which you shared. System prompt is just 31 token. I am not using past conversation and creating creating new chat completion each time. ---
**Author:** ![\_j](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/_j/32/766292_2.png) [@\_j](https://community.openai.com/u/_j)\ **Post date:** [October 1, 2023, 6:12pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/13 "2023-10-01T18:12:05Z")
Maybe there is some kind of re-encoding going on. For example, use of unicode that is written to different byte sequences. The API doesn’t count wrong. That same screenshot user message sent to -16k by Python API code: ```auto response = openai.ChatCompletion.create( messages = [{"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": user}], model = model, top_p = 0.0, stream = True, max_tokens = 23456) ``` > openai.error.InvalidRequestError: This model’s maximum context length is 16385 tokens. However, you requested 30387 tokens ( **6931** in the messages, 23456 in the completion). Please reduce the length of the messages or completion. ---
**Author:** ![sanandataleyk](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/sanandataleyk/32/146743_2.png) [@sanandataleyk](https://community.openai.com/u/sanandataleyk)\ **Post date:** [October 1, 2023, 6:18pm UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/14 "2023-10-01T18:18:20Z")
Thanks a lot for your answers. That could be an issue. I will debug and try to find maybe there is some other encoding which I am missing. ---
**Author:** ![curt.kennedy](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/curt.kennedy/32/709249_2.png) [@curt.kennedy](https://community.openai.com/u/curt.kennedy)\ **Post date:** [December 29, 2025, 5:05am UTC](https://community.openai.com/t/official-tokenizer-has-huge-count-difference-from-openai-tokenizer/407270/15 "2025-12-29T05:05:01Z")