ChatGPT 3.5 Web version converts this string in Russian “сколько токенов в этом предложении?” to 7 tokens:
but when I use tiktoken I get a completely different result = 13 tokens:
what am I doing wrong? Or the web version uses different tokenizer?
Hi and welcome to the developer forum!
Asking the model for tokenisation is not reliable. You should only use TikToken to do that.
3 Likes