Rules of Thumb for number of source code characters to tokens

[restlessronin]

Thanks. I also found this older post that was interesting.

How to do a quick estimation of token count of a text?

Hi! Did any of you find a method to quickly estimate the token count of a text? For scripts or small time applications it’s a bit unpleasant to import a library or have it munch that 1.9MB cl100k_base.tiktoken file yourself, just to decide how much stuff you’re going to forward to ChatGPT or whether to use that new 32k model. There is that “one token is about 3/4 word” estimation, but that’s just for english plain text, and you’re lost for other languages and for source code it’s even worse. A…

I finally picked a simple character count for the samples, and left the choice of tokenizer to the library client code.