So I am aware of some 3rd party libs as well but I was hesitant using some of them even if they are openSource. I handle it with this one recommend in OpenAI cookbook or in tiktoken readme , which is gpt-3 encoder . My problem is that how accurate this libraries are and they will be up to date with gpt side changes if there is some. Thats why it will be alot better to use results returned from GPT itself, such as in regular completion api.
When we look at this table from tiktoken, we can see that gpt-3-encoder doesn’t even support cl100k_base encoding.
| Encoding name | OpenAI models |
|---|---|
| cl100k_base | gpt-4, gpt-3.5-turbo, text-embedding-ada-002 |
| p50k_base | Codex models, text-davinci-002, text-davinci-003 |
| r50k_base (or gpt2) | GPT-3 models like davinci |