I know you have libraries helping out in Python and such, but I’m using an “esoteric” programming language to interact with the API, and I need to know how I can manually calculate how many tokens a prompt will result in. I’ve tried length of string and divide by four, and it doesn’t work. I’ve tried byte count (to accommodate for UTF8) and it doesn’t work.
Does anyone here know the algorithm to calculate prompt length?
My reasons for asking is that I sometimes have large prompts, and I need to figure out max_tokens as I am invoking the prompt HTTP endpoint, which of course cannot exceed the tokens for whatever model I am using?
Thx @ruby_coder - It helps me understand tokens, but it doesn’t provide the algorithm. I need to know exactly how many tokens a specified string generates in the model, and I am not using NodeJS or Python.
Following up, I wrote this Ruby method (based on the reference above) which you might be able to convert to your favorite programming language.
Method to estimate tokens per OpenAI docs with method options
# method can be "average", "words", "chars", "max", "min", defaults to "max"
# "average" is the average of words and chars
# "words" is the word count divided by 0.75
# "chars" is the char count divided by 4
# "max" is the max of word and char\
# "min" is the min of word and char
word_count = text.split(" ").count
char_count = text.length
tokens_count_word_est = word_count.to_f / 0.75
tokens_count_char_est = char_count.to_f / 4.0
output = 0
if method == "average"
output = (tokens_count_word_est + tokens_count_char_est) / 2
elsif method == "words"
output = tokens_count_word_est
elsif method == "chars"
output = tokens_count_char_est
elsif method == 'max'
output = [tokens_count_word_est,tokens_count_char_est].max
elsif method == 'min'
output = [tokens_count_word_est,tokens_count_char_est].min
# return invalid method message
return "Invalid method. Use 'average', 'words', 'chars', 'max', or 'min'."
text="Curie is extremely powerful, yet very fast. While Davinci is stronger when it comes to analyzing complicated text, Curie is quite capable for many nuanced tasks like sentiment classification and summarization. Curie is also quite good at answe`ring questions and performing Q&A and as a general service chatbot."
We need a new OpenAI community rule, which I plan to propose to @logankilpatrick that all Q&A should take place in the community and not via email or private, direct messages; and we need to encourage everyone to post code because this is a developer community and the language of developers is code.
This type of “Q and A” tech forum rule that all Q & A should occur in the public forums is normal for tech communities because the purpose of having a public community is to share code and work on developer solutions in public to create a knowledge base for current and future users.
I don’t think you will find any tokenizer which is “100%” perfect at counting tokens all the time, @polterguy ; writing a lot of code and I find the token approximates work fine especially because you cannot control exactly how the GPT model will reply and exactly how many tokens they will use in a reply.
Of course, I would love to see code which does not use an API to estimate GPT-2 token count, because I prefer to estimate tokens without making another network API call because the more network API calls made, the more the chance for errors, costs and delays.
According to the GPT-3 docs I have read, using a GPT-2 tokenizer is also an approximation for GPT-3 (not exact) and the most recommended way to get a GPT-3 token count is to submit an API call to a GPT-3 endpoint.
Hence, what is why OpenAI offers the “rules to estimate” token count, I think at least.