Token count for Python 3.7

API
1

I would love to use tiktoken to count how many tokens are in my string, however my pythonanywhere.com app is currently runniny python 3.7 and there are a tonne of things I’d need to do to get 3.8 compatability with all my modules.

So is there any way to get tiktoken working on python 3.7, or can someone suggest an alternative module? I’m using textBlob at the moment, which doesn;t see mto be super accurate for gpt3.5 token counting.

Appreciate any help!
Dale

1 Like
3

thanks, would that be a closer approximation than the textblob method I’m using?

def count_tokens(vCountTokenStr):
    # Tokenize the input string
    blob = TextBlob(vCountTokenStr)
    tokens = blob.words

    # Count the number of tokens
    num_tokens = len(tokens)
    return num_tokens