Truncating based on tokens

Hey folks, I’m wondering if there any good libraries that help with truncating text based on tokens. Basically, I want a function that lets me input a string and truncate it to a maximum number of tokens (both from the head or tail of the string).

So far, I’ve been using some off the shelf tokenizers coupled with a loop searching for how many words I need to get rid of to fit a maximum token count.

2 Likes

What sort of tokenization methodology?