Can I assume that it is using GPT4o’s tokenizer? Because I kind of use tiktoken to roughly estimate my token count and based on what I see, it is slightly underestimating it. Unsure if this is because GPT4.1’s tokenizer is different from GPT4o’s or if that’s normal.
3 Likes