Hi! How many tokens is “Say this is a test!”?
Is it six via https://platform.openai.com/tokenizer or thirteen IIUC from the API itself?
3 Likes
Welcome to the forum.
@_j knows more, but I believe there’s 6 or 7 tokens that OpenAI uses that’s tacked on for ChatML…