What's the tokenization algorithm GPT-4.1 uses?

API
1

Is there anyone who knows what tokenization algorithm GPT-4.1 uses?
Since the latest version of tiktoken (0.9.0) does not include support for GPT-4.1, I am not sure which encoding I should use.

2

GPT-4.1 uses the same tokenizer as 4o; same encoding.

2 Likes
4

Thank you for the information! :smiley:

1 Like