Which embedding tokenizer should I use?

[SomebodySysop]

@walaszczykm

If you use OpenAI’s tiktoken (GitHub - openai/tiktoken) according to the documentation, it not only allows you to specify the toknizer directly by get_encoding function, but what is even greater, you can get tokenizer by providing the name of the model you would like to use leaving a choice of corresponding tokenizer to the library itself: tiktoken.encoding_for_model("text-davinci-003")

I am actually using cURL in PHP to make the calls. Do you use BERT or CL100K_base?