Diacritics tokens

Hello, I am using diacritics and it seems like with diacritics openAI takes more tokens. Check image, can you advise how to avoid this? thanks a lot

Welcome to the community.

Unfortunately, as far as I know, there’s no way to control how tokens are broken up.

Hope this helps.

1 Like