Hello, I am using diacritics and it seems like with diacritics openAI takes more tokens. Check image, can you advise how to avoid this? thanks a lot
Welcome to the community.
Unfortunately, as far as I know, there’s no way to control how tokens are broken up.
Hope this helps.
1 Like