I hope this message finds you well. I am writing to seek clarification regarding the token counting mechanism in the GPT-3 API, specifically in the context of the Japanese language.
As I understand from your documentation, you charge based on tokens, with billing occurring every 1000 tokens. However, I would appreciate further information on how tokens are counted when processing Japanese text.
For example, let’s consider the name “梅沢良成,” which is my name. This name contains 11 characters. If we were to count based on character length, it would be 4 tokens. If we were to count based on phonetics (pronunciation), it would be 8 tokens (うめざわよしなり). I would like to understand the token counting method for Japanese text more clearly.
Furthermore, I would like to suggest the option of counting tokens in Japanese text based on character length (i.e., counting each character as one token) for ease of understanding and consistency with the language structure.
Could you please provide me with more information on how tokens are counted for Japanese text, and consider the possibility of introducing a character-based token counting option for Japanese language users?
I appreciate your prompt response and assistance in addressing these questions and suggestions. Thank you for your attention to this matter.