Hi Community,
Embedding a huge set of json files. How to calculate the token count? I used tictoken to estimate the count of character in the json. but an online json character showed a different count of character. What is the best way to estimate token needed relative to the character count?
can anyone help me with some insights? Thank you.