I am aware that gpt-3-encoder is not supporting gpt4 cl100k-base encoding. I only used it because it was recommended by OpenAI in token counting page or their cookbook since I wasnt sure how to trust other available libs. Somehow they just updated to dqbd/tiktoken on that page. When I shared with you in previous message, it was gpt-3-encode. I think it was good enough for my usage but I will change it especially now since they are recommending this now.
Clearly I am aware of the token limit. I wasn’t trying to use that 300k character in prompt but there are many different contents with varying lengths, such as 0 character to 500k character . So what I do usually to count token and truncate the content accordingly to pass in prompt. But I changed to handle token counting in chunk to improve the performance impact and only use what we need and ignore rest of the tokens in 300k character. I was curious if there is any other way or tool.