Does anyone have an idea on how to obtain the token count of my prompt in SharePoint SPFx? I’m using the SPFx framework with React.js and TypeScript. Additionally, I don’t have any Node.js or backend server. I’ve tried several libraries such as js-tiktoken, tiktoken, gpt-3-encoder, gpt-4-encoder, but none of them seem to work. Any suggestions?
I have not yet tested it, but Microsoft also has a JavaScript/TypeScript implementation of the tokenizer based on the Tiktoken reference implementation. It’s available at npm as @microsoft/tiktokenizer.
I try it already that it is also used server side scripting.