I know you have libraries helping out in Python and such, but I’m using an “esoteric” programming language to interact with the API, and I need to know how I can manually calculate how many tokens a prompt will result in. I’ve tried length of string and divide by four, and it doesn’t work. I’ve trie…

Thank you, it surely helps, but I need exact :confused: But thank you, you’ve been a champ … :slight_smile:

THAT would be SUPER interesting, as in getting the C# version - Assuming it’s 100% “perfect” in its calculations :slight_smile:

I’m happy to release the c# version There are differences that I have not nailed down. I am using the exact same token libraries .another member gave me a suggestion I never got time to follow through with I’ll extract it from my main app and package it up for others to use soon. (It’s embedded in…

[image] raymonddavey: I’m happy to release the c# version Thank you! We need a new OpenAI community rule, which I plan to propose to @logankilpatrick that all Q&A should take place in the community and not via email or private, direct messages; and we need to encourage everyone to post code…

[image] polterguy: THAT would be SUPER interesting, as in getting the C# version - Assuming it’s 100% “perfect” in its calculations I don’t think you will find any tokenizer which is “100%” perfect at counting tokens all the time, @polterguy ; writing a lot of code and I find the token appro…

Here is the correct link [image] Do you get billed extra when echo=true API I know you get billed for the prompt and the completion But I was wondering if you get billed for the prompt twice when you set echo=true? It comes back as part of the completion. Previous …

I couldn’t agree more. I’ve got 10K commits to GitHub the last 5 years or so, and everything we’re doing in the AI space is 100% Open Sauce ==> GitHub - polterguy/magic: Generate a web app in seconds with Low-Code and AI So I totally agree!

[image] polterguy: I couldn’t agree more. I’ve got 10K commits to GitHub the last 5 years or so, and everything we’re doing in the AI space… Respect! I am very biased toward coders and wish more people here would “speak” through code and working examples! After all, this is supposed to be …

Say we put a sample /etc/hosts file into the tokenizer . 127.0.0.1 localhost 127.0.1.1 ha-laptop # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters It says this would parse to 75 tokens. The sample abo…

What is the OpenAI algorithm to calculate tokens?

API

raymonddavey February 13, 2023, 5:56am 19

Here is the correct link

Previous post deleted to avoid confusion

Topic		Replies	Views
Does ChatGPT understand indented JSON better than minified? Prompting code-interpreter , json	26	5958	July 3, 2025
How to calculate the tokens when using function call API	58	50089	February 19, 2024
Prompt_tokens vs tiktoken.encoding_for_model().encode() Prompting gpt-35-turbo , token	4	5624	August 3, 2023
Counting tokens for chat API calls (gpt-3.5-turbo) Documentation	5	28435	December 13, 2023
Struggling to get correct token count Community gpt-4 , gpt-35-turbo , api	3	2114	December 29, 2025

What is the OpenAI algorithm to calculate tokens?

Related topics