28

I also ended up writing my findings up here: Counting OpenAI tokens • Harry Marr

2 Likes
29

Hi there, is there a way we could count the total_tokens using tiktoken library (am using python) ?

30

The whole point of this thread is that the functions and parameters passed to the API are not the countable tokens the API gives to the AI engine.

If you want to investigate your own function or function list, before deploying a static solution, simply include or don’t include function= in the identical python library call, and calculate the difference between the prompt_tokens responses (of a non-streaming response).

To make a token counter for functions, one must essentially write a json_schema → function_pickle_output that replicates what OpenAI does exactly, for all possible inputs.

1 Like
31

FYI for others reading this post.

As a moderator I used a beautify on the code as it was not readable in the original format. @_j notified me that he would like it back to the original but I was unable to revert it.

So if you also don’t agree with the formatting, it is not his fault it is mine. Sorry.

32

Just an update, from testing there are two main problems I find with their current parser; the lack of descriptions for the members of an object-as-parameter, it also seems like gpt4 has some problems recognising “arrays of arrays of objects”; I’m currently using my own parser that adds these missing ‘comment lines’ to the “Ts definition”, and moves the default comment to the same line as the object parameters name’s definition, and also prepends an ‘array of array of… objects’ notation on the same line, which seems to make it a bit more responsive and stable for my personal use-case…

What I end up with looks like
# Tools ## functions et.c. ...

// Function description here
type some_function = (_: {
// First parameter description here
param_one: { // array of array of objects. default: ...(redacted)
  // First parameter's name description
  name: string,
  // First parameter's count description
  count: number,
}[][],
}) => any;

Rather than what seems to be OpenAI’s parser of

// Function description here
type some_function = (_: {
// First parameter description here
param_one: {
  name: string,
  count: number,
}[][], // default: ...(redacted)
}) => any;
33

Hi! Is there any Python implementation of token counting which at least semi-works with raw json schemas? AutoGPT is nice, but it works on top of its own internal representation, which means it cannot be used off the shelf.

1 Like
34

Hi and welcome to the forum!

I did a quick tiktoken python example here GitHub - foxabilo/TikToken-Web

It’s using the cl100k base model and is the one used by gpt3.5 and 4 so should be good for your use case, this example is a web interface to tiktoken that will take any form of text as an input, the python (server.py) should show the method, you can see the code running over at Tiktoken Web Interface cl100k_base

35

The question is whether any solution will count the tokens consumed by the JSON of a functions list — a list of functions that would be sent to the API, reinterpreted, and then passed to the AI in an unseen form only documented by the experimentation and revelation seen earlier in this thread.

There are prior links to some code approximation attempts earlier in the thread, but your tiktoken alone is not that.

36

Looking at the API code the, json text in the function header is included as a string to the model as would any normal bit of text, so I think if you were to tiktoken that string and add on a few tokens for possible “start of function” “end of function” boundary markers, it should be possible to get an accurate count of the tokens sent, I’ve not gotten round to it, but I did that some time ago for the standard API calls to calculate the additional tokens used by markers and such.

37

This is an incorrect assumption.

I’ll scroll back for you, where you can see the last code block in my post. The AI-received-language is not the input JSON and not as one might predict: How to calculate the tokens when using function call - #24 by _j

With the secret sauce (jailbreaks I don’t want patched and AI crippled by disclosure into necessarily never following a single instruction) one can replay example arbitrary functions again and again and formulate the pattern. I just don’t have any reason to sit down and code this as I’m not trying to eek out the last of input context length with dynamic functions.

Omit “max_tokens” and all context is yours.

1 Like
38

Ok, but the API right now is passing “something” to the model via a connection, it’s that “something” we are interested in, correct? It should be trivial to look at the open source API library to see what is being done with a prompt that contains a function element, it should be deterministic in nature and reproducible with tiktoken.

39

The text is transformed by OpenAI’s internal “function” endpoint after the model selector, load distributor, accounting server paths, other undocumented routing internals. Not any public code.

Here I extract and show the byte string transmitted by the python API module, including a function: Typo in OpenAI API Documentation - #2 by _j

The input is validated against a standard JSON schema, and rejected for malformed requests (“bool” or “float” instead of “boolean” or “number”), but no warning is given when the programmer’s intentions are damaged by the rewriting (omitting keywords like “example”, “maximum”, or discarding any nested descriptions)

Passing a function even selects a differently-trained model than without.

40

Interesting, I’ll have a wander through the repo when I get time.

Cheers.

41

FWIW, I’ve validated that hmarr’s method is the most accurate and it validates itself against OpenAI’s API which is “the right way” to do all this.

Big kudos since none of this is really documented, even within OpenAI’s cookbook code

2 Likes
42

I have played with function calling for a few months now I guess, and I can tell for sure that output passed to arguments is not validated against the definition I have seen so many times, the arguments being just plain text which was supposed to be in content, sometimes some simple text instead of arguments like The color is #ff0000, width is 2px instead of actual JSON.

P.S. Function names are also not protected, so many times I am getting random function names it decided it exists

43

ok, so this means we can provide descriptions as long as I want in the function JSON, and it will not waste the token limit of the api? that’s really great news.

44

The descriptions are text that is inserted into the AI language that allow it to understand what the function is for. Likewise, AI will also receive the text of function names and property names.

The actual text received by the AI is what costs you the tokens. So no, you are still charged for descriptions, which consume context length.

You can also misuse descriptions in higher nesting levels, and then the AI never receives them.

This thread is about the exact and accurate method of calculating not just the sent language, but the overhead of using functions.

45

The description for the function parameter is charged by openAI. It need to be clear but as short as you can. :grinning:

46

Made a python version of hmarrs typescript program. Was looking for one myself so thought it might come in handy for some!

2 Likes
47

Ultimately, the only way to correctly count tokens is to render the full system message as the AI receives it. Only then will you have a container that is not affected by what surrounds it (because there is no joining with the special non-dictionary tokens enclosing a message.)

あなたは数学が好きな役立つアシスタントです

# Tools

## functions

namespace functions {

// Description of example function
type function_name = (_: {
// description of function property 1: string (required)
property1: string,
// description of function property 2: string w enum
property2?: "enum_yes" | "enum_no",
}) => any;

} // namespace functions

Take that Japanese system prompt. The last token of the system prompt is です Then the next token that must be produced is two carriage returns (\n\n)

If instead we put a Japanese period at the end of the line (。) then the tokenizer will use the token for (。\n\n) - a different system prompt doesn’t need a (\n\n) token to advance down to the next line of the tools description.

We can get a different token at the end of the system prompt than if we hadn’t included a function. When using functions, most system prompts will get the two line feeds “for free”, but others won’t.

Also note that the encoder must use the “required” property, which removes the question mark. of properties.