Why are my context tokens used so quickly?

[hanw9121]

When using the Retrieval function of the Assistants API (uploading six books for retrieval), why are my context tokens used so quickly? One question requires 20,000-30,000 context tokens. Is this normal? It is because of the polling method. Is it the result of multiple inquiries or does it require tokens to find the content of the book? Is there any optimization method?

[_j]

If you upload retrieval information, the assistant backend will remorselessly fill the AI model’s context window with information, or if results are not optimum, the AI has its own function to call for more documentaiton in another turn…

I wrote a response earlier about sources of token use, click “Context tokens…”:

Context tokens in Assistant API

The language of model usage was changed around a bit in the new usage page in November, with the release of assistants (and with that page showing less detailed information). Like the terms “GPT” and “Assistant” already used for other purposes, our ability to refer to things clearly is also now conflated by the new use of “context” to refer to yet another thing. Pricing Page Usage page Completions API object Input Context tokens prompt_tokens Output Generated tokens completion_tokens …

https://platform.openai.com/docs/assistants/how-it-works/context-window-management or https://platform.openai.com/docs/assistants/tools/how-it-works

[hanw9121]

@_j

If you upload retrieval information, the assistant backend will remorselessly fill the AI model’s context window with information, or if results are not optimum, the AI has its own function to call for more documentaiton in another turn

Thank you, can I understand it this way? No matter what questions I ask, it will occupy 16K tokens. Plus some other information, my tokens will reach 20,000+. Is there any way to limit the use of tokens, or relevant code recommendations?

[_j]

The cost associated with the autonomous nature of agents has been an ongoing concern.

Identified the day of release:

Assistant API - Run won't cancel, keeps billing usage

It is likely that it got stuck iterating trying to call a function in an alternating loop of code failures, - or you just gave it a huge document to be set loose on. With no specifying “max_tokens”, and their own conversation management possibly filling the context or omitting failed iterations, it seems like it could be “I put my API key into autoGPT or langchain and it emptied my account” forum scenarios all over again. You can possibly go in and edit the assistant and any code calling and t…

And more the same week:

Assistants API pricing details per message

Exactly. And unlike ChatGPT, OpenAI has no incentive to minimize the conversation loaded into the AI every iteration. They limit gpt-4-turbo output to 4k because the generation is what actually costs money and CPU time, not the loading of $1 billed conversation into it. OpenAI doesn’t describe any techniques such as an embedding database that could extend the illusion of memory, but they do say they’ll truncate only when conversation won’t fit into AI context. You could pull down the thread oc…

There is no implementation of controls in two months following, except by the model selected and its maximum context length.

The solution to budget control is to continue using the chat completions endpoint, with which you have the inconveniencing of needing to unpack “files” into text the AI can understand, but the advantage that you can make knowledge injected (search “RAG”) be very relevant to the current user input and not rely on continued external calls to internal tools. You can also control how much chat history length is really sent.