Conversation context and quadratic billing

API
1

Hi,

After having wonderful chats with gpt4 via chatgpt plus I wanted to check the API.

but I noticed billing per token is applied again and again to the entire history of the growing conversation.

This means quadratic billing in the length of the conversation.

this sound crazy and I do not comprehend how it is considered remotely usable.

I was sure there must be a context mechanism that prevents this quadratic cost (and wasted computation) but I could not find one in the docs, by asking gpt itself, or users in the discord server.

Is there such a thing?

Thanks,

Nir

2

I think you are observing that a “conversation” is actually stateless. The conversation history is not remembered but has to be repeated every time.

Making the cost grow with every request.

That is how it works, yes.

1 Like
3

Hi @nir.01,

as @paul.armstrong mentioned this is the case.

There are some strategies you could deploy to help you on this, for example: OpenAI API: chat completion pruning methods this is a great way to reduce tokens. Or to limit the resubmitted messages to the last 5 ones in your request.

1 Like
4

I think there’s a fair bit of misunderstanding about OpenAI’s intentions concerning their GPT apps. I defer to @logankilpatrick, but I believe apps like ChatGPT are intended to be demonstrable examples that help everyone use LLMs and discover possibilities while envisioning comprehensive solutions.

In my view, ChatGPT is not to be regarded as a solution to any specific personal or business product.

The entire point of a comprehensive API and a community that leans into the development of OpenAI apps is to use that API to build stuff that solves problems with AGI. If you are looking for a solution that avoids the pitfalls of quadratic billing costs, you need to look to developers who know how to craft a solution that optimizes for costs using all of the tools that OpenAI has made available.

This might involve more than one of the APIs and likely other technologies like real-time databases, vector data stores, and other supporting infrastructure that would make your vision financially practical while meeting your objectives.