Will a pause in the streaming also pause the cost?

API
, ,
1

I created an assistant where users can ask some things.
Some answers could be HUGE, so I am considering implementing this:

  • Enable Straming
  • After 500 tokens, stop the streaming and ask the user if they want to “Continue reading”
  • If the user click on “Continue”, it will charge the user for it.

This will allow me to charge the user for every bunch of 500 tokens.

Now… I asked about it to ChatGPT 4o and 4.1, and 4o says: “Yes, pausing the streaming will pause the costs”, and then 4.1 says: “No, it won’t. You’ll be charged anyway”.

Who is right?

Does somebody has some code example?

2

That is not productive or useful.

Every API call, you’ll be billed for the entire chat context.

So “continue” would be another API call, costing the entire conversation history again as token input, including what was just written by the AI.

In an API return, you are provided a “usage” object to track your costs per call.

The AI models currently offered won’t write much more than 1500 tokens; they’ll find ways to wrap up. If you want to completely cut off the output with truncation, like if the user still got the AI to recite nonstop loops of text, a more reasonable API parameter on a non-reasoning AI model is max_completion_tokens = 2500 (the name of the output budget parameter for Chat Completions).