How to abort create chat completion streaming? I use Nodejs + Typescript

Hello everyone,

I would like to know how to stop streaming create chat completion. According to the documentation of OpenAI Node API library ([OpenAI Node API Library]) it is necessary to do you can break from the loop or call stream.controller.abort()

But I have the impression that this doesn’t seem to work at all. He still continues to generate.

For streaming i am using SSE

const response = await openai.chat.completions.create({
        model: "gpt-3.5-turbo-16k",
        messages: [
          {
            role: "user",
            content: userMsg,
          },
          {
            role: "system",
            content: systemMsg(project)
          }
        ],
        stream: true,
        temperature: 1,
        max_tokens: 16185,
        top_p: 1,
        frequency_penalty: 0,
        presence_penalty: 0,
      });

      io.on("connection", (socket) => {
        socket.on("cancel_request", (data) => {
          if (data === true) {
            keepGoing = false;
            response.controller.abort();
          }
        });
      });

      for await (const chunk of response) {
        if (chunk === undefined) return;

          res.write(`data: ${JSON.stringify(chunk.choices[0].delta.content)}\n\n`);

        if(!keepGoing){
          break;
        }
      }

From my Frontend i use Socket IO. When user click on button ‘Cancel’ i emit message and i listening on my backend.

Small clarification, my frontend is separate from the backend.

Thank you in advance for your answers

I’ve investigated, and seemingly you can’t make the AI model stop generating and billing. You can only close the connection and ignore the subscription of server-sent events.

I believe this was a bug and has now been fixed actually.

I genuinely would like to see this. I was under the impression that they would also close the connection as the server is notified when a client prematurely closes their subscription. I haven’t dug into this myself though.

I just tried it myself. Now have to wait ~5 minutes to see the billing.

Unfortunately not. It was just something that never occurred when I first starting using ChatGPT, then started happening, and now seems to have resolved.

Confirmed not true:

Asked for (hopefully a long prompt) & cancelled

Billing details:

1 Like

Or it’s how ChatGPT operated until they fixed it. And it looks like they did.

Cancelled output (gets no titling either):

image

Recalling just gets you a few more tokens instead of a full reply.

image

Now, method to do that in all the libraries, please…

Ok great. Very thank you much for your answer.

1 Like