How to abort create chat completion streaming? I use Nodejs + Typescript

API
1

Hello everyone,

I would like to know how to stop streaming create chat completion. According to the documentation of OpenAI Node API library ([OpenAI Node API Library]) it is necessary to do you can break from the loop or call stream.controller.abort()

But I have the impression that this doesn’t seem to work at all. He still continues to generate.

For streaming i am using SSE

const response = await openai.chat.completions.create({
        model: "gpt-3.5-turbo-16k",
        messages: [
          {
            role: "user",
            content: userMsg,
          },
          {
            role: "system",
            content: systemMsg(project)
          }
        ],
        stream: true,
        temperature: 1,
        max_tokens: 16185,
        top_p: 1,
        frequency_penalty: 0,
        presence_penalty: 0,
      });

      io.on("connection", (socket) => {
        socket.on("cancel_request", (data) => {
          if (data === true) {
            keepGoing = false;
            response.controller.abort();
          }
        });
      });

      for await (const chunk of response) {
        if (chunk === undefined) return;

          res.write(`data: ${JSON.stringify(chunk.choices[0].delta.content)}\n\n`);

        if(!keepGoing){
          break;
        }
      }

From my Frontend i use Socket IO. When user click on button ‘Cancel’ i emit message and i listening on my backend.

Small clarification, my frontend is separate from the backend.

Thank you in advance for your answers

2

I’ve investigated, and seemingly you can’t make the AI model stop generating and billing. You can only close the connection and ignore the subscription of server-sent events.

3

This is correct.

This is related to how on ChatGPT when you click “Stop Generating” on a response which isn’t going the way you are hoping, edit your message, and save and submit you often get an error message saying something about not being able to send multiple messages at once.

4

I believe this was a bug and has now been fixed actually.

I genuinely would like to see this. I was under the impression that they would also close the connection as the server is notified when a client prematurely closes their subscription. I haven’t dug into this myself though.

I just tried it myself. Now have to wait ~5 minutes to see the billing.

5

Interesting. Did you read that somewhere? I just encountered this very recently—within the past few days, though I’m unable to replicate it right now.

6

Unfortunately not. It was just something that never occurred when I first starting using ChatGPT, then started happening, and now seems to have resolved.

7

Confirmed not true:

Asked for (hopefully a long prompt) & cancelled

Screenshot from 2023-09-17 12-07-28
Screenshot from 2023-09-17 12-07-281265×808 46.9 KB

Billing details:

Screenshot from 2023-09-17 12-07-37
Screenshot from 2023-09-17 12-07-37814×349 22.2 KB

Screenshot from 2023-09-17 12-12-01
Screenshot from 2023-09-17 12-12-011177×810 91.6 KB

8

Or it’s how ChatGPT operated until they fixed it. And it looks like they did.

Cancelled output (gets no titling either):

image

Recalling just gets you a few more tokens instead of a full reply.

image

Now, method to do that in all the libraries, please…

9

Ok great. Very thank you much for your answer.

1 Like