What is this new what is this new what is this new

API
1

Heres me making a blank post because OpenAI mods REFUSE to delete them themselves. Delted post. deleted post.

2

It basically allows you to receive tokens back in batches so that you can give the appearance of generation like with chatgpt

2 Likes
3

what does that mean appearance of generation?

4

Streaming is the sending of words as they are created by the AI language model one at a time, so you can show them as they are being generated.


(technical: Subscription to a server-sent push event)

1 Like
5

oh no shit huh? Does that cost more or its just a fun thing you can add?

6

It is the same cost per token.

Instead of waiting 30 seconds for the complete answer, you start receiving almost immediately.

It is not “just fun”, imagine if you had to stare at ChatGPT for half a minute wondering what it was going to say.

4 Likes
7 

  const response = await openai.chat.completions.create({
    model: "gpt-4",
    messages: [
      {
        role: "system",
        content: message,
      },
    ],
    temperature: 1.1,
    max_tokens: 600,
    top_p: 1,
    frequency_penalty: 0.3,
    presence_penalty: 0.5,
  });

So this works fine, but when I add stream: true…

  const response = await openai.chat.completions.create({
    model: "gpt-4",
    messages: [
      {
        role: "system",
        content: message,
      },
    ],
    stream: true,
    temperature: 1.1,
    max_tokens: 600,
    top_p: 1,
    frequency_penalty: 0.3,
    presence_penalty: 0.5,
  });

  res.json({
    message: response.choices[0].message.content.trim(),
    usage: response.usage,
  });
});

All of a sudden i get this error

    message: response.choices[0].message.content.trim(),
                             ^

TypeError: Cannot read properties of undefined (reading '0')
    at C:\Users\tvent\OneDrive\Desktop\gpt\server\index.js:204:30  
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
8

Yes, you’re going to get chunk objects, a format documented in the API reference, or dumped out for me just now.


{
  "id": "chatcmpl-82dsfjjaofldfa3mOIDJFOIJ",
  "object": "chat.completion.chunk",
  "created": 1695524999,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": " for"
      },
      "finish_reason": null
    }
  ]
}

You get them until finish_reason: stop or length

9

right im saying when i adjust my code though I don’t know how to get it to work.

  const response = await openai.chat.completions.create({
    model: "gpt-4",
    messages: [
      {
        role: "system",
        content: message,
      },
    ],
    stream: true,
    temperature: 1.1,
    max_tokens: 600,
    top_p: 1,
    frequency_penalty: 0.3,
    presence_penalty: 0.5,
  });

  for await (const chunk of response) {
    console.log(chunk.choices[0].delta.content); // This correctly streams it in the terminal 
  }

  res.json({
    message: response.choices[0].delta.content.trim(), // does not stream to the front end
    usage: response.usage,
  });
});

Basically how do i get the res.json to stream? Im sending the message variable to the front end, that works and is all set up. So how do I make it work within instead of that for await loop in my res.json?

11

The “all of a sudden” is likely that you are not handling the finish_reason case.
Or that you get a function_call, which just appears at the object root.

End of stream:


{
  "id": "chatcmpl-jdfj933jaf03kar03raf",
  "object": "chat.completion.chunk",
  "created": 1111111111,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": "?"
      },
      "finish_reason": null
    }
  ]
}
{
  "id": "chatcmpl-82ojfoaFsD7",
  "object": "chat.completion.chunk",
  "created": 112222222,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "delta": {},
      "finish_reason": "stop"
    }
  ]
}
1 Like
12

so i should add stop to the end of the function?

It works on the console.log it is streaming and logging it

  for await (const chunk of response) {
    console.log(chunk.choices[0].delta.content); // this code from the doc runs
  }

its not working in the res.json

  res.json({
    message: response.choices[0].delta.content.trim(), // this is causing the script to fail
    usage: response.usage, // What happened to the usage object?
  });
1 Like
13

I provided the second to the last and the last object received in a stream.

See that part in your code where it says .delta.content?

See any “content” in the last chunk?

You need to short-circuit out by detecting the finish_reason first.

15

I don’t know the purpose of res.json in this context. The tools/environment you’re using are not my forte.

Typically what you’ll need to do is have two different mechanisms going

if not a finish_reason or massive_error:

  • display the content of the chunk
  • append the content of the chunk to your variable for capturing the message

then do message stuff like chat history with the completed message.

Here’s a little python chatbot at least.

16

what happens to the usage object if you convert to streaming?

17

You have to count tokens on your own with streaming which is the trade-off…

18

How do I count my own tokens without being told what my tokenage is? -_-

19

If you search the forum, you can find out more on tiktokken

20

Also, if you are embedding the input and outputs, which you should do anyway to maintain good context within the bot, a nice side effect of Ada-002 is that is reports out the token counts.

1 Like
21

Here is new docs providing chunks of streaming. https://platform.openai.com/docs/api-reference/streaming