BUG: Streaming packets changed

This should be fixed in v5.2 - thanks @obiefernandez

This is how we fixed it - we moved to Azure. You’re breaking our product’s functionality, changing the API without even notifying about it?

This is a fairly new issue, it may not have been counted as an outage because it has not been noticed yet. I would recommend contacting help.openai.com.

Hi folks, we are actively investigating this issue. Our current hypothesis is that you may be seeing this issue if your SSE client is not correctly parsing responses with buffering.

For anyone affected, if you are able, can you:

  1. Share if you are using any proxy layer (e.g. Cloudflare) or third party library (non-OpenAI SDK, e.g. Langchain etc.)

  2. Share logs and request IDs from a broken response?

Ideally the output from a request like this:

curl https://api.openai.com/v1/chat/completions -u :$OPENAI_API_KEY -H 'Content-Type: application/json' -d '{"model": "gpt-3.5-turbo", "messages": 
[{"role": "user", "content": "Write a 5 line poem."}], "stream": true}' -vv

(WARNING: please strip out the Authorization header in the content!)

You can post here or email it to me at atty@openai.com. Thanks!

2 Likes

This is also affecting my company’s app. We use gpt-4 as the model.

@robb can you share logs of a failed request, and which SSE client / code you are using to parse the response? I suspect this is an issue with your client is parsing the output.

Hello, Atty. It seems now the issue fixed but,
we were receiving this, and the app only printed “today?”.

[2023-10-31 10:30:53] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“content”:" can"},“finish_reason”:null}]}

[2023-10-31 10:30:53] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“content”:" I"},“finish_reason”:null}]}

[2023-10-31 10:30:53] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“content”:" assist"},“finish_reason”:null}]}

[2023-10-31 10:30:53] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“content”:" you"},“finish_reason”:null}]}

data: {“id”:“chatcmpl-8FU8RVlLJTbu4U6ScYtTwYQlUOKiM”,“object”:“chat.completion.chunk”,“created”:1698701451,“model”:“gpt-4-0613”,“choices”:[{“index”:0,“delta”:{“content”:" today"},“finish_reason”:null}]}

data: {“id”:“chatcmpl-8FU8RVlLJTbu4U6ScYtTwYQlUOKiM”,“object”:“chat.completion.chunk”,“created”:1698701451,“model”:“gpt-4-0613”,“choices”:[{“index”:0,“delta”:{“content”:“?”},“finish_reason”:null}]}

data: {“id”:“chatcmpl-8FU8RVlLJTbu4U6ScYtTwYQlUOKiM”,“object”:“chat.completion.chunk”,“created”:1698701451,“model”:“gpt-4-0613”,“choices”:[{“index”:0,“delta”:{},“finish_reason”:“stop”}]}

@BrianLovesAI these responses all seem well formed to me, were you seeing an error?

Thanks @artistsinbusiness, any chance you could share the logs of what you are seeing and the request IDs please (mentioned in the response header X-Request-Id). Feel free to post here or email me at atty@openai.com.

Hmm… I feel like,

{“index”:0,“delta”:{“content”:" you"},“finish_reason”:null}]} ← This format was new just today, and just before.

{“id”:“chatcmpl-8FUXAInfxgCttp1a7iMTjdD1qkncy”,“object”:“chat.completion.chunk”,“created”:1698702984,“model”:“gpt-4-0613”,“choices”:[{“index”:0,“delta”:{“content”:" steps"},“finish_reason”:null}]} ← This format was normal all time before.

So, it looks like the returning value was changed?

Atty, I’m actually not at a place I can share a log with you at the moment but I can confirm that it has now corrected itself.

I am experiencing this issue also, using this application: https://github.com/LagPixelLOL/ChatGPTCLIBot/

Error looks like this: Error when calling API: Error parsing JSON: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected ‘:’; expected ‘[’, ‘{’, or a literal

For me as well, everything is okay now.

All the logs are looking fine now. :slight_smile:

I will keep monitoring my app… thanks for your assist.

When the error occurred, I received what looked like a broken JSON string. It could be the same issue.

[2023-10-31 10:30:52] server.DEBUG: Issue Track: Test Mode: event: message
data: {“id”:“chatcmpl-8FU8RVlLJTbu4U6ScYtTwYQlUOKiM”,“object”:“chat.completion.chunk”,“created”:169870

[2023-10-31 10:30:52] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“role”:“assistant”,“content”:“”},“finish_reason”:null}]}

[2023-10-31 10:30:52] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“content”:“Hello”},“finish_reason”:null}]}

[2023-10-31 10:30:53] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“content”:“!”},“finish_reason”:null}]}

Sorry you are right, that getting only this is invalid JSON: data: {“index”:0,“delta”:{“role”:“assistant”,“content”:“”},“finish_reason”:null}]}.

What client are you using for SSE parsing? Can you share your parsing code?

Symptom not experienced with a quick test. Python library, which I assume would also die if fed half a json as would my code.

For 4 trials of gpt-3.5-turbo @ 2023-10-30 03:12PM:

Stat Minimum Maximum Average
latency (s) Min: 0.4415 Max: 0.801 Avg: 0.649
total response (s) Min: 0.8426 Max: 1.1771 Avg: 1.042
total rate Min: 21.239 Max: 29.67 Avg: 24.398
stream rate Min: 51.9 Max: 72.7 Avg: 62.000
response tokens Min: 25 Max: 25 Avg: 25.000

For 4 trials of gpt-4 @ 2023-10-30 03:12PM:

Stat Minimum Maximum Average
latency (s) Min: 0.5016 Max: 1.2001 Avg: 0.741
total response (s) Min: 2.6458 Max: 4.1234 Avg: 3.203
total rate Min: 6.063 Max: 9.449 Avg: 8.030
stream rate Min: 7.1 Max: 11.9 Avg: 10.150
response tokens Min: 25 Max: 25 Avg: 25.000

It looks like this library is not parsing SSE outputs correctly. It assumes that when the stream content is split by \n\ndata:, each line is valid JSON. However, SSE clients should buffer output until they hit a separator character and only then parse the chunk into JSON.

1 Like

Regardless the way its streamed has changed today or this weekend.
and sending partial JSON packets between 2 packets is just annoying

If you are using Javascript/Typescript, make sure to use GitHub - rexxars/eventsource-parser: Streaming, source-agnostic EventSource/Server-Sent Events parser for buffering the chunk data.

Hi all, upon further investigation we have determined that there was indeed a bug that affected a subset of streaming responses on v1/chat/completions from 10:19 AM PT to 2:41 PM PT. This was fixed and impact stopped at 2:41 PM PT. Apologies for the downtime, and we are fixing the root cause as a follow-up.

5 Likes