BUG: Streaming packets changed

API Bugs
,
1

the streaming packets are supposed to come in chunks with data: {} now they are getting cut off mid JSON delta JSONs are getting split across multiple packets

4 Likes
2

Yes, same. been trying to figure out how to parse/work with it for the last hour.

3

You’re not the only one. Everyone using ruby-openai is affected.

My service is completely down at the moment. Really messed up that it’s not being acknowledged as an outage - if it’s not considered an outage then it’s really messed up that we were not given a heads up that the API behavior is changing.

1 Like
4

I had to buffer the data and use a regex to parse the complete packets in php

preg_match_all("/^data: ({.*})[\n|$]/m",$buffer,$matches);
			
		$parts = [];
		if(count($matches) == 2){
			
			foreach($matches[0] as $json){	
				$buffer = trim(str_replace($json,'',$buffer));
			}

			foreach($matches[1] as $json){
				$parts[] = $json;
			}
		}
1 Like
5

This should be fixed in v5.2 - thanks @obiefernandez

6

This is how we fixed it - we moved to Azure. You’re breaking our product’s functionality, changing the API without even notifying about it?

7

This is a fairly new issue, it may not have been counted as an outage because it has not been noticed yet. I would recommend contacting help.openai.com.

8

Hi folks, we are actively investigating this issue. Our current hypothesis is that you may be seeing this issue if your SSE client is not correctly parsing responses with buffering.

For anyone affected, if you are able, can you:

  1. Share if you are using any proxy layer (e.g. Cloudflare) or third party library (non-OpenAI SDK, e.g. Langchain etc.)

  2. Share logs and request IDs from a broken response?

Ideally the output from a request like this:

curl https://api.openai.com/v1/chat/completions -u :$OPENAI_API_KEY -H 'Content-Type: application/json' -d '{"model": "gpt-3.5-turbo", "messages": 
[{"role": "user", "content": "Write a 5 line poem."}], "stream": true}' -vv

(WARNING: please strip out the Authorization header in the content!)

You can post here or email it to me at atty@openai.com. Thanks!

2 Likes
9

This is also affecting my company’s app. We use gpt-4 as the model.

10

@robb can you share logs of a failed request, and which SSE client / code you are using to parse the response? I suspect this is an issue with your client is parsing the output.

11

Hello, Atty. It seems now the issue fixed but,
we were receiving this, and the app only printed “today?”.

[2023-10-31 10:30:53] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“content”:" can"},“finish_reason”:null}]}

[2023-10-31 10:30:53] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“content”:" I"},“finish_reason”:null}]}

[2023-10-31 10:30:53] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“content”:" assist"},“finish_reason”:null}]}

[2023-10-31 10:30:53] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“content”:" you"},“finish_reason”:null}]}

data: {“id”:“chatcmpl-8FU8RVlLJTbu4U6ScYtTwYQlUOKiM”,“object”:“chat.completion.chunk”,“created”:1698701451,“model”:“gpt-4-0613”,“choices”:[{“index”:0,“delta”:{“content”:" today"},“finish_reason”:null}]}

data: {“id”:“chatcmpl-8FU8RVlLJTbu4U6ScYtTwYQlUOKiM”,“object”:“chat.completion.chunk”,“created”:1698701451,“model”:“gpt-4-0613”,“choices”:[{“index”:0,“delta”:{“content”:“?”},“finish_reason”:null}]}

data: {“id”:“chatcmpl-8FU8RVlLJTbu4U6ScYtTwYQlUOKiM”,“object”:“chat.completion.chunk”,“created”:1698701451,“model”:“gpt-4-0613”,“choices”:[{“index”:0,“delta”:{},“finish_reason”:“stop”}]}

12

@BrianLovesAI these responses all seem well formed to me, were you seeing an error?

14

Thanks @artistsinbusiness, any chance you could share the logs of what you are seeing and the request IDs please (mentioned in the response header X-Request-Id). Feel free to post here or email me at atty@openai.com.

15

Hmm… I feel like,

{“index”:0,“delta”:{“content”:" you"},“finish_reason”:null}]} ← This format was new just today, and just before.

{“id”:“chatcmpl-8FUXAInfxgCttp1a7iMTjdD1qkncy”,“object”:“chat.completion.chunk”,“created”:1698702984,“model”:“gpt-4-0613”,“choices”:[{“index”:0,“delta”:{“content”:" steps"},“finish_reason”:null}]} ← This format was normal all time before.

So, it looks like the returning value was changed?

16

Atty, I’m actually not at a place I can share a log with you at the moment but I can confirm that it has now corrected itself.

17

I am experiencing this issue also, using this application: https://github.com/LagPixelLOL/ChatGPTCLIBot/

Error looks like this: Error when calling API: Error parsing JSON: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected ‘:’; expected ‘[’, ‘{’, or a literal

18

For me as well, everything is okay now.

All the logs are looking fine now. :slight_smile:

I will keep monitoring my app… thanks for your assist.

19

When the error occurred, I received what looked like a broken JSON string. It could be the same issue.

[2023-10-31 10:30:52] server.DEBUG: Issue Track: Test Mode: event: message
data: {“id”:“chatcmpl-8FU8RVlLJTbu4U6ScYtTwYQlUOKiM”,“object”:“chat.completion.chunk”,“created”:169870

[2023-10-31 10:30:52] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“role”:“assistant”,“content”:“”},“finish_reason”:null}]}

[2023-10-31 10:30:52] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“content”:“Hello”},“finish_reason”:null}]}

[2023-10-31 10:30:53] server.DEBUG: Issue Track: Test Mode: event: message
data: {“index”:0,“delta”:{“content”:“!”},“finish_reason”:null}]}

…

20

Sorry you are right, that getting only this is invalid JSON: data: {“index”:0,“delta”:{“role”:“assistant”,“content”:“”},“finish_reason”:null}]}.

What client are you using for SSE parsing? Can you share your parsing code?

21

Symptom not experienced with a quick test. Python library, which I assume would also die if fed half a json as would my code.

For 4 trials of gpt-3.5-turbo @ 2023-10-30 03:12PM:

Stat Minimum Maximum Average
latency (s) Min: 0.4415 Max: 0.801 Avg: 0.649
total response (s) Min: 0.8426 Max: 1.1771 Avg: 1.042
total rate Min: 21.239 Max: 29.67 Avg: 24.398
stream rate Min: 51.9 Max: 72.7 Avg: 62.000
response tokens Min: 25 Max: 25 Avg: 25.000

For 4 trials of gpt-4 @ 2023-10-30 03:12PM:

Stat Minimum Maximum Average
latency (s) Min: 0.5016 Max: 1.2001 Avg: 0.741
total response (s) Min: 2.6458 Max: 4.1234 Avg: 3.203
total rate Min: 6.063 Max: 9.449 Avg: 8.030
stream rate Min: 7.1 Max: 11.9 Avg: 10.150
response tokens Min: 25 Max: 25 Avg: 25.000