Malformed streaming answers from GPT-4 completions API lately

Since this morning, I’m noticing that the completions API is replying with malformed answers every now and then. Anyone noticing the same thing? Here is an example:

Chunk 1:

Chunk 2:

Normally, I parse each of these chunks to JSON first in order to extract “choices”, but these chunks cannot be parsed now. This is the first time I see this happening. Anyone noticed the same thing? I’m guessing it might be related to the outage happening now?

Same here. My API calls return gibberish text. However, the web application chat interface works fine.

We face the same issue. Will this stick so should i implement a fix in my code or should we wait for a fix from openai team?

I can confirm, this probably started within the last 24h, all of our previously working code (working for months) has started having a problem with the first couple of chunks.

We are talking about streaming mode here, so:

	const chatResponse = await openai.createChatCompletion({
		stream: true
	}, {
		responseType: 'stream'

and then handling the stream (as per lots of example code - this has been working for a long time):

			chatStream.on('data', (chunk: Buffer) => {
				const messages = chunk
					.filter(m => m.length > 0)
				for (const message of messages) {
					... // skipped code
					const data = message.slice(dataMarker.length)
					... // skipped code
					const completion: CreateChatCompletionStreamResponse = JSON.parse(data)
					... // etc.

suddenly, the first chunk has a newline in it and results in two “messages”:

data: {"id":"chatcmpl-8J04wP33MFVY0pQoljcFiBsiBW3uC","object":"chat.completion.chunk","created":1699539226,"model":"gpt-4-0613","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-8J04wP33MFVY0pQoljcFiBsiBW3uC","object":"chat.completion.chunk","created":1699539226,"model":"gpt-4-0613"

the first parses to “” the second fails to parse as it is not valid JSON (it’s only half the object)

then the next chunk continues:


which does not start with the data marker 'data: ’ so is ignored by code (like How to use stream: true? · Issue #18 · openai/openai-node · GitHub - which is pretty much the code we have been running for months)

the next and subsequent chunks come in as usual eg:

data: {"id":"chatcmpl-8J04wP33MFVY0pQoljcFiBsiBW3uC","object":"chat.completion.chunk","created":1699539226,"model":"gpt-4-0613","choices":[{"index":0,"delta":{"content":" journey"},"finish_reason":null}]}

It’s easy enough to mitigate probably, but why has this happened? Will it be reverted? One other issue opened about it: createChatCompletion Regression - payload is split across 2 chunks

1 Like

#6 Something random so I can post similar

1 Like

The issue has not been resolved entirely.
The malformed streaming answers still persist.

1 Like

Is there any solution to this? I’m also experiencing this today

The issue resolved itself for me. Working correctly again.

1 Like

This exact symptom hit a few weeks ago, the answer: “you should wait for complete objects and fix them, it’s within the protocol rules”.

Whatever was causing it was temporary and seems to be resolved now.

I think that the point that the SSE code needs to buffer waiting for complete JSON objects is completely correct. I would imagine though that most people (myself included) are running code that is much like all the example code that openai has themselves provided, which does not do any such buffering.

I think the best thing to do is to use the latest API which exposes the stream result as an async iterable and you don’t have to worry about SSE responses or JSON parsing yourself… (that’s assuming that the npm package handles this case correctly itself…)

Yes, that’s right. I just incorporated the openai npm package.