GPT-4 model, unexpected returns in stream mode

API
,
1

When I use the GPT-4 model in stream mode, the returned result is not a complete object. Why is that?

e.g: data: {"id":"chatcmpl-88fnMltq7wZuouVuosLcOcJqHRODS","object":"chat.completion.chunk","crea

1 Like
2

Let me guess, it is node.js and you aren’t reassembling the chunked network stream of pieces into complete objects.

4

Yes, I’m using Node.js. How should I modify my code?

5

It may be related to this issue:

I have not tested on GPT4 - But GPT3.5 is cutting off the stream unusually.

It might be best to wait until this is resolved to see if it is related before changing too much code

1 Like
6

GitHub - rexxars/eventsource-parser: Streaming, source-agnostic EventSource/Server-Sent Events parser Try this module, it consumes raw chunk data and emits events when complete data is parsed.

7

I encountered the same issue today. Only GPT-4 with stream response raises this error. GPT-3.5 works fine.
I compared the two response messages and found the root cause - the GPT-4 response has an additional incomplete message:

data: "{\"rate_limit_usage\": {\

It leads this issue.

3.5 stream response sample as below

data: {"id":"chatcmpl-88jQwKBQWAfggwDrWG0LjynlY9Auu","object":"chat.completion.chunk","created":1697091962,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-88jQwKBQWAfggwDrWG0LjynlY9Auu","object":"chat.completion.chunk","created":1697091962,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jQwKBQWAfggwDrWG0LjynlY9Auu","object":"chat.completion.chunk","created":1697091962,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jQwKBQWAfggwDrWG0LjynlY9Auu","object":"chat.completion.chunk","created":1697091962,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" How"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jQwKBQWAfggwDrWG0LjynlY9Auu","object":"chat.completion.chunk","created":1697091962,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" can"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jQwKBQWAfggwDrWG0LjynlY9Auu","object":"chat.completion.chunk","created":1697091962,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" I"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jQwKBQWAfggwDrWG0LjynlY9Auu","object":"chat.completion.chunk","created":1697091962,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" assist"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jQwKBQWAfggwDrWG0LjynlY9Auu","object":"chat.completion.chunk","created":1697091962,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" you"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jQwKBQWAfggwDrWG0LjynlY9Auu","object":"chat.completion.chunk","created":1697091962,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" today"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jQwKBQWAfggwDrWG0LjynlY9Auu","object":"chat.completion.chunk","created":1697091962,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"?"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jQwKBQWAfggwDrWG0LjynlY9Auu","object":"chat.completion.chunk","created":1697091962,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

GPT4 stream response as below

data: {"id":"chatcmpl-88jRh4oconzcWrsEKTZONQKIQgwtQ","object":"chat.completion.chunk","created":1697092009,"model":"gpt-4-0613","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-88jRh4oconzcWrsEKTZONQKIQgwtQ","object":"chat.completion.chunk","created":1697092009,"model":"gpt-4-0613","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jRh4oconzcWrsEKTZONQKIQgwtQ","object":"chat.completion.chunk","created":1697092009,"model":"gpt-4-0613","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jRh4oconzcWrsEKTZONQKIQgwtQ","object":"chat.completion.chunk","created":1697092009,"model":"gpt-4-0613","choices":[{"index":0,"delta":{"content":" How"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jRh4oconzcWrsEKTZONQKIQgwtQ","object":"chat.completion.chunk","created":1697092009,"model":"gpt-4-0613","choices":[{"index":0,"delta":{"content":" can"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jRh4oconzcWrsEKTZONQKIQgwtQ","object":"chat.completion.chunk","created":1697092009,"model":"gpt-4-0613","choices":[{"index":0,"delta":{"content":" I"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jRh4oconzcWrsEKTZONQKIQgwtQ","object":"chat.completion.chunk","created":1697092009,"model":"gpt-4-0613","choices":[{"index":0,"delta":{"content":" assist"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jRh4oconzcWrsEKTZONQKIQgwtQ","object":"chat.completion.chunk","created":1697092009,"model":"gpt-4-0613","choices":[{"index":0,"delta":{"content":" you"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jRh4oconzcWrsEKTZONQKIQgwtQ","object":"chat.completion.chunk","created":1697092009,"model":"gpt-4-0613","choices":[{"index":0,"delta":{"content":" today"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jRh4oconzcWrsEKTZONQKIQgwtQ","object":"chat.completion.chunk","created":1697092009,"model":"gpt-4-0613","choices":[{"index":0,"delta":{"content":"?"},"finish_reason":null}]}

data: {"id":"chatcmpl-88jRh4oconzcWrsEKTZONQKIQgwtQ","object":"chat.completion.chunk","created":1697092009,"model":"gpt-4-0613","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: "{\"rate_limit_usage\": {\
8

so… to handle this in a chunk parsing loop when getting the response directly and stripping "data: " instead of getting a python error thrown when reading from the generator, something like:

if response_chunk[:1] == "\"{\\\"rate_limit_usage\\\": "[:1]:
    # print("\ngot a stupid rate error or a quote mark")
    continue
# your normal chunk processing
9

Yes, fix it is easy. I just want to report this and tell them there is a issue.

10

This also must be the reason behind the symptom in the playground where after the streamed response is completed, the button doesn’t change from “cancel” back to “submit”.

11

The GPT4 stream response back to normal. Good job.