Updated (not a bug) : BUG in new endpoint - Streaming not working - Hanging connection

I could be mistaken - but I am reasonably confident the streaming option is not working with the new chat endpoint. (Has anyone got it working)

When I make a blocking call (streaming=false), I get a message. (id removed for privacy reasons, also manually spaced to highlight the role response)


{"role":"assistant","content":"Hello! How can I assist you today?"},


When I make an async (streaming=true) call, I also get a valid response - but the assistant text is blank.




Notice the choices is a delta instead of a message. The next part of the async loop returns a blank string (no content - but the connection is still up and running with the API - Done is never received)

It never sends the [DONE] tag and gets stuck in a timeout loop (possibly hanging the connections at OpenAI’s end too)

This is for the same request. The only difference is the streaming true/false setting.


Confirmed - not a bug. The behavior is a bit different. I will document when I finish fixing my end

{'role': 'assistant'}{'content': '\n\n'}{'content': 'Hello'}{'content': ' there'}{'content': '!'}{'content': ' How'}{'content': ' can'}{'content': ' I'}{'content': ' assist'}{'content': ' you'}{'content': ' today'}

The snippet above is not a full response.

But it turns out that the role comes once, and then the content comes multiple times. You may also get blank responses midstream.


Here is the raw stream from the “Hello!” example

Each row is an async response chunk

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:“\n\n”},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:“Hello”},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" there"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:“!”},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" How"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" can"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" I"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" assist"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" you"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" today"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:“?”},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{},“index”:0,“finish_reason”:“stop”}]}

data: [DONE]

1 Like

I removed the ID values and the object is chat.completion.chunk and the result is in the

choices[0]->delta->content property

1 Like

I’ve heard on Discord you might get blank in front too… Thanks for posting!

1 Like

The blank made me think it wasn’t working.

It looks like it sends new lines at the start (maybe). I’m still bedding it in.

Streaming is not documented yet.

1 Like

I’ve seen some python and node (??) examples on Discord i think…but I’m sure you can figure it out!

Thanks again.

I got similar situation today. All of APIs works well at synchronous mode. And then when I set streaming & asynchronous mode, The API connection hangs and then time out.
Anyone can give me a hint? My code worked before upgrading to openai-python 0.27 version yesterday. I got confused.