Updated (not a bug) : BUG in new endpoint - Streaming not working - Hanging connection

API Feedback
1

I could be mistaken - but I am reasonably confident the streaming option is not working with the new chat endpoint. (Has anyone got it working)

When I make a blocking call (streaming=false), I get a message. (id removed for privacy reasons, also manually spaced to highlight the role response)

{"id":"","object":"chat.completion","created":1677718504,"model":"gpt-3.5-turbo-0301","usage":{"prompt_tokens":8,"completion_tokens":11,"total_tokens":19}, 

"choices":[{"message":
{"role":"assistant","content":"Hello! How can I assist you today?"},

"finish_reason":"stop","index":0}]}

When I make an async (streaming=true) call, I also get a valid response - but the assistant text is blank.

{"id":"","object":"chat.completion.chunk","created":1677718890,"model":"gpt-3.5-turbo-0301",

"choices":[{"delta":
{"role":"assistant"},

"index":0,"finish_reason":null}]}

Notice the choices is a delta instead of a message. The next part of the async loop returns a blank string (no content - but the connection is still up and running with the API - Done is never received)

It never sends the [DONE] tag and gets stuck in a timeout loop (possibly hanging the connections at OpenAI’s end too)

This is for the same request. The only difference is the streaming true/false setting.

2 Likes
2

Confirmed - not a bug. The behavior is a bit different. I will document when I finish fixing my end

{'role': 'assistant'}{'content': '\n\n'}{'content': 'Hello'}{'content': ' there'}{'content': '!'}{'content': ' How'}{'content': ' can'}{'content': ' I'}{'content': ' assist'}{'content': ' you'}{'content': ' today'}

The snippet above is not a full response.

But it turns out that the role comes once, and then the content comes multiple times. You may also get blank responses midstream.

2 Likes
3

Here is the raw stream from the “Hello!” example

Each row is an async response chunk

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:“\n\n”},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:“Hello”},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" there"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:“!”},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" How"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" can"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" I"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" assist"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" you"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:" today"},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{“content”:“?”},“index”:0,“finish_reason”:null}]}

data: {“id”:“”,“object”:“chat.completion.chunk”,“created”:1677725062,“model”:“gpt-3.5-turbo-0301”,“choices”:[{“delta”:{},“index”:0,“finish_reason”:“stop”}]}

data: [DONE]

1 Like
4

I removed the ID values and the object is chat.completion.chunk and the result is in the

choices[0]->delta->content property

1 Like
5

I’ve heard on Discord you might get blank in front too… Thanks for posting!

1 Like
6

The blank made me think it wasn’t working.

It looks like it sends new lines at the start (maybe). I’m still bedding it in.

Streaming is not documented yet.

1 Like
7

I’ve seen some python and node (??) examples on Discord i think…but I’m sure you can figure it out!

Thanks again.

8

I got similar situation today. All of APIs works well at synchronous mode. And then when I set streaming & asynchronous mode, The API connection hangs and then time out.
Anyone can give me a hint? My code worked before upgrading to openai-python 0.27 version yesterday. I got confused.