BUG: Streaming packets changed

atty-openai · October 30, 2023, 10:18pm

It looks like this library is not parsing SSE outputs correctly. It assumes that when the stream content is split by \n\ndata:, each line is valid JSON. However, SSE clients should buffer output until they hit a separator character and only then parse the chunk into JSON.

github.com

LagPixelLOL/ChatGPTCLIBot/blob/583ac859d09e020d8ebfd9018d2789f32cdb72b6/main/interface/network/Network.cpp#L38-L89


      
          void handle_streamed_response(const std::vector<char>& raw_vec, const bool& is_new_api,
                                        const std::function<void(const std::string& streamed_response)>& stream_callback) {
              std::vector<std::string> split_str;
              boost::split_regex(split_str, std::string(raw_vec.begin(), raw_vec.end()), boost::regex("\\n\\ndata: *"));
              for (auto& s : split_str) {
                  if (boost::starts_with(s, "data:")) {
                      s.erase(0, 5);
                  }
                  //Remove leading and trailing whitespaces and new lines.
                  s.erase(s.begin(), std::find_if(s.begin(), s.end(), [](unsigned char c){return !std::isspace(c) && c != '\n';}));
                  s.erase(std::find_if(s.rbegin(), s.rend(), [](unsigned char c){return !std::isspace(c) && c != '\n';}).base(), s.end());
                  if (s.empty()) {
                      continue;
                  }
                  if (s == "[DONE]") {
                      break;
                  }
                  try {
                      nlohmann::json j = nlohmann::json::parse(s);
                      std::string response;

This file has been truncated. show original

rklatman · October 31, 2023, 12:03am

Regardless the way its streamed has changed today or this weekend.
and sending partial JSON packets between 2 packets is just annoying

kjordan · October 31, 2023, 3:49am

If you are using Javascript/Typescript, make sure to use GitHub - rexxars/eventsource-parser: Streaming, source-agnostic EventSource/Server-Sent Events parser for buffering the chunk data.

atty-openai · November 1, 2023, 8:37pm

Hi all, upon further investigation we have determined that there was indeed a bug that affected a subset of streaming responses on v1/chat/completions from 10:19 AM PT to 2:41 PM PT. This was fixed and impact stopped at 2:41 PM PT. Apologies for the downtime, and we are fixing the root cause as a follow-up.

Topic		Replies	Views
Malformed streaming answers from GPT-4 completions API lately API	11	2343	November 13, 2023
Receiving "{\"rate_limit_usage\": {\ in completion stream Bugs bug	42	5833	January 3, 2024
Streaming events returned bunched up Bugs	8	226	July 19, 2024
Was there an intentional change to the streaming responses? (multiple chunks in stream event) API bug , api , streaming	9	2737	July 23, 2024
GPT-4 model, unexpected returns in stream mode API gpt-4 , api	10	3335	December 16, 2023

BUG: Streaming packets changed

Related topics