Chatgpt api (openai-node v4.26.0) stream issue with gpt-4 models

API Bugs
1

When using the chatgpt API with stream on version 4.26.0 and selecting models like gpt-4, gpt-4-turbo-preview, or gpt-4-0125-preview, an issue arises where the output is duplicated. For example:

  • Prompt: hello Output: Hello! How How can I assist assist you today?

This problem is consistent across gpt-4 models. Interestingly, gpt-3.5-turbo functions correctly without the duplication issue.

The same phenomenon occurs when using Japanese prompts:

  • Prompt: こんにちは Output: こんにちは！ 何何かお手手伝いできることがありありますか？

This behavior is consistent across English and Japanese inputs on the specified models.

The observed issue with the functionality occurred post-January 31, 2024, and was working without problems until then. However, starting from February 1, 2024, the described phenomenon became apparent.

3 Likes
2

Welcome to the forum!

Have you tried a prompt other than just “hello”… ?

3

Yes, the observed phenomenon also happens with other prompts.

4

What temperature settings are you using?

What’s the system prompt?

Is it only one word prompts?

1 Like
5

  1. Current temp is 0.8. I tried adjusting the temperature setting from 0.1 to 0.8, but it did not make a difference.

  2. {
    “role”: “system”,
    “content”: “Please answer the user’s question.”,
    }
    While the current system prompt is kept simple, it yields the same results as the more complex ones we actually use. Similarly, the same phenomenon occurs in the case of Japanese.

  3. Prompt: “Tell me about open.ai”
    Response: "OpenAI is is an artificial intelligence intelligence research lab made made up of both both for-profit and and non-profit arms arms. It was was established in December December 20155. The organization organization aims to ensure ensure that artificial general intelligence (AGI) benefits benefits all of humanity humanity. They plan plan to build safe and beneficial AG AGI directly, but are also also committed to aiding aiding others in achieving achieving this outcome.

.

OpenAI follows follows a set of of key principles… They are broadly broadly distributed in benefits benefits, long-term-term safety oriented, technically leadership focused focused, and cooperative cooperative in orientation… They are committed committed to assisting value value-aligned, safety safety-conscious projects that that come close to to building AGII before they do do. They are are also dedicated to to doing the research research required to make make AGI safe safe and driving its its broad adoption across across the AI community community.

OpenAIAI has been involved involved in various high high-profile research projects and advancements in in the field of of AI, including including the development of of AI models like like GPT–3, which which is capable of of generating human-like-like text based on on the input it it’s given…"

1 Like
6

Some other users are reporting similar issues. While they are dealing with omitted tokens, it appears that you’re encountering duplicates. :thinking:

2 Likes
7

I tested using openai-node v4.26.0, running basic node.js express server and I cannot replicate the problem in my end.

{
  messages: [
    {
      role: 'system',
      content: 'You are a helpful personal assistant.\n' +
        '# Tools\n' +
        'You have the following tools that you can invoke based on the user inquiry.\n' +
        '- get_weather, when the user wants to know the weather forecast from a given location and date.\n' +
        '- get_stock_moving_average_trend, when the user wants to get the moving average trend of a stock.\n' +
        'Today is: Fri Feb 02 2024 14:04:26 GMT+0900 (Japan Standard Time)'
    },
    { role: 'user', content: 'こんにちは' }
  ],
  model: 'gpt-4-0125-preview',
  temperature: 0.8,
  max_tokens: 2048,
  stream: true,
  tools: [
    { type: 'function', function: [Object] },
    { type: 'function', function: [Object] }
  ]
}

{“index”:0,“delta”:{“role”:“assistant”,“content”:“”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“こんにちは”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“！”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“ど”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“の”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“よ”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“う”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“に”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“お”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“手”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“伝”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“い”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“で”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“き”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“ます”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“か”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{“content”:“？”},“logprobs”:null,“finish_reason”:null}
{“index”:0,“delta”:{},“logprobs”:null,“finish_reason”:“stop”}

1 Like
8

Thank you for the test. If there are no issues in other environments, there might be any problem with my settings, so I will review them again. However, considering that there are no problems when using the “gpt-3.5-turbo” model, there could be other possibilities to explore.

9

What is your environment running it (both backend and frontend)? Perhaps we can determine which is causing the problem if others with same setup can check.

10

I am operating in the frontend environment, implementing Server-Sent Events (SSE) for React Native using the react-native-sse.

11

We have the exact same error, and followed the documentation about implementing stream. The only solution we found was to disable stream.

12

did you ever get an opportunity to log deltas like in this post?

13

English:

 {"delta": {"content": "", "role": "assistant"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "", "role": "assistant"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "Hello"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "!"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": " How"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": " How"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": " can"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": " I"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": " assist"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": " assist"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": " you"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": " today"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "?"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {}, "finish_reason": "stop", "index": 0, "logprobs": null}

Japanese:

{"delta": {"content": "", "role": "assistant"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "", "role": "assistant"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "こんにちは"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "、"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "何"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "何"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "か"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "お"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "困"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "困"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "り"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "の"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "こ"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "こ"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "と"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "は"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "あり"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "あり"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "ます"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "か"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "？"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {"content": "？"}, "finish_reason": null, "index": 0, "logprobs": null}
 {"delta": {}, "finish_reason": "stop", "index": 0, "logprobs": null}
14

yeah that looks pretty bad, but I can’t reproduce it either anymore.

if you want to isolate that it’s an API issue and not a lib issue you can run a jupyter cell or something :confused:

test code 

import requests
import json
import os

# Ensure you have your OpenAI API key set in the environment variables
openai_api_key = os.getenv("OPENAI_API_KEY")
if openai_api_key is None:
    raise ValueError("OpenAI API key is not set in environment variables.")

url = "https://api.openai.com/v1/chat/completions"

headers = {
    "Content-Type": "application/json",
    "Authorization": f"Bearer {openai_api_key}"
}

data = {
    "model": "gpt-4-0125-preview",  # Keep only this model
    "temperature": 1, 
    "max_tokens": 1000,
    "seed": 20,
    #"logit_bias": {1734:-100},
    "messages": [
        {
            "role": "user",
            "content": "こんにちは"
        }
    ],
    "stream": True,  # Changed to True to enable streaming
}

response = requests.post(url, headers=headers, json=data, stream=True)

messageout = ''
finish_reason = 'unknown'
if response.status_code == 200:
    for line in response.iter_lines():
        if line:
            decoded_line = line.decode('utf-8')
            print(decoded_line)
            # Check if the stream is done
            if '[DONE]' in decoded_line:
                # print("\nStream ended by the server.")
                break
            json_str = decoded_line[len('data: '):]
            try:
                json_response = json.loads(json_str)
                if json_response['choices'][0]['finish_reason']:
                    finish_reason = json_response['choices'][0]['finish_reason']
                delta = json_response['choices'][0]['delta']
                if 'content' in delta and delta['content']:
                    messageout += delta['content']
                    #print(delta['content'], end='', flush=True) 
                    pass
            except json.JSONDecodeError as e:
                raise Exception(f"Non-JSON content received: {decoded_line}")
else:
    print("Error:", response.status_code, response.text)
    
print('\n\n' + messageout)
print('\n\nfinish_reason: '+finish_reason)

(fwiw we are using axios in prod)

1 Like
15

We are also experiencing the exact same problem, in React Native using EventSource from ‘react-native-sse’ . Seems like it could be related to that library, will see if there’s an update.

 LOG  [2024-02-14T19:37:48.392Z] content:  {"delta": {"content": "", "role": "assistant"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:48.503Z] content:  {"delta": {"content": "G"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:48.507Z] content:  {"delta": {"content": "lad"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:48.638Z] content:  {"delta": {"content": "lad"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:48.641Z] content:  {"delta": {"content": " to"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:48.648Z] content:  {"delta": {"content": " hear"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:48.779Z] content:  {"delta": {"content": " hear"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:48.781Z] content:  {"delta": {"content": " it"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:48.786Z] content:  {"delta": {"content": "'s"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:48.888Z] content:  {"delta": {"content": "'s"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:48.891Z] content:  {"delta": {"content": " been"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:48.895Z] content:  {"delta": {"content": " okay"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:49.015Z] content:  {"delta": {"content": " okay"}, "finish_reason": null, "index": 0, "logprobs": null}
 LOG  [2024-02-14T19:37:49.018Z] content:  {"delta": {"content": "."}, "finish_reason": null, "index": 0, "logprobs": null}

@Pan0201 - did you find a solution?

16

By the way, updating to the latest “react-native-sse”, 1.2.0 did not fix the issue.

As mentioned above, gpt-3.5-turbo works fine.

17

I’d like to start by warning that unless your chat interface is meant to be a self-hosted private solution the business logic should not be handled in React.

When I hear about duplications in React I always want to know: are you monitoring the profiler? It’s pretty common and easy to accidentally cause React to excessively render more times than anticipated.

Then, I also wonder, how are you processing the data? React asynchronously batches updates. You can cause this duplication issue by running a reduce function to output text. It will duplicate.

Regardless, you should view the thread in the playground first to determine if it’s your end, or openai.

18

Here’s another log with the Debug option of EventSource enabled:

[EventSource] Will open new connection in 50 ms.
 DEBUG  [EventSource][onreadystatechange] ReadyState: HEADERS_RECEIVED(2), status: 200
 DEBUG  [EventSource][onreadystatechange] ReadyState: LOADING(3), status: 200
 LOG  [2024-02-14T20:08:03.521Z] [AICompanion]  {"group":"AICompanion"} Opened SSE connection.
 DEBUG  [EventSource][onreadystatechange][OPEN] Connection opened.
 LOG  [2024-02-14T20:08:03.524Z] content:  {"content": "", "role": "assistant"}
 DEBUG  [EventSource][onreadystatechange] ReadyState: LOADING(3), status: 200
 LOG  [2024-02-14T20:08:03.588Z] content:  {"content": "I"}
 LOG  [2024-02-14T20:08:03.591Z] content:  {"content": "'m"}
 DEBUG  [EventSource][onreadystatechange] ReadyState: LOADING(3), status: 200
 LOG  [2024-02-14T20:08:03.671Z] content:  {"content": "'m"}
 LOG  [2024-02-14T20:08:03.673Z] content:  {"content": " glad"}
 LOG  [2024-02-14T20:08:03.675Z] content:  {"content": " to"}
 DEBUG  [EventSource][onreadystatechange] ReadyState: LOADING(3), status: 200
 LOG  [2024-02-14T20:08:03.886Z] content:  {"content": " to"}
 LOG  [2024-02-14T20:08:03.894Z] content:  {"content": " hear"}
 LOG  [2024-02-14T20:08:03.898Z] content:  {"content": " you"}
 DEBUG  [EventSource][onreadystatechange] ReadyState: LOADING(3), status: 200
 LOG  [2024-02-14T20:08:03.903Z] content:  {"content": " you"}
 LOG  [2024-02-14T20:08:03.905Z] content:  {"content": "'re"}
 LOG  [2024-02-14T20:08:03.909Z] content:  {"content": " okay"}
 DEBUG  [EventSource][onreadystatechange] ReadyState: LOADING(3), status: 200
 LOG  [2024-02-14T20:08:03.926Z] content:  {"content": " okay"}
 LOG  [2024-02-14T20:08:03.931Z] content:  {"content": "!"}

It looks like the duplication is happening at the boundary, where the last token is being repeated in the next “ReadyStateChange”.

I’m not saying it’s an OpenAI issue, but 3.5 works fine and 4 and 4-turbo-preview have the issues so I’m looking for some insights on how to fix this.

2 Likes
19

I identified it as an issue on the react-native-sse side and was able to fix it. Thank you all for your collaboration. I left a comment on react-native-sse, please refer to it.

1 Like