Stop=['\n'] in "gpt-3.5-turbo" leads to 500 error

[ruby_coder]

@saintkyumi

As @atty-openai mentioned, newline stop itself causes error in the backend side.

That may well be, but I have tried repeatedly to confirm this error, including just now, and I cannot get an newline to cause any error.

Example Setup using newline as stop

Example completion (works fine)

FWIW

I have tried many (at least 10) times with newlines as stops and turbo as the model and have never experienced an error. However, in practice I do not use newlines as stops.

HTH

[semlar]

@atty-openai

Hi folks, I’m an engineer at OpenAI. This is a known bug and something we’re working on fixing. It is slightly complicated due to the way we are parsing the model’s response into the message format (the parsing logic ascribes some special meaning to new lines, so if you specify a new line as a stop token, it interferes with parsing). We have an idea of how to fix this, will share an update here once it’s fixed.

This bug is not limited to newline characters, it happens on the ChatCompletion endpoint whenever the first token generated is a stop token; I ran into it last week but wasn’t sure how to report it.

It can be easily reproduced by setting the logit_bias of any stop token to 100:

openai.ChatCompletion.create(
  model = "gpt-3.5-turbo",
  messages = [{"role": "user", "content": "hello"}],
  temperature = 0,
  logit_bias = {12340: 100},
  stop = ["!!!"],
)

[ruby_coder]

@semlar

It can be easily reproduced by setting the logit_bias of any stop token to 100:

And why would anyone earnestly writing API calls submit a logit_bias for a stop?

[semlar]

@ruby_coder

And why would anyone earnestly writing API calls submit a logit_bias for a stop?

The bug occurs when a stop token is generated at the start of a response, logit_bias is used here to demonstrate how to consistently reproduce the error to help OpenAI track down what’s causing it on their end.

[MikePFrank]

Thanks for the thread; I set my stop sequence to something else and it fixed the problem. But now
I’m getting similar errors with gpt-4, but they seem to be frequent no matter what I set stop to. I thought it might be due to rate limits, but I don’t always get the RateLimitError. And I don’t have this problem in every application. Pretty baffling.

[ruby_coder]

@semlar

logit_bias = {12340: 100},

We had a fairly comprehensive discussion on this logit_bias == 100 in another topic, and our testing showed that the bug was because the API has trouble with high values of 'logit_bias`, stop or not.

Example 1: Hello World, Stop “####”, Logit Bias World, 0

Example 1: Results OK

Example 2: Hello World, Stop “####”, Logit Bias World, 100

Example 2: Results: Pesky Timeout Error Again:

This happens today with large values of logit_bias so I’ll table these tests for now until the models are preforming better.

Note: As this is a lab, I want to see the timeout errors, so I have no internal timeout or retry code active (not production).

Example 3 Results: Hello World, Stop “####”, Logit Bias World, -50 (OK)

Example 4 Results: Hello World, Stop “####”, Logit Bias World, 50

This is consistent with earlier results, where large positive logit_bias values are problematic.

[semlar]

You are correct, evidently I just happened to uncover a second bug with logit_bias while trying to use it to demonstrate the first one. What can I say, I’m good at breaking things.

However, since I do not use a line break as a stop token or logit_bias in the code that keeps triggering this error I’ve done some more testing and it looks like any stop token that simply begins with a line break will cause this.

Here’s a revised example to reproduce the bug, which is still present as of today:

openai.ChatCompletion.create(
  model = "gpt-3.5-turbo",
  messages = [
    {"role": "user", "content": "Marco"},
  ],
  max_tokens = 5,
  temperature = 0,
  stop = ["\nPolo"],
)

[ruby_coder]

@semlar

I’m good at breaking things.

I’m good at analysis, coding, building, debugging and testing… so we match

Here is your example above, tested with your same params:

The Chat Setup

The Results: No Error (Success)

HTH

See Also, Appendix:

Same Params, but Using Stop Array:

Results: Success

[semlar]

Try using the API directly instead of through a web interface, if I had to guess the input for the stop token is escaping the backslash and treating it as a literal instead of a newline character.

[ruby_coder]

OK !!!

Will test from the command line in the Rails console tomorrow.

[colemanhindes]

@ruby_coder i appreciate you looking into this but it seems like the issue is with the python bindings which seem to behave differently

[ruby_coder]

@colemanhindes

you looking into this but it seems like the issue is with the python bindings which seem to behave differently

Yeah, I guessed the same. The Ruby API wrapper works very well and I do not use the Python wrapper.

[imgly]

I’m reproducing the same issue with newline and “```” stops with “http post” nushell command (equivalent to curl).
It seems to produce error 500 only if the stop is detected in the response.

[saintkyumi]

it’s now fixed.
see newline char(`\n`) in stop param leads to 500 error in gpt-3.5-turbo · Issue #290 · openai/openai-python · GitHub