High values in logit_bias in OpenAI chat completion endpoint params causes error

[raymonddavey]

This may be a problem with the documentation. I can’t get logit_bias to work either.

It also makes sense considering lobprobs is not a valid parameter

Update: The discord channel has indicated that it does work - but you need the correct encoding. Chat GPT 3.5 uses cl100k_base for its tokens

[t.mcgregor]

Helpful! Thanks I’ll look into this token encoding

[t.mcgregor]

Hi,

So I have now done the correct encoding. I can confirm that /chat just does not allow “logit_bias” in the request body, it causes a 500

I have been using logit_bias on davinci endpoints because it works really well for classification where you want to ensure the output is only restricted to a set of specific tokens. I can’t get it work operate on /chat and give the docs omit logprobs and in addition the response body of /chat does not return logprobs than I can only assume that this is a type in the OpenAI docs for /chat and in fact logit_bias is not available

[curt.kennedy]

According to the docs, the token value is a string and the logit_bias is an integer, are you sending this?

# personality = "You are a snarky sarcastic chatbot."
personality = "You are a truthful factual chatbot."

payload = {
    "model": "gpt-3.5-turbo",
    "logit_bias": {"1234": -100}, # cl100k_base tokens only for turbo
    "messages": [
        {"role": "system", "content": personality},
        {"role": "user", "content": "Who won the world series in 2020?"},
        {"role": "user", "content": "Where was it played?"}
    ]
}

response = requests.post('https://api.openai.com/v1/chat/completions', headers=headers, json=payload)

print(response.json())

[ruby_coder]

@curt.kennedy

According to the docs, the token value is a string

That is because this param is a JSON object so they are key-value pairs and the keys are strings.

[ruby_coder]

@t.mcgregor

If I add a logit_bias using a syntax which works on the completion endpoint, I get a 500 error response from the chat endpoint

@t.mcgregor

Please post the exact JSON object you are sending as a param and I will test it for you.

[curt.kennedy]

Indeed, JSON, duh, so picky

Only trick here is you ban/allow/influence tokens using the cl100k_base tokenizer, which you can grab quickly from a CoLab notebook after importing tiktoken. Hassle they don’t have cl100k_base on the tokenizer website yet

[ruby_coder]

@curt.kennedy

Indeed, JSON, duh, so picky

Just tested this and it is slightly more complex than that because the API describes login_bias as a JSON object, but in fact it an “odd” way of doing things, in my view, but it works.

Without regard to the tokenizer, I simply selected a random token and tested for errors, not if the token was the token expected to effect the completion , which is another topic (which is the “what tokenizer to use” question). I only tested the format of the param object:

Test 1: Single JSON Object

{"2435":0}

Results Above: Passed chat completion API , no errors.

Test 2: Array of JSON objects:

[ {"2435":0,"2431":0} ]

Results Above: Failed chat completion API , error from API:

{message=>[{'2435': 0, '2431': 0}] is not of type 'object' - 'logit_bias', type=>invalid_request_error, param=>nil, code=>nil}

Test 3: Strange Looking logit_bias single JSON object:

{"2435":0,"2431":0} 

Results Above: Passed chat completion API , no errors.

Summary

This test only focused on the format of the login_bias param object, not the result of the param object on the completion.

There was no error (500 or otherwise) when the logit_bias param was formatted as single JSON object.

I expected an array of logit_bias entries to work, since that is the JSON standard of how arrays of objects should be formatted, but that test failed.

Formatting multiple logit_bias entries as a single JSON object worked fine, which was unexpected, as I guessed the array of JSON logit_bias JSON objects would be the proper format.

I repeated these tests multiple times with the same results.

Hope the helps.

[ruby_coder]

@t.mcgregor

I can confirm that /chat just does not allow “logit_bias” in the request body, it causes a 500

Hi @t.mcgregor

Sorry, this is not correct. See tests above. The logit_bias param is accepted is the data is properly formatted (as OpenAI requires it):

Kindly post your exact logit_bias param formatted with Markdown triple backticks as follow:

```
# your logit_bias param JSON data object here
```

I will validate your data as a “single JSON object” and test for you.

[curt.kennedy]

I can confirm it too … use a single dictionary, not an array. My code [above] is actual code I wrote and it doesn’t give 500! (Logit_bias in ChatGPT endpoint causes error - #6 by curt.kennedy)

[ruby_coder]

@curt.kennedy

My code [above] is actual code I wrote and it doesn’t give 500!

Yeah, but your code above is for a single logic_bias entry.

To fully confirm, add more than one logic_bias entry to your single JSON object and retest

[curt.kennedy]

Ahh I see. I just assumed that since the docs said use a map, this means to use a dict with multiple entries.

[ruby_coder]

@curt.kennedy

this means to use a dict with multiple entries.

Actually, the docs specify it must be a JSON object, which is generic to all programming languages,

The issue is that many people who use JSON regularly may think that the proper format is an “array of JSON objects” which is how a developer (as least me, at least) would expect things to work

However, from reading the API docs (in this case) they would not realize it is not really the “standard” JSON way when sending multiple objects with the same key value pairs, such as this. Anyway … it’s just my bias

Normally, a JSON object with the same key value pairs, at least in my mind, is as follows:

JSON as an array of objects

[
  {"key" : "value"}, 
  {"key" :"value"}, 
  {"key" : "value"}, 
  {"key" : "value"}, 
  {"key" : "value"}
]

Not:

JSON as a single object

 {"key" : "value", "key" :"value" ,"key" : "value" ,"key" : "value" , "key" :"value"}

Note, for "completeness, I tested sending the value as a string:

{"2435":"0","2431":"0"}

Does it fail?

Yes, here is the error:

{message=>All values in 'logit_bias' must be reals, but found a , type=>invalid_request_error, param=>logit_bias, code=>nil}

Interesting, I thought the values were all ints, but the API specifies the values are floats / reals.

[ruby_coder]

@curt.kennedy

said use a map

I think they mean “map each token to a bias value”, at least that is how it all “tests out”. Maybe I am wrong, but in this case I think the API doc needs a bit more clarification so others does waste time trying to figure out what we just figured out by brute force testing.

Or, devs can just visit this topic / thread and see the results from community testers.

Appendix: Sample Test Results and Error Message:

[curt.kennedy]

Weird, I have the opposite expectation. At least from working with AWS for so many years, it is always

{"key1": "val1", "key2": "val2"}

OpenAI Docs:

AWS (inside a database entry from the console, just a dictionary, or JSON single object):

So “map” just made sense to me that it was a dictionary. I guess they should clear up the docs since it isn’t universal.

[ruby_coder]

@curt.kennedy

So “map” just made sense to me that it was a dictionary. I guess they should clear up the docs since it isn’t universal.

Agreed.

I am a Ruby person (obviously by user name, haha), I do not use the term “dictionary” referring to arrays and objects and so forth, at least for me, using the term “dictionary” is (overly) programming language specific.

JSON, is the “programming language neutral” term, and “map” seems to me (as mentioned) is referring to mapping a token (“key”) to a value (in this case a float / real).

[curt.kennedy]

Agree that dictionary is very Python centric. I have worked in many languages over the years, and most but not all tend to have some “map” data structure. Since my code was is Python, I called it a dictionary, not be be confused with map

[ruby_coder]

@curt.kennedy

have worked in many languages over the years

…and it shows. You are well rounded and are a major contributor to the “signal” in the “signal-to-noise” ratio here, which seems to have been on the rise lately.

The community seems to be getting more focused on software engineering, developing software, code and coding errors. which is GREAT.

Personally, I want to thank everyone who is contributing to keep the software development signal-to-noise ratio high and who are discouraging posts (not related to software development) which drags down the SNR.

Hopefully, this topic will help future folks who have issues with the logic_bias param in the chat completion API method.

Thanks for getting down into the coding weeds with me, @curt.kennedy

End Note:

If we could have a community rule which prohibits posting ChatGPT generated code which is not declared in the beginning as ChatGPT generated, that would be a big improvement, I think!

[t.mcgregor]

Hi all, thanks for all the responses. @ruby_coder my JSON object was all good.

What I found now is that
“logit_bias”: {“8505”: 0, “3919”: 0} WORKS
“logit_bias”: {“8505”: 20, “3919”: 20} WORKS

but “logit_bias”: {“8505”: 100, “3919”: 100} DOES NOT WORK, despite that the docs saying that 100 and -100 are the valid min/max values

Anyway a value of 20 will suffice as it will tend to ensure I only get my binary outcome and not some other random text

Thanks again

[ruby_coder]

@t.mcgregor

{“8505”: 100, “3919”: 100}

Lets test, shall we:

Test I: 100, 100

{"2435":100,"2431":100}

Results:

{"message"=>"The server had an error processing your request. Sorry about that! You can retry your request, or contact us through our help center at help.openai.com if you keep seeing this error. (Please include the request ID 40f0f15888d8250c244d047cd07427d7 in your email.)", "type"=>"server_error", "param"=>nil, "code"=>nil}

Test 2: -100, -100

{"2435":-100,"2431":-100}

Results: Success

Test 3: 99.999, 99.999

{"2435":99.999"2431":99.999}

Results:

{"message"=>"The server had an error processing your request. Sorry about that! You can retry your request, or contact us through our help center at help.openai.com if you keep seeing this error. (Please include the request ID 0258bec6dd04fcecdc392b487feaa5ad in your email.)", "type"=>"server_error", "param"=>nil, "code"=>nil}

Test 4: 99, 99

{"2435":99"2431":99}

Results:

 {"message"=>"The server had an error processing your request. Sorry about that! You can retry your request, or contact us through our help center at help.openai.com if you keep seeing this error. (Please include the request ID 3e23a4f3b34dd352aad196f76c101b26 in your email.)", "type"=>"server_error", "param"=>nil, "code"=>nil}

Test 5: 20, 20

{"2435":20"2431":20}

Results: Success

Summary:

Does appear to be some OpenAI API issue related to very high logit_bias values.

However, I’m not going to keep testing values today to try to mine what are the “working limits” of logit_bias values