Hi all,
Despite logit_bias being a parameter in the Chat endpoint (as shown in docs: OpenAI API)
If I add a logit_bias using a syntax which works on the completion endpoint, I get a 500 error response from the chat endpoint.
I note that the Chat endpoint docs OMIT the logprobs parameter in the body request, so perhaps that inclusion of the logit_bias param in the docs is an error. Perhaps chat does not support logit_bias?
anyone know?
Thanks
It is supported @t.mcgregor and the more likely issue is some missed detail on your (the client) end.
logit-bias
See also, this OpenAI blog post:
https://help.openai.com/en/articles/5247780-using-logit-bias-to-define-token-probability
HTH
This may be a problem with the documentation. I can’t get logit_bias to work either.
It also makes sense considering lobprobs is not a valid parameter
Update: The discord channel has indicated that it does work - but you need the correct encoding. Chat GPT 3.5 uses cl100k_base for its tokens
Helpful! Thanks I’ll look into this token encoding
Hi,
So I have now done the correct encoding. I can confirm that /chat just does not allow “logit_bias” in the request body, it causes a 500
I have been using logit_bias on davinci endpoints because it works really well for classification where you want to ensure the output is only restricted to a set of specific tokens. I can’t get it work operate on /chat and give the docs omit logprobs and in addition the response body of /chat does not return logprobs than I can only assume that this is a type in the OpenAI docs for /chat and in fact logit_bias is not available
According to the docs, the token value is a string and the logit_bias is an integer, are you sending this?
# personality = "You are a snarky sarcastic chatbot."
personality = "You are a truthful factual chatbot."
payload = {
"model": "gpt-3.5-turbo",
"logit_bias": {"1234": -100}, # cl100k_base tokens only for turbo
"messages": [
{"role": "system", "content": personality},
{"role": "user", "content": "Who won the world series in 2020?"},
{"role": "user", "content": "Where was it played?"}
]
}
response = requests.post('https://api.openai.com/v1/chat/completions', headers=headers, json=payload)
print(response.json())
That is because this param is a JSON object so they are key-value pairs and the keys are strings.
Please post the exact JSON object you are sending as a param and I will test it for you.
Indeed, JSON, duh, so picky 
Only trick here is you ban/allow/influence tokens using the cl100k_base tokenizer, which you can grab quickly from a CoLab notebook after importing tiktoken. Hassle they don’t have cl100k_base on the tokenizer website yet 
Just tested this and it is slightly more complex than that because the API describes login_bias as a JSON object, but in fact it an “odd” way of doing things, in my view, but it works.
Without regard to the tokenizer, I simply selected a random token and tested for errors, not if the token was the token expected to effect the completion , which is another topic (which is the “what tokenizer to use” question). I only tested the format of the param object:
{"2435":0}
Results Above: Passed chat completion API , no errors.
[ {"2435":0,"2431":0} ]
Results Above: Failed chat completion API , error from API:
{message=>[{'2435': 0, '2431': 0}] is not of type 'object' - 'logit_bias', type=>invalid_request_error, param=>nil, code=>nil}
{"2435":0,"2431":0}
Results Above: Passed chat completion API , no errors.
This test only focused on the format of the login_bias param object, not the result of the param object on the completion.
There was no error (500 or otherwise) when the logit_bias param was formatted as single JSON object.
I expected an array of logit_bias entries to work, since that is the JSON standard of how arrays of objects should be formatted, but that test failed.
Formatting multiple logit_bias entries as a single JSON object worked fine, which was unexpected, as I guessed the array of JSON logit_bias JSON objects would be the proper format.
I repeated these tests multiple times with the same results.
Hope the helps.
Hi @t.mcgregor
Sorry, this is not correct. See tests above. The logit_bias param is accepted is the data is properly formatted (as OpenAI requires it):
Kindly post your exact logit_bias param formatted with Markdown triple backticks as follow:
```
# your logit_bias param JSON data object here
```
I will validate your data as a “single JSON object” and test for you.
I can confirm it too … use a single dictionary, not an array. My code [above] is actual code I wrote and it doesn’t give 500! (Logit_bias in ChatGPT endpoint causes error - #6 by curt.kennedy)
Yeah, but your code above is for a single logic_bias entry.
To fully confirm, add more than one logic_bias entry to your single JSON object and retest 
Ahh I see. I just assumed that since the docs said use a map, this means to use a dict with multiple entries.
Actually, the docs specify it must be a JSON object, which is generic to all programming languages,
The issue is that many people who use JSON regularly may think that the proper format is an “array of JSON objects” which is how a developer (as least me, at least) would expect things to work
However, from reading the API docs (in this case) they would not realize it is not really the “standard” JSON way when sending multiple objects with the same key value pairs, such as this. Anyway … it’s just my bias
Normally, a JSON object with the same key value pairs, at least in my mind, is as follows:
[
{"key" : "value"},
{"key" :"value"},
{"key" : "value"},
{"key" : "value"},
{"key" : "value"}
]
Not:
{"key" : "value", "key" :"value" ,"key" : "value" ,"key" : "value" , "key" :"value"}
Note, for "completeness, I tested sending the value as a string:
{"2435":"0","2431":"0"}
Does it fail?
Yes, here is the error:
{message=>All values in 'logit_bias' must be reals, but found a , type=>invalid_request_error, param=>logit_bias, code=>nil}
I think they mean “map each token to a bias value”, at least that is how it all “tests out”. Maybe I am wrong, but in this case I think the API doc needs a bit more clarification so others does waste time trying to figure out what we just figured out by brute force testing.
Or, devs can just visit this topic / thread and see the results from community testers.
Weird, I have the opposite expectation. At least from working with AWS for so many years, it is always
{"key1": "val1", "key2": "val2"}
OpenAI Docs:
AWS (inside a database entry from the console, just a dictionary, or JSON single object):
So “map” just made sense to me that it was a dictionary. I guess they should clear up the docs since it isn’t universal.
Agreed.
I am a Ruby person (obviously by user name, haha), I do not use the term “dictionary” referring to arrays and objects and so forth, at least for me, using the term “dictionary” is (overly) programming language specific.
JSON, is the “programming language neutral” term, and “map” seems to me (as mentioned) is referring to mapping a token (“key”) to a value (in this case a float / real).
Agree that dictionary is very Python centric. I have worked in many languages over the years, and most but not all tend to have some “map” data structure. Since my code was is Python, I called it a dictionary, not be be confused with map
…and it shows. You are well rounded and are a major contributor to the “signal” in the “signal-to-noise” ratio here, which seems to have been on the rise lately.
The community seems to be getting more focused on software engineering, developing software, code and coding errors. which is GREAT.
Personally, I want to thank everyone who is contributing to keep the software development signal-to-noise ratio high and who are discouraging posts (not related to software development) which drags down the SNR.
Hopefully, this topic will help future folks who have issues with the logic_bias param in the chat completion API method.
Thanks for getting down into the coding weeds with me, @curt.kennedy
If we could have a community rule which prohibits posting ChatGPT generated code which is not declared in the beginning as ChatGPT generated, that would be a big improvement, I think!
