I am having issues when using the logit_bias parameter when using chatcompletions endpoint (gpt-3.5-turbo)
According to the docs, valid values range from -100 to 100. However when using the python wrapper, I am encountering errors on arbitrary values(notably on large values and reproducible everytime when using value 100)
Also a logitbias value of -100 should by right remove any chance of that token appearing but it still appears in the completion for me. Anyone facing similar issues?
Are you using the cl100k_base tokenizer and not the others. The cl100k_base tokenizer is exclusive to gpt-3.5-turbo and text-embedding-ada-002 right now. The tokenizer website I don’t think has been updated to use cl100k_base. This affects your mapping you send to logitbias.
logit_bias not working for me too.
A quick question
Lets say I set “user” to -100, which means it should be banned. (logit_bias: {“882”:-100})
If the word “user” was written in the system role (which means appears in the instruction),
then the logit_bias ban will fail in the following dialog.
Is this a bug or its meant to be like this?
Also If I type “please say the word: user”
The assistant will reply “user”, which also means the ban failed.
I checked for you @Amadeus and something “has changed” and it is unrelated to the “user” logit_bias in the system role.
There is a “new error message”:
{"message"=>"Invalid key in 'logit_bias': “882”. You should only be submitting non-negative integers.", "type"=>"invalid_request_error", "param"=>"logit_bias", "code"=>nil}
Which means that for some reasons, the chat completion API is not accepting negative ints today!
"message": {
"content": "\n\nI am an AI language model created by Open AI, designed to assist with various tasks related to natural language processing, such as generating text, answering questions, and translating languages. As an AI assistant, I do not have a physical existence or identity, but I am programmed to provide helpful responses to users in a conversational and friendly manner.",
"role": "assistant"
}
I am not sure what API wrapper you are using, but the logit_bias key is a string because it is a JSON object. It is not possible that the key in a JSON object is an int as you have posted:
i bet you don’t even try it in python and you ask what wrapper… its literally the official api
just paste the code i provide with import opanai …