Logitbias not working properly for chatcompletions api

[curt.kennedy]

And for contrast, to prove they are different: (gpt2 is what GPT-3 uses)

To round it out, p50k_base is what davinci-003/002 uses:

OK done. Both different than what is needed

[Amadeus]

logit_bias not working for me too.
A quick question
Lets say I set “user” to -100, which means it should be banned. (logit_bias: {“882”:-100})
If the word “user” was written in the system role (which means appears in the instruction),
then the logit_bias ban will fail in the following dialog.
Is this a bug or its meant to be like this?
Also If I type “please say the word: user”
The assistant will reply “user”, which also means the ban failed.

[ruby_coder]

@Amadeus

Is this a bug or its meant to be like this?

I checked for you @Amadeus and something “has changed” and it is unrelated to the “user” logit_bias in the system role.

There is a “new error message”:

 {"message"=>"Invalid key in 'logit_bias': “882”. You should only be submitting non-negative integers.", "type"=>"invalid_request_error", "param"=>"logit_bias", "code"=>nil}

Which means that for some reasons, the chat completion API is not accepting negative ints today!

Let me test some more and get back to you.

[Amadeus]

no thats just the qoute mark is not right
try the following simple python sample which i banned the word “AI”

openai.ChatCompletion.create(
  model="gpt-3.5-turbo",
  logit_bias= {15836:-100},
  messages=[
        {"role": "user", "content": "Introduce yourself"}
    ]
)

it return

"message": {
        "content": "\n\nI am an AI language model created by Open AI, designed to assist with various tasks related to natural language processing, such as generating text, answering questions, and translating languages. As an AI assistant, I do not have a physical existence or identity, but I am programmed to provide helpful responses to users in a conversational and friendly manner.",
        "role": "assistant"
      }

clearly the word AI is not banned

[ruby_coder]

@ruby_coder

Let me test some more and get back to you.

@Amadeus

logit_bias= {15836:-100}

I am not sure what API wrapper you are using, but the logit_bias key is a string because it is a JSON object. It is not possible that the key in a JSON object is an int as you have posted:

logit_bias= {15836:-100}

We (in this community) ran tests all yesterday and and it worked fine with the JSON key as a string, see @curt.kennedy below:

Also, from my testing:

Reference Community Tests:

High values in logit_bias in OpenAI chat completion endpoint params causes error

Just tested this and it is slightly more complex than that because the API describes login_bias as a JSON object, but in fact it an “odd” way of doing things, in my view, but it works. Without regard to the tokenizer, I simply selected a random token and tested for errors, not if the token was the token expected to effect the completion , which is another topic (which is the “what tokenizer to use” question). I only tested the format of the param object: Test 1: Single JSON Object {"2435":…

[Amadeus]

… this is the way to do it in python.
if you do it in json yes, use qoute mark, but thats not really the main problem

[ruby_coder]

@ruby_coder

but the logit_bias key is a string because it is a JSON object. It is not possible that the key in a JSON object is an int as you have posted:

Yes, I just tested it. again:

@logit_bias = {"370":0, "20554":0, "329":0, "45032":0}

Works fine, because the logit_bias key is always a string:

[Amadeus]

no… clearly not working. did you even check if the word is actually banned>

[ruby_coder]

@Amadeus

if you do it in json yes, use qoute mark, but thats not really the main problem

I have no idea about which wrapper you are using @Amadeus … but you posted it to me that I was wrong, so I am just correcting the record.

To be clear, the logit_bias param is a JSON object and the key is always a string.

Whatever Python wrapper you are using is “incorrectly coded” because it is a string not an int, so they must covert the int to a string.

[Amadeus]

the problem here is, when you set a word to -100, it should not appear in reply.
if you banned “AI”, it shout not reply I am AI

[Amadeus]

ok… you insist to use string i use string to test again and its not working.
why don’t you test my prompt? to see if the word is banned?

[ruby_coder]

@Amadeus

the problem here is, when you set a word to -100, it should not appear in reply.
if you banned “AI”, it shout not reply I am AI

Yes, I know. You are the one who posted incorrect information here:

@Amadeus

try the following simple python sample which i banned the word “AI”

openai.ChatCompletion.create(
  model="gpt-3.5-turbo",
  logit_bias= {15836:-100},
  messages=[

Let’s move on…

Thanks

[ruby_coder]

@Amadeus

why don’t you test my prompt?

Please calm down and provide your system prompt, exactly.

I am trying to help you but when you tell me “you are wrong” you should expect me to correct your error.

[Amadeus]

i bet you don’t even try it in python and you ask what wrapper… its literally the official api
just paste the code i provide with import opanai …

[ruby_coder]

@Amadeus

i bet you don’t even try it in python and you ask what wrapper…

Of course I don’t.

I code in Ruby, not Python and I have no intention to code in Python.

OK. Goodbye.

Good luck.

I’m going to the gym

[Amadeus]

and the prompt
just paste
Introduce yourself
to the role user
and
set 15836, or string "15836" to -100

[Amadeus]

so simple, and just try it. i will not trying to debate which one is correct with you, why not try both.

[AI.Dev]

expect me to correct your error

He’s not wrong though
In Using logit bias to define token probability | OpenAI Help Center
You can see the example given by OpenAI

openai.Completion.create(
 engine="davinci",
 max_tokens=50,
 temperature=0,
 prompt = "Once upon a",
 logit_bias={2435:-100, 640:-100},
)

So your claim

To be clear, the logit_bias param is a JSON object and the key is always a string.

Whatever Python wrapper you are using is “incorrectly coded” because it is a string not an int, so they must covert the int to a string.

Is actually wrong

[ruby_coder]

@AI.Dev

Is actually wrong

No. It is not wrong.

Your Python API wrapper is poorly written.

Don’t you understand basic software good coding practices? The API calls for a JSON object. Your Python wrapper is incorrectly written to accept an integer as a JSON key and if you look inside that Python API wrapper, you will see that the developer then changes that integer to a string before the wrapper sends it to the API. This is a very poor coding practice.

Yes, your wrapper works using an int because the coder compensates for the error by mapping the int to a string in the code which is NOT visible to you.

I use a better written API wrapper which does not mismatch param types.

Also, I tested this using my OpenAPI lab just now, using:

   @logit_bias = {"882":0}

and it works fine.

If I change it to:

    @logit_bias = {"882":-100}

It still works fine, no errors:

So, at least for me, I have no problem with this:

   @logit_bias = {"882":-100}

as my best guess is that OpenAI was working on the system and the errors before were transitory.

Take care of yourself guys.

I wish you the best of luck with your poorly written Python API wrapper.

The Ruby ruby_openai OpenAI API wrapper does not mistype JSON keys because it is better written that the Python API wrapper you are using.

Why argue about it? I don’t have any errors on my end.

[AI.Dev]

Can you test

@logit_bias = {"15836":-100}

and then ask the assistant

Introduce yourself

If the word “AI” appears in assistant reply, then it means the logit_bias is not working fine. (Because 15836 is the token for “AI”, setting it to -100 should ban the word)
It may not return any error code, but its actually not functioning.

For clearification, @sharkcat who open this thread is asking two different question
1.when set to 100 → error
2.when set to -100 → not functioning
You are talking about question 1
while me and @Amadeus are talking about question 2
that’s the reason why we are talking past each other