Logit_bias not working how I would expect

API
1

I set up a bias dictionary, however it isn’t sampling from the bias I’ve set to 100. For the tokens it is returning, I’ve explicitly set them to -100 and they are still the highest probability.

bias = {'6395': 100, '8134': 100, '21943': -100, '5923': -100}
mapping = {'6395': 'Other', '8134': 'Ref', '21943': 'foo', '5923': ' AR'}

prompt = user_message + '\n\n###\n\n'
            response = openai.Completion.create(
                model=model_name,
                prompt=prompt,
                max_tokens=2,
                logprobs=5,
                temperature=0,
                logit_bias=bias
            )

The Response:

<OpenAIObject text_completion id=cmpl-7u3LKPEhRJaDJH9Wpj8Pn6eGV7zRQ at 0x7fc103a45810> JSON: {
  "id": "cmpl-7u3LKPEhRJaDJH9Wpj8Pn6eGV7zRQ",
  "object": "text_completion",
  "created": 1693593574,
  "model": "my model davinci model",
  "choices": [
    {
      "text": "foofoo",
      "index": 0,
      "logprobs": {
        "tokens": [
          "foo",
          "foo"
        ],
        "token_logprobs": [
          -0.058320418,
          -0.0004045105
        ],
        "top_logprobs": [
          {
            "foo": -0.058320418,
            " AR": -2.87082,
            "Ref": -55.63644,
            " Ref": -78.82004,
            "ref": -83.56125
          },
          {
            "foo": -0.0004045105,
            " AR": -7.812904,
            "!": -87.33654,
            "\"": -87.33654,
            "#": -87.33654
          }
        ],
        "text_offset": [
          160,
          163
        ]
      },
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 37,
    "completion_tokens": 2,
    "total_tokens": 39
  }
}

Anybody have ideas?

2

Can you provide the full API call, including the user_message, so it can be debugged?

Edit: Also, because you seem to be using a fine-tuned model, this will be a bit harder to debug than usual.

If you re-run the completion with some other logit_bias values, we might be able to get a better idea of what is happening. E.g.,

# Baseline
bias = {'6395': 100, '8134': 100, '21943': 0, '5923': -100}

# Slight bias against
bias = {'6395': 100, '8134': 100, '21943': -10, '5923': -100}

# Moderate bias against
bias = {'6395': 100, '8134': 100, '21943': -30, '5923': -100}

# Strong bias against
bias = {'6395': 100, '8134': 100, '21943': -70, '5923': -100}

# Slight bias for
bias = {'6395': 100, '8134': 100, '21943': 10, '5923': -100}

# Moderate bias for
bias = {'6395': 100, '8134': 100, '21943': 30, '5923': -100}

# Strong bias for
bias = {'6395': 100, '8134': 100, '21943': 70, '5923': -100}

# Absolute bias for
bias = {'6395': 100, '8134': 100, '21943': 100, '5923': -100}

By doing this we should be able to see a bit into what is happening with the logprobs of the foo token.

Because logit_bias is additive and logprobs have a domain of (-infty, 0], it is very weird that the foo token has a logprobs value so close to 0.

I don’t have any top-of-the-head ideas that are very good. My initial thought is your fine-tuned model is so biased in favor of foo that the math is breaking somewhere inside the model. But that’s crazy thinking.

But, that’s also why I want to see what logprobs values are reported when you try other biases.

Maybe someone else has seen this before though and can give you an answer from experience.

1 Like
3

One example:

response = openai.Completion.create(
                model=model_name,
                prompt="Requested call back Tues\n\n###\n\n",
                max_tokens=2,
                logprobs=5,
                temperature=0,
                logit_bias={'6395': 100, '8134': 100, '21943': -100, '5923': -100}
            )

I fed it a total of 200 different prompts and…

result_df.model_tag.unique()

array(['ark AR', 'foofoo', 'arkark', 'eanark', 'ean AR', 'AR AR'],
      dtype=object)

Edit:
It’s doing the same thing when I use davinci-002 instead of my fine-tuned model. Nearly the same thing if I used babbage-002. However, using an ada model that was previously fine-tuned… It is sampling appropriately.

4

Let me know if you want this copy pasted. Not sure how to attach a file… and it looks awful copy pasted.

Logprobs are identical in some many cases.

image
image1336×760 42.9 KB

5

I changed it to ony have one token in the bias dict (Other aka 6395). I increased it’s value by 2 each time. At 16, AR starts appearing.

image
image1272×792 50.9 KB

If I do the same thing for Ref (8134) the same thing happens but for foo.

image
image1224×682 47.1 KB

6

Thank you for this.

The formatting is enough that I don’t think you need to worry about making it look pretty.

NOTE: Everything that follows in this post is entirely incorrect. I’m leaving it because I think there is an interesting question about how temperature and logit_bias interact. I would expect the logit_bias to be applied before temperature but the results from the OP made me question that.
END NOTE

Anyway, looking at it again now while I’m not driving, I notice you’re using temperature = 0, and I feel really dumb because I expect that’s the culprit.

I’m not sure precisely how logit_bias interests with temperature (especially when temperature = 0) but there’s enough here that I can start to feel comfortable speculating.

temperature = 0 requires special handling mathematically and it is equivalent to setting the probability of the most likely token to 1 and the probability of all other tokens to 0.

So, if we did this first we end up with a vector of logprobs equal to,

[0, -Inf,  -Inf,  -Inf, ...]

If we then apply the logit_bias (assuming the first token is the one we want to avoid) we get,

[-100, -Inf,  -Inf,  -Inf, ...]

And we’d still select the first token despite the logit_bias against.

But, that brings us back to the issue of the logprobs which don’t seem to change with respect to the logit_bias. Someone who’s done more work with adjusting biases might be able to chime in with more details, but it seems like it might be reporting the unbiased logprobs of the five most likely tokens after the logit_bias is applied.

Weird.

1 Like
7

ALRIGHT

I was using

encoding = tiktoken.encoding_for_model("davinci")

I switched it to

encoding = tiktoken.get_encoding("cl100k_base")

And it’s working as expected. So it had something to do with the token encoder.

So as a general practice… I should probably just be using cl100K for all cases? I assumed setting it to the davinci model would have been the correct encoding.

8

Ah… there you go.

So you need to exclude token 8134 then?

And, thank you for humoring me by running the other test cases.

Easy mistake to make though with the encoder, I would have assumed your fine-tuned model would be using the davinci encoder as well.

Glad you got it sorted, sorry I couldn’t be more help.

Edit:
All of the current models use cl100k_base, I guess it depends on when and on which model you did your fine-tuning.

1 Like
9

I’m going to go look back at my ada model with temperature 0. I like your previous post and I am so curious to know how they handle the 0 temperature case.

From the tiktoken github… time for a PR? Well… maybe after January? Or at least explicitly add davinci-002 and babbage-002

image
image391×832 12.5 KB

1 Like
10

You need to use the correct BPE encoder for the model. You cannot freely switch between ada, text-davinci-003, or gpt-3.5-turbo and use the same logit numbers.

Input the text here after selecting the correct model (newest davinci-002 is same as 3.5)

https://tiktokenizer.vercel.app/