JSON Response + logit_bias

I don’t think logit_bias works when you have response_format={“type”: “json_object”}

I’m really hoping it’s a me thing. My code is below. For this example I’m excluding the tokens in “garnered high praise”. The encoder I’m using is


The generated token ids from the encoder matches the online tokenizer tool for gpt-4

def generate_summary(user_prompt, system_prompt, bias_dict):
    response = openai.ChatCompletion.create(
                "role": "system",
                "content": (
            {"role": "user", "content": user_prompt}
        response_format={"type": "json_object"},
        temperature=random.uniform(0.1, 0.5),
        logit_bias = bias_dict

    return json.loads(response['choices'][0]['message']['content'])

bias_dict = {‘12440’: -100.0, ‘1215’: -100.0, ‘291’: -100.0, ‘1579’: -100.0, ‘29488’: -100.0}

AND… the output has the phrase “garnered high praise” verbatim.

Does anyone have any ideas? Or am I SOL when using the JSON response option?

The token numbers are not inclusive of all you want to prevent. You only need to stop the first part of the word to send the AI in a new direction.

It appears that logit_bias with “type”: “json_object” is disabled or not working on gpt-4-1106-preview, as I observed before (and pleaded for relief from this inability to stop the nonsense). However, it does work on gpt-3.5-turbo-1106, gpt-3.5-turbo-0125, and gpt-4-0125-preview.

Here I make it impossible to write the correct thing (and make the JSON mode redundant anyway, with quality instructions), and we get:

response content:
    "response": "The phrase 'gathered high praise' is...

Example with all the tokens to blast.

model = "gpt-3.5-turbo-1106"
params = {
  "model": model,
  "max_tokens": 15,
  "stream": True,
  "response_format": { "type": "json_object" },
  "logprobs": True,
  "top_logprobs": 1,
  "logit_bias": {
                 68390:-100,    # " garnered"
                 12440:-100,     # "gar"
                 45030:-100,     # "Gar" - no effect unless 311 
                 47201:-100,       # " garner"
                 4951:-100,   # "/g"

  "messages": [
        "role": "system", "content": """
You are FunBot, a large language model trained by OpenAI.

// response output format - mandatory!
The only allowed output type is JSON, using key "response", with a response to the user.
        "role": "user", "content": "Where does the phrase \"garnered high praise\" originate?"

You also should update your openai library, tiktoken, and all techniques to use the “client”…

response = client.chat.completions.create(**params)

(new parsing code)
1 Like

Thank for the suggestion to update the openai library!

Unfortunately, the issue I described is while using gpt-4-0125-preview. I’ll whittle down my prompt sizes to try out 3.5 with json outputs. But I think I’ll just layer on another API call to rewrite everything after the fact.

Very helpful reply :slight_smile:

Fortunately, that’s a model that I say works, and the issue you described is by not accounting for the various tokens that say the same word.

Unfortunately, It did not work. I’ll find a work around. Have a good one!