JSON Response + logit_bias

API
1

I don’t think logit_bias works when you have response_format={“type”: “json_object”}

I’m really hoping it’s a me thing. My code is below. For this example I’m excluding the tokens in “garnered high praise”. The encoder I’m using is

tiktoken.get_encoding(“cl100k_base”)

The generated token ids from the encoder matches the online tokenizer tool for gpt-4

def generate_summary(user_prompt, system_prompt, bias_dict):
    response = openai.ChatCompletion.create(
        model="gpt-4-0125-preview",
        messages=[
            {
                "role": "system",
                "content": (
                    system_prompt
                )
            },
            {"role": "user", "content": user_prompt}
        ],
        response_format={"type": "json_object"},
        temperature=random.uniform(0.1, 0.5),
        logit_bias = bias_dict
    )


    return json.loads(response['choices'][0]['message']['content'])

bias_dict = {‘12440’: -100.0, ‘1215’: -100.0, ‘291’: -100.0, ‘1579’: -100.0, ‘29488’: -100.0}

AND… the output has the phrase “garnered high praise” verbatim.

Does anyone have any ideas? Or am I SOL when using the JSON response option?

2

The token numbers are not inclusive of all you want to prevent. You only need to stop the first part of the word to send the AI in a new direction.

It appears that logit_bias with “type”: “json_object” is disabled or not working on gpt-4-1106-preview, as I observed before (and pleaded for relief from this inability to stop the nonsense). However, it does work on gpt-3.5-turbo-1106, gpt-3.5-turbo-0125, and gpt-4-0125-preview.

Here I make it impossible to write the correct thing (and make the JSON mode redundant anyway, with quality instructions), and we get:

response content:
{
    "response": "The phrase 'gathered high praise' is...

Example with all the tokens to blast.

model = "gpt-3.5-turbo-1106"
print(f"---{model}---")
params = {
  "model": model,
  "max_tokens": 15,
  "top_p":0.001,
  "stream": True,
  "response_format": { "type": "json_object" },
  "logprobs": True,
  "top_logprobs": 1,
  "logit_bias": {
                 68390:-100,    # " garnered"
                 12440:-100,     # "gar"
                 45030:-100,     # "Gar" - no effect unless 311 
                 47201:-100,       # " garner"
                 4951:-100,   # "/g"
                 },

  "messages": [
    {
        "role": "system", "content": """
You are FunBot, a large language model trained by OpenAI.

// response output format - mandatory!
The only allowed output type is JSON, using key "response", with a response to the user.
""".strip()},
    {
        "role": "user", "content": "Where does the phrase \"garnered high praise\" originate?"
    },
    ],
}

You also should update your openai library, tiktoken, and all techniques to use the “client”…

response = client.chat.completions.create(**params)

(new parsing code)
1 Like
3

Thank for the suggestion to update the openai library!

Unfortunately, the issue I described is while using gpt-4-0125-preview. I’ll whittle down my prompt sizes to try out 3.5 with json outputs. But I think I’ll just layer on another API call to rewrite everything after the fact.

Very helpful reply :slight_smile:

4

Fortunately, that’s a model that I say works, and the issue you described is by not accounting for the various tokens that say the same word.

5

Unfortunately, It did not work. I’ll find a work around. Have a good one!