Gpt4o logit_bias maximum value is too low?

API
, ,
1

We use logit_bias to prevent the generation of LaTeX markup. Specifically:

    "logit_bias": {  # For gpt-4o family
        "112251": -100,  # "\("
        "46352": -100,  # " \("
    },

The openai API for chat completions seems to suddenly be reporting the following:

"Invalid key in \'logit_bias\': 112251. Maximum value is 100257."

Is this a deliberate restriction here to stay, or a temporary mistake in the API?

2

It seems the API is blocking token numbers for you like it would for models before gpt-4o, which only have 100k token numbers.

Which specific model are you using? Check that you didn’t switch to gpt-3.5-turbo, gpt-4, gpt-4-turbo.

You can try a dated model name if using the gpt-4o alias, or vice versa, in case they misconfigured the API.

3

Testing: “logit_bias”: {112251: -5}

  • gpt-3.5-turbo
    Exception: API returned error: {‘error’: {‘message’: “Invalid key in ‘logit_bias’: 112251. Maximum value is 101281.”, ‘type’: ‘invalid_request_error’, ‘param’: ‘logit_bias’, ‘code’: None}}

  • “gpt-4o-2024-08-06”
    No problems.
    in: {‘model’: ‘gpt-4o-2024-08-06’, ‘max_completion_tokens’: 500, ‘messages’: [{‘role’: ‘system’, ‘content’: [{‘type’: ‘text’, ‘text’: ‘You are a helpful assistant.’}]}, {‘role’: ‘user’, ‘content’: [{‘type’: ‘text’, ‘text’: ‘hello, show a math formula.’}]}], ‘stream’: True, ‘stream_options’: {‘include_usage’: True}, ‘logit_bias’: {112251: -5}, ‘tools’: [{‘type’: ‘function’, ‘function’: {‘name’: ‘get_current_weather’, ‘description’: ‘Get the current weather in a given location.’, ‘parameters’: {‘type’: ‘object’, ‘properties’: {‘location’: {‘type’: ‘string’, ‘description’: ‘query: Write full sentence questions for understanding.’}, ‘unit’: {‘type’: ‘string’, ‘enum’: [‘celsius’, ‘fahrenheit’], ‘description’: ‘The unit of temperature, fahrenheit for USA’}}, ‘required’: [‘location’, ‘unit’], ‘additionalProperties’: False}}}], ‘tool_choice’: ‘auto’}

  • “gpt-4o-2024-08-06”: structured response and/or structured function
    OK with tool call and response

Notable is a new range of unlocked tokens for gpt-3.5-turbo…Special tokens, or retrained model? gpt-4-1106-preview is 100257.

4

I’m pretty certain they were gpt4o calls, I will just have to assume it was a borked deploy on OpenAi’s end, we haven’t changed anything and the calls are working again. :shrug:

1 Like