Can't emulate optional properties with Structured Outputs

Here the documentation shows how an optional property can be emulated with Structured Outputs using a union type with null.

However, I can’t get this to work as expected. Below is an example adapted from the example in the documentation. I have set ‘unit’ property type to ‘[“string”, “null”]’, but despite what I put in the prompt, I can’t elicit a response with ‘“unit”:null’. If I turn off Structured Outputs by setting ‘“strict”:False’, I do get ‘“unit”:null’ with the example below. I am using gpt-4o-2024-08-06, api version 2024-10-21, openai-python 1.57.4

Am I doing something wrong, or is this a potential bug?

Example Code:

messages=[
    {"role": "system", "content": "You are a helpful assistant. If you are asked about weather information, extract the desired location and temperature unit in the provided get_weather format.  If the desired temperature unit is not explicitly given by the user, return null for that property"},
    {"role": "user", "content": "Give me the temperature in Portland"}
]

json_schema = {
  "name": "get_weather",
  "description": "Fetches the weather in the given location",
  "strict": True,
  "schema": {
      "type": "object",
      "properties": {
          "location": {
              "type": "string",
              "description": "The location to get the weather for"
          },
          "unit": {
              "type": ["string", "null"],
              "description": "The unit to return the temperature in",
              "enum": ["F", "C"]
          }
      },
      "additionalProperties": False,
      "required": [
          "location", "unit"
      ]
  }
}

response = client.chat.completions.create(
    model=deployment,
    messages=messages,
    response_format={
      "type": "json_schema",
      "json_schema": json_schema
    }
  )

print(json.dumps(json.loads(response.choices[0].message.content), indent=2))

Example Output:

{
  "location": "Portland",
  "unit": "C"
}

The documentation example is simply wrong.

This is a case that will work:

{
  "name": "get_weather",
  "description": "Fetches the weather in the given location",
  "strict": true,
  "parameters": {
    "type": "object",
    "properties": {
      "location": {
        "type": "string",
        "description": "The location to get the weather for"
      },
      "unit": {
        "type": [
          "null",
          "string"
        ],
        "description": "unit of temperature. Never set unless user-specified.",
        "enum": [
          null,
          "F",
          "C"
        ]
      }
    },
    "additionalProperties": false,
    "required": [
      "location",
      "unit"
    ]
  }
}

Null, not a string, has been added to the enum. The description discourages the AI’s high tendency to choose a string.

You can see it working by user specified or unspecified temperature unit. You receive null as the value and must deal with getting that instead of a string in the JSON, so not truly “optional”.

The strict output is restricted to only those in the enum list. The mixed use of types unexpectedly works.

1 Like

Hi @llamm
I experienced that issue and I found one unexpected result.
Did you confirm the temperature by the result with current real temperature in Portland?
The function result and real weather are not matched.
For instance, the functions returns the temperature with 12 Celsius degree, but real current temperature is 25 Celsius degree.
Is it reliable for the result using function?

Functions interact with code you write.

Maybe you are talking about and are confused by the simulated data produced by AI in the playground that saves you typing in your own function return text for experimentation?

You will see that in a more verbose function, the AI knows it is making examples when I want to make examples of someone:

You would need to find a service that can provide you weather by city, build an application that can track customer’s locations so they don’t have to be asked, map inputs to geospacial coordinates or weather grids, by separate city search or further AI to narrow multiple search results, etc.

1 Like

@_j thanks, that seems to do the trick!