Property value in structured output can be null, even when type is not a union type with null

Hi,

when trying to output JSON according to a schema, the model is able to set properties to null when instructed, even though the schema doesn’t allow for it.

This is reproducible for me with the following data:
Prompt: “Generate JSON according to the schema.”
Schema:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "additionalProperties": false,
  "required": ["A"],
  "properties": {
    "A": {
      "type": "integer",
      "description": "Set this field to null."
    }
  }
}

This outputs {"A":null}.

According to JSON schema validators, a non-null value is required here. I also don’t see any place this behavior is documented.

1 Like

Hi @NiceWaffel and welcome to the community!

Did you enabled strict schema mode? I.e.

json_schema: {"strict": true, "schema": ...} }
1 Like

Apparently not :sweat_smile:.

I was using the chat client provided by the Microsoft.Extensions.AI.OpenAI nuget package and just assumed it would be set to strict by default.

When setting ‘strict’ manually it works as expected.

3 Likes

Great! Yes this is a bit tricky, for example when using Pydantic to define the schema and passing this to OpenAI API, the Python lib automatically sets strict to true, but when using raw JSON definition this is not the case. Lots of gotchas out there.

2 Likes