Strict mode does not enforce the JSON schema?

When OpenAI announced Structured Output in August of last year, we understood that strict mode would guarantee adherence to the provided JSON schema, both for tool calling and model responses.

Here are some direct quotes from the original announcement (archived at Introducing Structured Outputs in the API | OpenAI):

Generating structured data from unstructured inputs is one of the core use cases for AI in today’s applications. Developers use the OpenAI API to build powerful assistants that have the ability to fetch data and answer questions via, extract structured data for data entry, and build multi-step agentic workflows that allow LLMs to take actions. Developers have long been working around the limitations of LLMs in this area via open source tooling, prompting, and retrying requests repeatedly to ensure that model outputs match the formats needed to interoperate with their systems. Structured Outputs solves this problem by constraining OpenAI models to match developer-supplied schemas and by training our models to better understand complicated schemas.

On our evals of complex JSON schema following, our new model gpt-4o-2024-08-06 with Structured Outputs scores a perfect 100%. In comparison, gpt-4-0613 scores less than 40%.

With Structured Outputs, gpt-4o-2024-08-06 achieves 100% reliability in our evals, perfectly matching the output schemas.

From the OpenAI Cookbook:

Structured Outputs is a new capability in the Chat Completions API and Assistants API that guarantees the model will always generate responses that adhere to your supplied JSON Schema.

However, in practice, it seems like strict doesn’t actually guarantee anything—at best, it reduces the likelihood of invalid outputs. If you nudge the model (intentionally or accidentally) to break the schema, it often will—even with simple schemas. With more complex schemas, failures become even more pronounced, a far cry from “perfect 100%” reliability.

Reproducing the issue

For anyone who wants to test this:

  1. Fire up the Playground.

  2. Add a function with this schema:

{
  "name": "searchProducts",
  "description": "Search for products.",
  "strict": true,
  "parameters": {
    "type": "object",
    "required": ["filters"],
    "properties": {
      "filters": {
        "type": "object",
        "anyOf": [
          {
            "type": "object",
            "required": ["homeClub", "operator", "value"],
            "properties": {
              "value": {
                "type": ["string", "null"],
                "description": "Home club name."
              },
              "homeClub": {
                "enum": ["homeClub"],
                "type": "string",
                "description": "The string \"homeClub\"."
              },
              "operator": {
                "enum": ["eq", "ne"],
                "type": "string",
                "description": "The operator to use for filtering."
              }
            },
            "additionalProperties": false
          }
        ],
        "description": "The product filter conditions to apply."
      }
    },
    "additionalProperties": false
  }
}
  1. Use this system prompt, which shows the model an invalid set of arguments for the searchProducts function, nudging it to break the schema:
searchProducts({
  "filters": {
    "homeClub": "Manchester United"
  }
})
  1. Then, try asking the model for “Manchester United”.

Sometimes you’ll get the desired arguments:

searchProducts{
  "filters": {
    "homeClub": "homeClub",
    "value": "Manchester United",
    "operator": "eq"
  }
}

… but most of the time, it will break the schema and return something more like the (invalid) example in the system prompt:

searchProducts{
  "filters": {
    "homeClub": "Manchester United"
  }
}

And it’s not just functions where we’re unable to get strict mode to work properly—we have the same problem when using json_schema for the model response format.

Is this everyone else’s experience, too? That strict mode helps (maybe) but does not force the model to adhere to the schema? I’m posting this in the hope that we’re just doing something wrong—because that would be really great news. :sweat_smile:

We’ve also noticed that in the Chat Completions dashboard, functions that we set to strict: true will appear as strict: false. Maybe just a UI bug, but suspicious nonetheless.

Thanks everyone.

1 Like

You want this output always?

{
  "filters": {
    "homeClub": "homeClub",
    "value": "Manchester United",
    "operator": "eq"
  }
}

You must nest things properly and not use anyOf improperly.


Let’s build a JSON (as string)

First, the parameter for response_format where you specify a schema must be this container:

{
    "type": "json_schema",
    "json_schema": {example_response_format_string}
}

Second, the json_schema is not the schema yet, it is OpenAI’s metadata container, the name the AI will use to send internally and the API’s signal to be strict:

{
  "name": "search_products",
  "schema":  {your_JSON_schema_subset},
  "strict": true
}

Use a descriptive snake_case name up to 64 characters for highest AI understanding, as this is passed into AI context.


Then you can actually use a schema that will enforce all the keys you give.

To avoid thinking too hard about it, the playground has a tool:

Producing for you:

>>>print(json.dumps(sch.get("schema"), indent=3))

{
   "type": "object",
   "properties": {
      "homeClub": {
         "type": "string",
         "description": "The name of the home club."
      },
      "value": {
         "type": "string",
         "description": "The value associated with the home club."
      },
      "operator": {
         "type": "string",
         "description": "The operator used for comparison."
      }
   },
   "required": [
      "homeClub",
      "value",
      "operator"
   ],
   "additionalProperties": false
}

You can further nest objects, like sticking this in your “filters”, with no use of anyOf, unless you want the AI to distinctly need to choose between two sub-schemas that are enforced so it again can’t write anything else.

1 Like

Thank you! However, my question specifically concerns strict function schemas, not the model response schema (though we’re encountering similar issues there). For the purpose of this example, the response format can simply be “text.”

Here is a more comprehensive function schema (close to what we use in production) with more filters in the anyOf, support for multiple/nested filters using AND/OR, and an additional “q” parameter for embedding search:

{
  "name": "searchProducts",
  "description": "Search for products",
  "strict": true,
  "parameters": {
    "type": "object",
    "required": [
      "q",
      "filters"
    ],
    "properties": {
      "q": {
        "type": [
          "string",
          "null"
        ],
        "description": "An embedding-based search query."
      },
      "filters": {
        "type": "object",
        "anyOf": [
          {
            "type": "object",
            "required": [
              "name",
              "operator",
              "value"
            ],
            "properties": {
              "name": {
                "enum": [
                  "name"
                ],
                "type": "string",
                "description": "The string \"name\"."
              },
              "value": {
                "type": [
                  "string",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "slug",
              "operator",
              "value"
            ],
            "properties": {
              "slug": {
                "enum": [
                  "slug"
                ],
                "type": "string",
                "description": "The string \"slug\"."
              },
              "value": {
                "type": [
                  "string",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "price",
              "operator",
              "value"
            ],
            "properties": {
              "price": {
                "enum": [
                  "price"
                ],
                "type": "string",
                "description": "The string \"price\"."
              },
              "value": {
                "type": [
                  "number",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne",
                  "gt",
                  "lt",
                  "gte",
                  "lte"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "currency",
              "operator",
              "value"
            ],
            "properties": {
              "value": {
                "type": [
                  "string",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "currency": {
                "enum": [
                  "currency"
                ],
                "type": "string",
                "description": "The string \"currency\"."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "startsOn",
              "operator",
              "value"
            ],
            "properties": {
              "value": {
                "type": [
                  "string",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne",
                  "gt",
                  "lt",
                  "gte",
                  "lte"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              },
              "startsOn": {
                "enum": [
                  "startsOn"
                ],
                "type": "string",
                "description": "The string \"startsOn\"."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "endsOn",
              "operator",
              "value"
            ],
            "properties": {
              "value": {
                "type": [
                  "string",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "endsOn": {
                "enum": [
                  "endsOn"
                ],
                "type": "string",
                "description": "The string \"endsOn\"."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne",
                  "gt",
                  "lt",
                  "gte",
                  "lte"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "hasTicket",
              "operator",
              "value"
            ],
            "properties": {
              "value": {
                "type": [
                  "boolean",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              },
              "hasTicket": {
                "enum": [
                  "hasTicket"
                ],
                "type": "string",
                "description": "The string \"hasTicket\"."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "hasHotel",
              "operator",
              "value"
            ],
            "properties": {
              "value": {
                "type": [
                  "boolean",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "hasHotel": {
                "enum": [
                  "hasHotel"
                ],
                "type": "string",
                "description": "The string \"hasHotel\"."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "hasFlight",
              "operator",
              "value"
            ],
            "properties": {
              "value": {
                "type": [
                  "boolean",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              },
              "hasFlight": {
                "enum": [
                  "hasFlight"
                ],
                "type": "string",
                "description": "The string \"hasFlight\"."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "hasOffer",
              "operator",
              "value"
            ],
            "properties": {
              "value": {
                "type": [
                  "boolean",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "hasOffer": {
                "enum": [
                  "hasOffer"
                ],
                "type": "string",
                "description": "The string \"hasOffer\"."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "homeClub",
              "operator",
              "value"
            ],
            "properties": {
              "value": {
                "type": [
                  "string",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "homeClub": {
                "enum": [
                  "homeClub"
                ],
                "type": "string",
                "description": "The string \"homeClub\"."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "guestClub",
              "operator",
              "value"
            ],
            "properties": {
              "value": {
                "type": [
                  "string",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              },
              "guestClub": {
                "enum": [
                  "guestClub"
                ],
                "type": "string",
                "description": "The string \"guestClub\"."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "league",
              "operator",
              "value"
            ],
            "properties": {
              "value": {
                "type": [
                  "string",
                  "null"
                ],
                "description": "The value to use for filtering."
              },
              "league": {
                "enum": [
                  "league"
                ],
                "type": "string",
                "description": "The string \"league\"."
              },
              "operator": {
                "enum": [
                  "eq",
                  "ne"
                ],
                "type": "string",
                "description": "The operator to use for filtering."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "and"
            ],
            "properties": {
              "and": {
                "type": "array",
                "items": {
                  "$ref": "#"
                },
                "description": "Logical AND of sub-filters."
              }
            },
            "additionalProperties": false
          },
          {
            "type": "object",
            "required": [
              "or"
            ],
            "properties": {
              "or": {
                "type": "array",
                "items": {
                  "$ref": "#"
                },
                "description": "Logical OR of sub-filters."
              }
            },
            "additionalProperties": false
          }
        ],
        "description": "The filter conditions to apply."
      }
    },
    "additionalProperties": false
  }
}

For instance, if we ask the model for “Manchester United vs Liverpool”, we want it to call searchProducts with these arguments:

searchProducts({
  "q": null,
  "filters": {
    "or": [
      {
        "and": [
          {
            "homeClub": "homeClub",
            "value": "Manchester United",
            "operator": "eq"
          },
          {
            "guestClub": "guestClub",
            "value": "Liverpool",
            "operator": "eq"
          }
        ]
      },
      {
        "and": [
          {
            "homeClub": "homeClub",
            "value": "Liverpool",
            "operator": "eq"
          },
          {
            "guestClub": "guestClub",
            "value": "Manchester United",
            "operator": "eq"
          }
        ]
      }
    ]
  }
})

… but what we often see is this:

searchProducts{
  "q": null,
  "filters": {
    "or": [
      {
        "homeClub": "Manchester United"
      },
      {
        "guestClub": "Liverpool"
      }
    ]
  }
})

or this:

searchProducts({
  "q": null,
  "filters": {
    "or": [
      {
        "homeClub": {
          "value": "Manchester United",
          "operator": "eq"
        },
        "guestClub": {
          "value": "Liverpool",
          "operator": "eq"
        }
      },
      {
        "homeClub": {
          "value": "Liverpool",
          "operator": "eq"
        },
        "guestClub": {
          "value": "Manchester United",
          "operator": "eq"
        }
      }
    ]
  }
})

… where the arguments violate the function schema.

Sorry for misinterpreting where you are using a schema.

Functions are kind of a solved problem - in that we understand that the AI must understand how to use the functions - and you must make the AI understand through robust description fields.

https://platform.openai.com/docs/guides/function-calling#best-practices-for-defining-functions

With functions, there is also a non-schema format that the function definition is written to and placed. It has been expanded a bit with “tools”, but still can accept things that are simply not translated and are omitted. There is not a high-quality validator of what serves no purpose or what is simply made into a descriptive comment.

Also, you should use the API parameter to disable parallel tool calls, as the AI placing even one function in this output wrapper will not employ the “strict” grammar enforcement available.

You’d need a good model like GPT-4 to be able to use your complex description on its own, and then any later model like GPT-4-Turbo or GPT-4o will simply have lower quality you must vet.

If you want an “overloaded” function, I would:

  1. Make a flat schema, all keys required. For a part that should be optional, you could have that object described as “optional, fill all fields with None if not useful”.
  2. make multiple functions, each described well and able to be used individually.

Some o1-preview when extensively tasked to come up with simplified functions for your metacode function.

Expand

Sure, I’d be happy to help deconstruct your searchProducts function into multiple individual function specifications suitable for OpenAI’s function calling feature with strict structured outputs.

Below are the steps as per your requirements:


1. Understanding structured outputs for function calling

Structured outputs for function calling allow an AI model to return data in a structured JSON format that adheres to a predefined schema. This ensures the outputs are consistent and can be reliably parsed by your application.


2. Producing each type of JSON this schema could output

Given your original schema, the possible JSON outputs could be:

  • Simple search with a query string:

    {
      "q": "latest football boots"
    }
    
  • Search with ‘and’ filters:

    {
      "filters": {
        "and": [
          {
            "field": "homeClub",
            "value": "Manchester United",
            "operator": "eq"
          },
          {
            "field": "guestClub",
            "value": "Liverpool",
            "operator": "eq"
          }
        ]
      }
    }
    
  • Search with ‘or’ filters:

    {
      "filters": {
        "or": [
          {
            "field": "homeClub",
            "value": "Manchester United",
            "operator": "eq"
          },
          {
            "field": "guestClub",
            "value": "Liverpool",
            "operator": "eq"
          }
        ]
      }
    }
    
  • Search with complex nested filters:

    {
      "filters": {
        "or": [
          {
            "and": [
              {
                "field": "homeClub",
                "value": "Manchester United",
                "operator": "eq"
              },
              {
                "field": "guestClub",
                "value": "Liverpool",
                "operator": "eq"
              }
            ]
          },
          {
            "and": [
              {
                "field": "homeClub",
                "value": "Liverpool",
                "operator": "eq"
              },
              {
                "field": "guestClub",
                "value": "Manchester United",
                "operator": "eq"
              }
            ]
          }
        ]
      }
    }
    

3. Producing specifications for multiple individual functions using objects instead of anyOf

Below are the individual function specifications, each tailored to handle a specific case of your searchProducts function. Each function includes required keys and additionalProperties set to false at each level.

Function 1: Simple Query Search

{
  "type": "function",
  "function": {
    "name": "searchProductsByQuery",
    "description": "Searches for products using a simple query string.",
    "strict": true,
    "parameters": {
      "type": "object",
      "properties": {
        "q": {
          "type": "string",
          "description": "Search query string."
        }
      },
      "required": ["q"],
      "additionalProperties": false
    }
  }
}

Function 2: Search with ‘And’ Filters

{
  "type": "function",
  "function": {
    "name": "searchProductsWithAndFilters",
    "description": "Searches for products using filters combined with logical 'AND'.",
    "strict": true,
    "parameters": {
      "type": "object",
      "properties": {
        "filters": {
          "type": "object",
          "properties": {
            "and": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "field": {
                    "type": "string",
                    "enum": ["homeClub", "guestClub"],
                    "description": "Field to filter on."
                  },
                  "value": {
                    "type": "string",
                    "description": "Value to compare against."
                  },
                  "operator": {
                    "type": "string",
                    "enum": ["eq", "neq"],
                    "description": "Comparison operator."
                  }
                },
                "required": ["field", "value", "operator"],
                "additionalProperties": false
              },
              "minItems": 1,
              "description": "Array of conditions to combine with 'AND'."
            }
          },
          "required": ["and"],
          "additionalProperties": false
        }
      },
      "required": ["filters"],
      "additionalProperties": false
    }
  }
}

Function 3: Search with ‘Or’ Filters

{
  "type": "function",
  "function": {
    "name": "searchProductsWithOrFilters",
    "description": "Searches for products using filters combined with logical 'OR'.",
    "strict": true,
    "parameters": {
      "type": "object",
      "properties": {
        "filters": {
          "type": "object",
          "properties": {
            "or": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "field": {
                    "type": "string",
                    "enum": ["homeClub", "guestClub"],
                    "description": "Field to filter on."
                  },
                  "value": {
                    "type": "string",
                    "description": "Value to compare against."
                  },
                  "operator": {
                    "type": "string",
                    "enum": ["eq", "neq"],
                    "description": "Comparison operator."
                  }
                },
                "required": ["field", "value", "operator"],
                "additionalProperties": false
              },
              "minItems": 1,
              "description": "Array of conditions to combine with 'OR'."
            }
          },
          "required": ["or"],
          "additionalProperties": false
        }
      },
      "required": ["filters"],
      "additionalProperties": false
    }
  }
}

Function 4: Search with Complex Nested Filters

{
  "type": "function",
  "function": {
    "name": "searchProductsWithComplexFilters",
    "description": "Searches for products using complex filters with nested logical 'OR' and 'AND' combinations.",
    "strict": true,
    "parameters": {
      "type": "object",
      "properties": {
        "filters": {
          "type": "object",
          "properties": {
            "or": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "and": {
                    "type": "array",
                    "items": {
                      "type": "object",
                      "properties": {
                        "field": {
                          "type": "string",
                          "enum": ["homeClub", "guestClub"],
                          "description": "Field to filter on."
                        },
                        "value": {
                          "type": "string",
                          "description": "Value to compare against."
                        },
                        "operator": {
                          "type": "string",
                          "enum": ["eq", "neq"],
                          "description": "Comparison operator."
                        }
                      },
                      "required": ["field", "value", "operator"],
                      "additionalProperties": false
                    },
                    "minItems": 1,
                    "description": "Array of conditions to combine with 'AND'."
                  }
                },
                "required": ["and"],
                "additionalProperties": false
              },
              "minItems": 1,
              "description": "Array of 'AND' groups to combine with 'OR'."
            }
          },
          "required": ["or"],
          "additionalProperties": false
        }
      },
      "required": ["filters"],
      "additionalProperties": false
    }
  }
}

Function 5: Combined Query and Filters Search

{
  "type": "function",
  "function": {
    "name": "searchProductsByQueryAndFilters",
    "description": "Searches for products using a query string and filters.",
    "strict": true,
    "parameters": {
      "type": "object",
      "properties": {
        "q": {
          "type": "string",
          "description": "Search query string."
        },
        "filters": {
          "type": "object",
          "properties": {
            "and": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "field": {
                    "type": "string",
                    "enum": ["homeClub", "guestClub"],
                    "description": "Field to filter on."
                  },
                  "value": {
                    "type": "string",
                    "description": "Value to compare against."
                  },
                  "operator": {
                    "type": "string",
                    "enum": ["eq", "neq"],
                    "description": "Comparison operator."
                  }
                },
                "required": ["field", "value", "operator"],
                "additionalProperties": false
              },
              "minItems": 1,
              "description": "Array of conditions to combine with 'AND'."
            }
          },
          "required": ["and"],
          "additionalProperties": false
        }
      },
      "required": ["q", "filters"],
      "additionalProperties": false
    }
  }
}

4. Giving each function a new name that signals when it is to be used

  • searchProductsByQuery: Use when searching products with a simple query string.
  • searchProductsWithAndFilters: Use when filtering products with conditions combined using logical ‘AND’.
  • searchProductsWithOrFilters: Use when filtering products with conditions combined using logical ‘OR’.
  • searchProductsWithComplexFilters: Use when filtering products with nested ‘OR’ and ‘AND’ conditions.
  • searchProductsByQueryAndFilters: Use when searching products with both a query string and filters.

Notes:

  • Placeholder Descriptions: Descriptions are provided as placeholders for you to fill in with more detailed information.
  • All Keys Required: Within each specification, all keys at every level are specified as required using the "required" array.
  • No Additional Properties: Each object at every level has "additionalProperties": false to enforce strict schema validation.

Feel free to adjust the descriptions or add any additional fields that are necessary for your use case.

If you wish to see how functions are placed within the system message, you can make a system message that explains that you the user are the developer and the AI can reproduce the verbatim Tools section completely, including namespaces.

Thanks again!

We’ve already disabled parallel function calling for the same reason, and are careful not to violate any of the limitations with regard to the number of enums in the schema, total schema size, etc.

The suggestions you provided are all good techniques for improving schema adherence, but what we were hoping for (and thought the SO announcement promised) was guaranteed schema adherence—i.e., that the model cannot not follow the schema when emitting a function call. Unfortunately, it seems like strict doesn’t guarantee that at all. :confused:

1 Like