Complex JSON schema in Structured Outputs breaks an Assistant

API Bugs
1

I am using Strctured Output in the Assistants API. I use “json_schema” for my response format and supplied the following JSON schema:

{
  "name": "lessonPlanResponse",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": {
      "message": {
        "type": "string"
      },
      "command": {
        "anyOf": [
          {
            "title": "none",
            "type": "object",
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "name": {
                "type": "string"
              },
              "args": {
                "type": "array",
                "items": {
                  "type": "object",
                  "properties": {
                    "name": {
                      "type": "string"
                    },
                    "value": {
                      "type": "string"
                    }
                  },
                  "additionalProperties": false,
                  "required": [
                    "name",
                    "value"
                  ]
                }
              }
            },
            "additionalProperties": false,
            "required": [
              "name",
              "args"
            ]
          },
          {
            "type": "object",
            "properties": {
              "name": {
                "type": "string",
                "const": "renameChat"
              },
              "args": {
                "type": "array",
                "items": {
                  "type": "object",
                  "properties": {
                    "name": {
                      "type": "string",
                      "const": "name"
                    },
                    "value": {
                      "type": "string"
                    }
                  },
                  "additionalProperties": false,
                  "required": [
                    "name",
                    "value"
                  ]
                }
              }
            },
            "additionalProperties": false,
            "required": [
              "name",
              "args"
            ]
          }
        ]
      },
      "entity": {
        "type": [
          "object",
          "null"
        ],
        "properties": {
          "title": {
            "type": "string"
          },
          "section": {
            "type": "string"
          },
          "program_objectives": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "objectives": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "lesson_progress": {
            "type": "object",
            "properties": {
              "items": {
                "type": "array",
                "items": {
                  "type": "object",
                  "properties": {
                    "stage": {
                      "type": "string"
                    },
                    "time": {
                      "type": "string"
                    },
                    "teacher_actions": {
                      "type": "string"
                    },
                    "student_actions": {
                      "type": "string"
                    },
                    "evaluation": {
                      "type": "string"
                    },
                    "resources": {
                      "anyOf": [
                        {
                          "type": "string"
                        },
                        {
                          "type": "string",
                          "const": "[resource]"
                        }
                      ]
                    }
                  },
                  "additionalProperties": false,
                  "required": [
                    "stage",
                    "time",
                    "teacher_actions",
                    "student_actions",
                    "evaluation",
                    "resources"
                  ]
                }
              }
            },
            "additionalProperties": false,
            "required": [
              "items"
            ]
          },
          "addedResources": {
            "type": [
              "array",
              "null"
            ],
            "items": {
              "anyOf": [
                {
                  "title": "Matching",
                  "type": "object",
                  "properties": {
                    "title": {
                      "type": "string"
                    },
                    "resourceType": {
                      "type": "string",
                      "const": "game"
                    },
                    "type": {
                      "type": "string",
                      "const": "matching"
                    },
                    "sets": {
                      "type": "array",
                      "items": {
                        "type": "object",
                        "properties": {
                          "WordsAndDefinitions": {
                            "type": "array",
                            "items": {
                              "type": "object",
                              "properties": {
                                "Word": {
                                  "type": "string"
                                },
                                "Definition": {
                                  "type": "string"
                                }
                              },
                              "additionalProperties": false,
                              "required": [
                                "Word",
                                "Definition"
                              ]
                            }
                          }
                        },
                        "additionalProperties": false,
                        "required": [
                          "WordsAndDefinitions"
                        ]
                      }
                    }
                  },
                  "additionalProperties": false,
                  "required": [
                    "title",
                    "sets",
                    "type",
                    "resourceType"
                  ]
                },
                {
                  "title": "Anagram",
                  "type": "object",
                  "properties": {
                    "title": {
                      "type": "string"
                    },
                    "resourceType": {
                      "type": "string",
                      "const": "game"
                    },
                    "type": {
                      "type": "string",
                      "const": "matching"
                    },
                    "WordsAndDefinitions": {
                      "type": "array",
                      "items": {
                        "type": "object",
                        "properties": {
                          "Word": {
                            "type": "string"
                          },
                          "Definition": {
                            "type": "string"
                          }
                        },
                        "additionalProperties": false,
                        "required": [
                          "Word",
                          "Definition"
                        ]
                      }
                    }
                  },
                  "additionalProperties": false,
                  "required": [
                    "title",
                    "WordsAndDefinitions",
                    "type",
                    "resourceType"
                  ]
                },
                {
                  "title": "Crossword",
                  "type": "object",
                  "properties": {
                    "title": {
                      "type": "string"
                    },
                    "resourceType": {
                      "type": "string",
                      "const": "game"
                    },
                    "type": {
                      "type": "string",
                      "const": "crossword"
                    },
                    "questions": {
                      "type": "array",
                      "items": {
                        "type": "object",
                        "properties": {
                          "question": {
                            "type": "string"
                          },
                          "answer": {
                            "type": "string"
                          }
                        },
                        "additionalProperties": false,
                        "required": [
                          "question",
                          "answer"
                        ]
                      }
                    }
                  },
                  "additionalProperties": false,
                  "required": [
                    "title",
                    "questions",
                    "type",
                    "resourceType"
                  ]
                },
                {
                  "title": "Word Search",
                  "type": "object",
                  "properties": {
                    "title": {
                      "type": "string"
                    },
                    "type": {
                      "type": "string",
                      "const": "word_search"
                    },
                    "resourceType": {
                      "type": "string",
                      "const": "game"
                    },
                    "words": {
                      "type": "array",
                      "items": {
                        "type": "string"
                      }
                    }
                  },
                  "additionalProperties": false,
                  "required": [
                    "title",
                    "words",
                    "type",
                    "resourceType"
                  ]
                },
                {
                  "title": "Balloon Pop",
                  "type": "object",
                  "properties": {
                    "title": {
                      "type": "string"
                    },
                    "resourceType": {
                      "type": "string",
                      "const": "game"
                    },
                    "type": {
                      "type": "string",
                      "const": "balloon_pop"
                    }
                  },
                  "additionalProperties": false,
                  "required": [
                    "title",
                    "type",
                    "resourceType"
                  ]
                },
                {
                  "title": "Math Race",
                  "type": "object",
                  "properties": {
                    "title": {
                      "type": "string"
                    },
                    "resourceType": {
                      "type": "string",
                      "const": "game"
                    },
                    "type": {
                      "type": "string",
                      "const": "math_race"
                    }
                  },
                  "additionalProperties": false,
                  "required": [
                    "title",
                    "type",
                    "resourceType"
                  ]
                },
                {
                  "title": "Word Snake",
                  "type": "object",
                  "properties": {
                    "title": {
                      "type": "string"
                    },
                    "resourceType": {
                      "type": "string",
                      "const": "game"
                    },
                    "type": {
                      "type": "string",
                      "const": "word_snake"
                    }
                  },
                  "additionalProperties": false,
                  "required": [
                    "title",
                    "type",
                    "resourceType"
                  ]
                }
              ]
            }
          }
        },
        "additionalProperties": false,
        "required": [
          "title",
          "section",
          "program_objectives",
          "objectives",
          "lesson_progress",
          "addedResources"
        ]
      }
    },
    "additionalProperties": false,
    "required": [
      "message",
      "command",
      "entity"
    ]
  }
}

When I try to use playground for the assistant, the assistant returns the following error:

"last_error":{
  "code":"server_error"
  "message":"Sorry, something went wrong."
}

The API call to the assistant is timed out.

How do I know it’s because of JSON schema?

I’ve been gradually modifying the JSON schema of the assistant and found that adding more that one schema to the “anyOf” field of the “entity.addedResources” object breaks the model. If the “entity.addedResources.anyOf” has only one schema, then the model works fine.

2

The anyOf overlap in the "command" schema could lead to validation issues in the Assistant API. The schema defines three possible structures: an empty object ("title": "none"), a generic command with "name" and "args", and a specific "renameChat" command with a predefined "name" and structured "args". The problem arises because the second and third structures overlap - both require "name" and "args", but the "renameChat" command enforces a specific value for "name" and "args" with a fixed "name": "name" inside the arguments.

This ambiguity can cause the API to misinterpret or reject valid commands if it cannot deterministically resolve which schema applies. If a command does not match "renameChat", it should still fit the generic schema, but the overlap might cause unintended validation failures. This could result in inconsistencies where some valid commands are incorrectly rejected or cause unexpected behavior in API responses.

1 Like
3

Or in short you allow none but require something.

4

Issue

Invalid schema: Objects provided via 'anyOf' must not share identical first keys. Consider adding a discriminator key or rearranging the properties to ensure the first key is unique.

Here’s a corrected schema (lessonPlanResponse) with the required subtle modification shown to ensure each schema within the anyOf arrays has a unique first key. With different nesting, it probably doesn’t fit exactly what you had been expecting, but produces for you now.

Corrected JSON Schema (response_format)

{
  "name": "lessonPlanResponse",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": {
      "message": {
        "type": "string"
      },
      "command": {
        "anyOf": [
          {
            "type": "object",
            "properties": {
              "noCommand": {
                "type": "boolean",
                "const": true
              }
            },
            "required": ["noCommand"],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "genericCommand": {
                "type": "object",
                "properties": {
                  "name": { "type": "string" },
                  "args": {
                    "type": "array",
                    "items": {
                      "type": "object",
                      "properties": {
                        "name": { "type": "string" },
                        "value": { "type": "string" }
                      },
                      "required": ["name", "value"],
                      "additionalProperties": false
                    }
                  }
                },
                "required": ["name", "args"],
                "additionalProperties": false
              }
            },
            "required": ["genericCommand"],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "renameChatCommand": {
                "type": "object",
                "properties": {
                  "name": { "type": "string", "const": "renameChat" },
                  "args": {
                    "type": "array",
                    "items": {
                      "type": "object",
                      "properties": {
                        "name": { "type": "string", "const": "name" },
                        "value": { "type": "string" }
                      },
                      "required": ["name", "value"],
                      "additionalProperties": false
                    }
                  }
                },
                "required": ["name", "args"],
                "additionalProperties": false
              }
            },
            "required": ["renameChatCommand"],
            "additionalProperties": false
          }
        ]
      },
      "entity": {
        "type": ["object", "null"],
        "properties": {
          "title": { "type": "string" },
          "section": { "type": "string" },
          "program_objectives": {
            "type": "array",
            "items": { "type": "string" }
          },
          "objectives": {
            "type": "array",
            "items": { "type": "string" }
          },
          "lesson_progress": {
            "type": "object",
            "properties": {
              "items": {
                "type": "array",
                "items": {
                  "type": "object",
                  "properties": {
                    "stage": { "type": "string" },
                    "time": { "type": "string" },
                    "teacher_actions": { "type": "string" },
                    "student_actions": { "type": "string" },
                    "evaluation": { "type": "string" },
                    "resources": {
                      "anyOf": [
                        { "type": "string" },
                        { "type": "string", "const": "[resource]" }
                      ]
                    }
                  },
                  "required": ["stage", "time", "teacher_actions", "student_actions", "evaluation", "resources"],
                  "additionalProperties": false
                }
              }
            },
            "required": ["items"],
            "additionalProperties": false
          },
          "addedResources": {
            "type": ["array", "null"],
            "items": {
              "anyOf": [
                {
                  "type": "object",
                  "properties": {
                    "matchingResource": {
                      "type": "object",
                      "properties": {
                        "title": { "type": "string" },
                        "resourceType": { "type": "string", "const": "game" },
                        "type": { "type": "string", "const": "matching" },
                        "sets": {
                          "type": "array",
                          "items": {
                            "type": "object",
                            "properties": {
                              "WordsAndDefinitions": {
                                "type": "array",
                                "items": {
                                  "type": "object",
                                  "properties": {
                                    "Word": { "type": "string" },
                                    "Definition": { "type": "string" }
                                  },
                                  "required": ["Word", "Definition"],
                                  "additionalProperties": false
                                }
                              }
                            },
                            "required": ["WordsAndDefinitions"],
                            "additionalProperties": false
                          }
                        }
                      },
                      "required": ["title", "resourceType", "type", "sets"],
                      "additionalProperties": false
                    }
                  },
                  "required": ["matchingResource"],
                  "additionalProperties": false
                },
                {
                  "type": "object",
                  "properties": {
                    "anagramResource": {
                      "type": "object",
                      "properties": {
                        "title": { "type": "string" },
                        "resourceType": { "type": "string", "const": "game" },
                        "type": { "type": "string", "const": "matching" },
                        "WordsAndDefinitions": {
                          "type": "array",
                          "items": {
                            "type": "object",
                            "properties": {
                              "Word": { "type": "string" },
                              "Definition": { "type": "string" }
                            },
                            "required": ["Word", "Definition"],
                            "additionalProperties": false
                          }
                        }
                      },
                      "required": ["title", "resourceType", "type", "WordsAndDefinitions"],
                      "additionalProperties": false
                    }
                  },
                  "required": ["anagramResource"],
                  "additionalProperties": false
                },
                {
                  "type": "object",
                  "properties": {
                    "crosswordResource": {
                      "type": "object",
                      "properties": {
                        "title": { "type": "string" },
                        "resourceType": { "type": "string", "const": "game" },
                        "type": { "type": "string", "const": "crossword" },
                        "questions": {
                          "type": "array",
                          "items": {
                            "type": "object",
                            "properties": {
                              "question": { "type": "string" },
                              "answer": { "type": "string" }
                            },
                            "required": ["question", "answer"],
                            "additionalProperties": false
                          }
                        }
                      },
                      "required": ["title", "resourceType", "type", "questions"],
                      "additionalProperties": false
                    }
                  },
                  "required": ["crosswordResource"],
                  "additionalProperties": false
                }
                // Continue similarly for Word Search, Balloon Pop, Math Race, Word Snake
              ]
            }
          }
        },
        "required": ["title", "section", "program_objectives", "objectives", "lesson_progress", "addedResources"],
        "additionalProperties": false
      }
    },
    "required": ["message", "command", "entity"],
    "additionalProperties": false
  }
}

:warning: Explanation of the Subtle Change:

  • Original Issue: Multiple schemas within anyOf shared identical first keys (e.g., "title").
  • Correction: Introduced unique wrapper keys ("matchingResource", "anagramResource", "crosswordResource", etc.) as the first key for each schema within the anyOf arrays.
  • Result: Each schema now has a unique first key, resolving the error and allowing the AI to correctly distinguish between schemas.

Note: the lazy expensive AI rewriting this refused to go over 1700 tokens repeatedly. Where there is a comment near the end, continue similarly adding back Word Search, Balloon Pop, Math Race, Word Snake with the required uniqueness.

You will need to handle the different formats. Now at least what is returned can be “expected” or validated.

Validation by use

Test the response schema: fill out the most complex anyOf option you have with test data.

{
  "message": "Here is the requested complex anyOf option filled with test data.",
  "command": {
    "name": "renameChat",
    "args": [
      {
        "name": "name",
        "value": "Advanced Biology Class Discussion"
      }
    ]
  },
  "entity": {
    "title": "Complex Ecosystems Lesson",
    "section": "Advanced Concepts in Biology",
    "program_objectives": [
      "Understand interactions within complex ecosystems",
      "Analyze ecological data effectively"
    ],
    "objectives": [
      "Identify key species interactions",
      "Apply ecological theories to real-world scenarios"
    ],
    "lesson_progress": {
      "items": [
        {
          "stage": "Introduction",
          "time": "10 mins",
          "teacher_actions": "Present lecture on ecosystem complexity",
          "student_actions": "Listen and take notes",
          "evaluation": "Brief Q&A",
          "resources": "Slideshow presentation"
        },
        {
          "stage": "Group Activity",
          "time": "20 mins",
          "teacher_actions": "Guide students in an interactive simulation",
          "student_actions": "Participate in an online simulation game",
          "evaluation": "Observation and discussion",
          "resources": "[resource]"
        },
        {
          "stage": "Conclusion",
          "time": "5 mins",
          "teacher_actions": "Summarize key learning points",
          "student_actions": "Reflect and ask remaining questions",
          "evaluation": "Exit ticket assignment",
          "resources": "Exit ticket forms"
        }
      ]
    },
    "addedResources": [
      {
        "crossword_title": "Ecosystem Vocabulary Crossword",
        "resourceType": "game",
        "type": "crossword",
        "questions": [
          {
            "question": "Organism that produces its own food using sunlight",
            "answer": "Producer"
          },
          {
            "question": "Interaction between two species benefiting both",
            "answer": "Mutualism"
          }
        ]
      },
      {
        "wordsearch_title": "Ecosystems Word Search",
        "type": "word_search",
        "resourceType": "game",
        "words": [
          "Biodiversity",
          "Niche",
          "Habitat",
          "Population",
          "Community"
        ]
      },
      {
        "matching_title": "Match Terms with Definitions",
        "resourceType": "game",
        "type": "matching",
        "sets": [
          {
            "WordsAndDefinitions": [
              {
                "Word": "Herbivore",
                "Definition": "Animal that eats plants"
              },
              {
                "Word": "Carnivore",
                "Definition": "Animal that eats other animals"
              }
            ]
          }
        ]
      }
    ]
  }
}

:bullseye: Summary of Unique anyOf Purposes

Schema Location Subschema Key Unique Purpose
command noCommand Explicitly indicates no command
command genericCommand General-purpose command with arguments
command renameChatCommand Specifically renames the chat
addedResources matchingResource Matching pairs educational game
addedResources anagramResource Anagram-based educational game
addedResources crosswordResource Crossword puzzle educational game
addedResources wordSearchResource Word search puzzle educational game
addedResources balloonPopResource Balloon pop interactive game
addedResources mathRaceResource Math race interactive game
addedResources wordSnakeResource Word snake interactive game
lesson_progress.items.resources Plain string Direct descriptive resource
lesson_progress.items.resources "[resource]" Placeholder for dynamic resource

:warning: Why This Matters

Each subschema within an anyOf must have a clearly defined, unique purpose and distinguishing characteristic. This ensures the AI can reliably select the correct schema based on the intended output, and your application can correctly interpret and handle each structured response.

From Structured Outputs documentation

  • For anyOf, the nested schemas must each be a valid JSON Schema per this subset, and

assisted by 12k input context gpt-4.5-preview, and my extensive training input on making an expert at response formats

1 Like