Tool calls rejecting valid JSON from correct specification

API Bugs
1

I’m trying to use tool calling with a 2 level deep structure for the parameters, but the API’s preprocessing is returning the following error before it even reaches the LLM:

{
  "error": {
    "message": "Invalid schema for function 'Setup'. Please ensure it is a valid JSON Schema.",
    "type": "invalid_request_error",
    "param": "tools[0].function.parameters",
    "code": "invalid_function_parameters"
  }
}

I’ve taken the following steps:

  • Updated to latest version of the json schema library I am using, and ensured it is generating the 2020-12 version the OpenAI API supports (per another reply from an OpenAI employee in this forum that it’s not letting me link here)
  • Ran my schema through a schema validator :white_check_mark:
  • Ran the schema through the most popular Python schema validator library I could find, jsonschema :white_check_mark:
  • Called another OpenAI API-compliant LLM API (in this case, Groq’s implementation), and LLama 3 70b responded and in the correct schema (just not a very smart answer compared to gpt4-o) :white_check_mark:

I’m convinced at this point that something is wrong with the way the schema validation is being done in OpenAI’s implementation in particular, or with whatever library is being used. I’ve created the simplest, stripped down test case I can, and it seems to have something to do with the nested oneOf object.

If I convert the child object in this test case to a concrete type instead of a oneOf, it goes through fine. Here is a full Chat Completion API call that can be run through curl to replicate this:

{
  "model": "gpt-4o",
  "messages": [
    {
      "role": "system",
      "content": "Call the function provided with test data"
    }
  ],
  "max_tokens": null,
  "temperature": 0.0,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "Setup",
        "parameters": {
          "$defs": {
            "BasicCommands": {
              "oneOf": [
                {
                  "properties": {
                    "type": {
                      "enum": [
                        "FirstCommandType"
                      ],
                      "type": "string"
                    }
                  },
                  "required": [
                    "type"
                  ],
                  "type": "object"
                },
                {
                  "properties": {
                    "type": {
                      "enum": [
                        "SecondCommandType"
                      ],
                      "type": "string"
                    }
                  },
                  "required": [
                    "type"
                  ],
                  "type": "object"
                }
              ]
            },
            "Instruction": {
              "properties": {
                "commands": {
                  "items": {
                    "$ref": "#/$defs/BasicCommands"
                  },
                  "type": "array"
                }
              },
              "required": [
                "commands"
              ],
              "type": "object"
            }
          },
          "$schema": "https://json-schema.org/draft/2020-12/schema",
          "properties": {
            "instructions": {
              "items": {
                "$ref": "#/$defs/Instruction"
              },
              "type": "array"
            }
          },
          "required": [
            "instructions"
          ],
          "title": "Setup",
          "type": "object"
        }
      }
    }
  ],
  "tool_choice": {
    "type": "function",
    "function": {
      "name": "Setup"
    }
  }
}

Based on the fact that a less capable model, Llama 3 70b was able to return correct, compliant JSON in another API, I see no reason GPT4 class models wouldn’t be able to handle this structure, so I can’t imagine why invalidating this type of schema would be intentional. (also fwiw, I’m currently working around this by utilizing JSON mode instead and passing the schema in context, and it works fine there)

Would love if you guys could swap out your json schema validator so I can finish my project! Assuming you’re using Python, the jsonschema library works fine with this on the latest version.

2 Likes