GPT-4o doesn't consistently respect JSON schema on tool use

I’m using {“tool_choice”: “required”} to guarantee a Pydantic structured response.

HOWEVER, I noticed 4o does not consistently respect the JSON schema. For example, if it chooses to return markdown, it will completely ignore the required json schema. GPT-4 Turbo on the other hand does this very consistently, even when returning markdown, it will return the markdown content in the required JSON field.

Does anyone else experienced this?

1 Like

Yes, I just experienced that as well - where gpt-4o does not respect the JSON mode requirement.

I have similar results, using the same instructions across gpt3.5-1106, gpt4turbo, and gpt4o, only gpt4o is consisently bad at JSON output, I dont use function calling and instead do JSON output schema instructions, and its pretty bad. usually using the json schema instructions itself instead of the intended JSON structure.

I used to get this but no more what do your schema instructions look like?

This format works for me

{"qux":String,"foo":Int, "baz":[String], "bat":<Do|Ray|Me>}

Where baz an array and bat an enum

Im just giving it 2 simple keys like so:

{ “answer”: string, “sources”: }

Note, i also include a doc string and description for each as its from a Pydantic class

This one is based on the early JSON instructions from langchain

The output should be formatted as a JSON instance that conforms to the JSON schema below.

Here is the output schema:


{
  "type": "object",
  "properties": {
    "task_details": {
      "type": "string",
      "description": "task details for the risk assessment report formatted as categorized bullet points following the outline: Brief Overview, Context of Task, Nature of Chemicals/Materials Involved, Equipment Required, Personnel Involved, Objective and Output."
    },
    "health_surveillance": {
      "type": "string",
      "description": "Systematic, regular and appropriate procedures to detect and act on early signs of work-related ill health for people exposed to certain health risks."
    }
  },
  "required": [
    "task_details",
    "health_surveillance"
  ]
}

I have even deeper and wider instructions particularly one like this deep

The output should be formatted as a JSON instance that conforms to the JSON schema below.

Here is the output schema:


{
  "type": "object",
  "properties": {
    "task": {
      "type": "string",
      "description": "Name of the task for the risk assessment report."
    },
    "process": {
      "type": "string",
      "description": "High level process of the task."
    },
    "people_affected": {
      "type": "string",
      "description": "People that may be affected by the task."
    },
    "hazard_controls": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "hazard": {
            "type": "string",
            "description": "A description of the hazard."
          },
          "hazard_sign": {
            "type": "object",
            "properties": {
              "sign_id": {
                "type": "integer",
                "description": "Database ID of the sign."
              },
              "title": {
                "type": "string",
                "description": "Database Title of the sign."
              },
              "category": {
                "type": "string",
                "description": "Database Category of the sign."
              },
              "image_name": {
                "type": "string",
                "description": "Database Image name of the sign."
              }
            },
            "required": [
              "sign_id",
              "title",
              "category",
              "image_name"
            ]
          },
          "controls": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "sign": {
                  "type": "object",
                  "properties": {
                    "sign_id": {
                      "type": "integer",
                      "description": "Database ID of the sign."
                    },
                    "title": {
                      "type": "string",
                      "description": "Database Title of the sign."
                    },
                    "category": {
                      "type": "string",
                      "description": "Database Category of the sign."
                    },
                    "image_name": {
                      "type": "string",
                      "description": "Database Image name of the sign."
                    }
                  },
                  "required": [
                    "sign_id",
                    "title",
                    "category",
                    "image_name"
                  ]
                },
                "description": {
                  "type": "string",
                  "description": "Description of the control."
                }
              },
              "required": [
                "sign",
                "description"
              ]
            },
            "description": "Controls for the hazard"
          },
          "emergency_controls": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "sign": {
                  "type": "object",
                  "properties": {
                    "sign_id": {
                      "type": "integer",
                      "description": "Database ID of the sign."
                    },
                    "title": {
                      "type": "string",
                      "description": "Database Title of the sign."
                    },
                    "category": {
                      "type": "string",
                      "description": "Database Category of the sign."
                    },
                    "image_name": {
                      "type": "string",
                      "description": "Database Image name of the sign."
                    }
                  },
                  "required": [
                    "sign_id",
                    "title",
                    "category",
                    "image_name"
                  ]
                },
                "description": {
                  "type": "string",
                  "description": "Description of the control."
                }
              },
              "required": [
                "sign",
                "description"
              ]
            },
            "description": "Emergency controls for the hazard"
          },
          "risk_matrix": {
            "type": "object",
            "properties": {
              "before_control_severity": {
                "type": "integer",
                "description": "A score from 1 to 5 of the severity of the hazard before the controls are applied."
              },
              "before_control_likelihood": {
                "type": "integer",
                "description": "A score from 1 to 5 of the likelihood of the hazard before the controls are applied."
              },
              "after_control_severity": {
                "type": "integer",
                "description": "A score from 1 to 5 of the severity of the hazard after the controls are applied."
              },
              "after_control_likelihood": {
                "type": "integer",
                "description": "A score from 1 to 5 of the likelihood of the hazard after the controls are applied."
              }
            },
            "required": [
              "before_control_severity",
              "before_control_likelihood",
              "after_control_severity",
              "after_control_likelihood"
            ]
          }
        },
        "required": [
          "hazard",
          "hazard_sign",
          "controls",
          "emergency_controls",
          "risk_matrix"
        ]
      },
      "description": "Hazard controls for the task."
    }
  },
  "required": [
    "task",
    "process",
    "people_affected",
    "hazard_controls"
  ]
}

Gpt4o for the first schema will follow the output schema, and put its answers on the “description” field.

I find that using Typescript to define a JSON Schema is much easier than providing it JSON to follow.

For example, this works flawlessly for me:

Now consider the following TypeScript Interface for the JSON schema:

interface Basics {
    name: string;
    summary: string;   
}

Write the basics section according to the Basics schema. On the response, include only the JSON.

Then, I do some checks after the output has been received:

def clean_json_output(output):
    output = output.strip()
    if output.startswith("```json"):
        output = output[7:]
    if output.endswith("```"):
        output = output[:-3]
    cleaned_output = output.strip()

    try:
        json_data = json.loads(cleaned_output)
    except json.JSONDecodeError as e:
        logging.error(f"JSON decoding error: {e}")
        return cleaned_output

    def clean_json(data):
        if isinstance(data, dict):
            return {key: clean_json(value) for key, value in data.items()}
        elif isinstance(data, list):
            return [clean_json(item) for item in data]
        elif isinstance(data, str):
            return "" if data.lower() in ["unknown", "na", "null"] else data
        else:
            return data

    cleaned_json_data = clean_json(json_data)
    cleaned_output = json.dumps(cleaned_json_data, ensure_ascii=False)

    return cleaned_output

This is just how I do it. Hope this helps :smiley:

1 Like

I am experiencing similar issues.
My prompts that have GPT4t follow a JSON structure with 100% success rate do not work well at all for GPT4o. Very disappointing regression.

I wonder what has caused this?

I encountered a similar problem last time, the outputs gave errors in almost all of them and I had to find the errors and solve them. I guess they will solve it with a micro-update