I’m using {“tool_choice”: “required”} to guarantee a Pydantic structured response.
HOWEVER, I noticed 4o does not consistently respect the JSON schema. For example, if it chooses to return markdown, it will completely ignore the required json schema. GPT-4 Turbo on the other hand does this very consistently, even when returning markdown, it will return the markdown content in the required JSON field.
Does anyone else experienced this?
Yes, I just experienced that as well - where gpt-4o does not respect the JSON mode requirement.
I have similar results, using the same instructions across gpt3.5-1106, gpt4turbo, and gpt4o, only gpt4o is consisently bad at JSON output, I dont use function calling and instead do JSON output schema instructions, and its pretty bad. usually using the json schema instructions itself instead of the intended JSON structure.
I used to get this but no more what do your schema instructions look like?
This format works for me
{"qux":String,"foo":Int, "baz":[String], "bat":<Do|Ray|Me>}
Where baz an array and bat an enum
Im just giving it 2 simple keys like so:
{ “answer”: string, “sources”: }
Note, i also include a doc string and description for each as its from a Pydantic class
This one is based on the early JSON instructions from langchain
The output should be formatted as a JSON instance that conforms to the JSON schema below.
Here is the output schema:
{
"type": "object",
"properties": {
"task_details": {
"type": "string",
"description": "task details for the risk assessment report formatted as categorized bullet points following the outline: Brief Overview, Context of Task, Nature of Chemicals/Materials Involved, Equipment Required, Personnel Involved, Objective and Output."
},
"health_surveillance": {
"type": "string",
"description": "Systematic, regular and appropriate procedures to detect and act on early signs of work-related ill health for people exposed to certain health risks."
}
},
"required": [
"task_details",
"health_surveillance"
]
}
I have even deeper and wider instructions particularly one like this deep
The output should be formatted as a JSON instance that conforms to the JSON schema below.
Here is the output schema:
{
"type": "object",
"properties": {
"task": {
"type": "string",
"description": "Name of the task for the risk assessment report."
},
"process": {
"type": "string",
"description": "High level process of the task."
},
"people_affected": {
"type": "string",
"description": "People that may be affected by the task."
},
"hazard_controls": {
"type": "array",
"items": {
"type": "object",
"properties": {
"hazard": {
"type": "string",
"description": "A description of the hazard."
},
"hazard_sign": {
"type": "object",
"properties": {
"sign_id": {
"type": "integer",
"description": "Database ID of the sign."
},
"title": {
"type": "string",
"description": "Database Title of the sign."
},
"category": {
"type": "string",
"description": "Database Category of the sign."
},
"image_name": {
"type": "string",
"description": "Database Image name of the sign."
}
},
"required": [
"sign_id",
"title",
"category",
"image_name"
]
},
"controls": {
"type": "array",
"items": {
"type": "object",
"properties": {
"sign": {
"type": "object",
"properties": {
"sign_id": {
"type": "integer",
"description": "Database ID of the sign."
},
"title": {
"type": "string",
"description": "Database Title of the sign."
},
"category": {
"type": "string",
"description": "Database Category of the sign."
},
"image_name": {
"type": "string",
"description": "Database Image name of the sign."
}
},
"required": [
"sign_id",
"title",
"category",
"image_name"
]
},
"description": {
"type": "string",
"description": "Description of the control."
}
},
"required": [
"sign",
"description"
]
},
"description": "Controls for the hazard"
},
"emergency_controls": {
"type": "array",
"items": {
"type": "object",
"properties": {
"sign": {
"type": "object",
"properties": {
"sign_id": {
"type": "integer",
"description": "Database ID of the sign."
},
"title": {
"type": "string",
"description": "Database Title of the sign."
},
"category": {
"type": "string",
"description": "Database Category of the sign."
},
"image_name": {
"type": "string",
"description": "Database Image name of the sign."
}
},
"required": [
"sign_id",
"title",
"category",
"image_name"
]
},
"description": {
"type": "string",
"description": "Description of the control."
}
},
"required": [
"sign",
"description"
]
},
"description": "Emergency controls for the hazard"
},
"risk_matrix": {
"type": "object",
"properties": {
"before_control_severity": {
"type": "integer",
"description": "A score from 1 to 5 of the severity of the hazard before the controls are applied."
},
"before_control_likelihood": {
"type": "integer",
"description": "A score from 1 to 5 of the likelihood of the hazard before the controls are applied."
},
"after_control_severity": {
"type": "integer",
"description": "A score from 1 to 5 of the severity of the hazard after the controls are applied."
},
"after_control_likelihood": {
"type": "integer",
"description": "A score from 1 to 5 of the likelihood of the hazard after the controls are applied."
}
},
"required": [
"before_control_severity",
"before_control_likelihood",
"after_control_severity",
"after_control_likelihood"
]
}
},
"required": [
"hazard",
"hazard_sign",
"controls",
"emergency_controls",
"risk_matrix"
]
},
"description": "Hazard controls for the task."
}
},
"required": [
"task",
"process",
"people_affected",
"hazard_controls"
]
}
Gpt4o for the first schema will follow the output schema, and put its answers on the “description” field.
I find that using Typescript to define a JSON Schema is much easier than providing it JSON to follow.
For example, this works flawlessly for me:
Now consider the following TypeScript Interface for the JSON schema:
interface Basics {
name: string;
summary: string;
}
Write the basics section according to the Basics schema. On the response, include only the JSON.
Then, I do some checks after the output has been received:
def clean_json_output(output):
output = output.strip()
if output.startswith("```json"):
output = output[7:]
if output.endswith("```"):
output = output[:-3]
cleaned_output = output.strip()
try:
json_data = json.loads(cleaned_output)
except json.JSONDecodeError as e:
logging.error(f"JSON decoding error: {e}")
return cleaned_output
def clean_json(data):
if isinstance(data, dict):
return {key: clean_json(value) for key, value in data.items()}
elif isinstance(data, list):
return [clean_json(item) for item in data]
elif isinstance(data, str):
return "" if data.lower() in ["unknown", "na", "null"] else data
else:
return data
cleaned_json_data = clean_json(json_data)
cleaned_output = json.dumps(cleaned_json_data, ensure_ascii=False)
return cleaned_output
This is just how I do it. Hope this helps 
I am experiencing similar issues.
My prompts that have GPT4t follow a JSON structure with 100% success rate do not work well at all for GPT4o. Very disappointing regression.
I wonder what has caused this?
I encountered a similar problem last time, the outputs gave errors in almost all of them and I had to find the errors and solve them. I guess they will solve it with a micro-update
Same problem here. The new model dont even call functions correctly, adding some “functionCall” or “tool_use” properties, instead of return on the appropriate place.
I start getting problems when “the keys are not fixed”.
For example, if you want to extract information about some entities, where the entity name then it sometimes… just doesn’t work.
I want to extract names of foods from the image and their quantity. Then if I write:
Prompt:
--- other instructions ---
## Follow the response format below in JSON
` ` `
{
"food_name_1": "<integer> <unit>", // name of food with quantity (e.g., pizza: 100 grams)
"food_name_2": "<integer> <unit>",
"food_name_N": "<integer> <unit>",
}
` ` `
Follow the above structure for all food items. Make sure no food item is omitted.
But sometimes, it doesn’t do the expected job and returns the following:
{
"food_name_1": "Pizza 100 grams",
"food_name_2": "French fries 40 grams",
"food_name_N": "Red wine 200 ml",
}
This breaks the system. I mean of course the above example is a toy but there are times when you need to have “dynamic keys”.
In such times, I got success with using the <brackets> in the keys.
Prompt:
--- other instructions ---
## Follow the response format below in JSON
` ` `
{
"<food_name_1>": "<integer> <unit>", // name of food with quantity (e.g., pizza: 100 grams)
"<food_name_2>": "<integer> <unit>",
"<food_name_N>": "<integer> <unit>",
}
` ` `
Follow the above structure for all food items. Make sure no food item is omitted.
And… that works.