Json_schema response format handling changed

API Bugs
,
1

non strict json schema seems to have changed in the last 24hrs. I have a schema that used to work being generated in rust from the serde_json crate. The response_format is set to jsonschema and strict=false. Frequently I get either responses that don’t match the schema but actually look like they are the schema definition itself. Sometimes with data but usually empty.

When I changed my schema to manually defined and strict=true, it is now working again. I had to change my schema because the serde_json output caused this error when strict=true:
“Invalid schema for response_format ‘equipment’: In context=(), ‘additionalProperties’ is required to be supplied and to be false.”,

I prob should have had strict=true before but it was successful 99% time with strict=false with better data and then in the last 48hrs went to 90% failing to parse correctly.

UPDATE:
The structured outputs handling of json schemas have just seemed to have changed. I posted a couple examples but the rust serde_json schema with strict=false is no longer giving results that match the schema when it had until ~48 hrs ago. The results are formatted like a schema not like a json instance of that schema. Manually creating the schema has worked around the issue but something must have changed with openai because we had no changes on our side and the behavior is dramatic. This is an example expected result:

"{"attributes":{"name":"Oil Boiler","type":"Boiler","brand":"Peerless","model":"EC-03","serial":"337780220002","category":"HVAC","room":"Utility Room"},"issues":[{"description":"Burner fails to ignite due to a faulty igniter or clogged nozzle.","difficulty":"High","name":"Failing to Ignite","parts":["Igniter","Nozzle"]},{"description":"Kettling noise caused by trapped air or limescale.","difficulty":"Medium","name":"Kettling","parts":["Heat Exchanger"]},{"description":"Water leaks due to pressure valve failure.","difficulty":"Medium","name":"Leaking","parts":["Pressure Valve"]}],"maintenance":[{"description":"Check for leaks and ensure proper pressure.","frequency":"Monthly","name":"Pressure Check","parts":[]},{"description":"Clean the burner and inspect the igniter for soot and residue buildup.","frequency":"Quarterly","name":"Burner Cleaning","parts":["Igniter","Nozzle"]},{"description":"Flush the system to remove limescale and debris.","frequency":"Annually","name":"System Flush","parts":["Heat Exchanger"]}],"parts":[{"description":"Used to ignite the fuel mixture for combustion.","name":"Igniter"},{"description":"Controls the flow of heated water or steam.","name":"Pressure Valve"},{"description":"Channels and disperses the input fuel within the burner compartment.","name":"Nozzle"}]}"

Instead we got the result below that has fields nested under a properties fieldwith atype` field instead of a value. In short, OpenAI is responding with a schema instead of json.

{"title":"AiEquipmentResult","type":"object","properties":{"attributes":{"type":"object","properties":{"brand":{"type":"string"},"model":{"type":"string"},"room":{"type":"string"},"serial":{"type":"string"},"type":{"type":"string"},"category":{"type":"string"}},"additionalProperties":false},"issues":{"type":"array","items":{"$ref":"#/definitions/EquipmentIssue"}},"maintenance":{"type":"array","items":{"$ref":"#/definitions/EquipmentMaintenance"}},"parts":{"type":"array","items":{"$ref":"#/definitions/EquipmentPart"}}},"definitions":{"EquipmentIssue":{"type":"object","properties":{"name":{"type":"string","nullable":true},"description":{"type":"string","nullable":true},"difficulty":{"type":"string","nullable":true},"parts":{"type":"array","items":{"type":"string"}}}},"EquipmentMaintenance":{"type":"object","properties":{"name":{"type":"string","nullable":true},"frequency":{"anyOf":[{"$ref":"#/definitions/MaintenanceSchedule"},{"type":"null"}]},"description":{"type":"string","nullable":true},"parts":{"type":"array","items":{"type":"string"}}}},"EquipmentPart":{"type":"object","properties":{"name":{"type":"string","nullable":true},"description":{"type":"string","nullable":true}}},"MaintenanceSchedule":{"type":"string","enum":["Annually","Quarterly","Monthly","Weekly","AsNeeded","Unknown"]}}}"
4 Likes
2

Why not just use strict: true?

OpenAI is constantly changing their models and it’s been noticed that it’s attention has been shifted.

It’s interesting that you have better results with strict: false

1 Like
3

Here’s what’s interesting: using {“response_format”: “json_schema”, …} expects the AI model to simply know how to use what is placed in to context, with no other instructions at all.

input schema:

{
  "name": "my_output_schema_name",
  "strict": false,
  "schema": {
    "type": "object",
    "properties": {
      "response_to_developer": {
        "type": "string",
        "description": "The response message."
      }
    },
    "required": [
      "response_to_developer"
    ],
    "additionalProperties": false
  }
}

Schema appearing in AI context

[developer role message appears here]

# Response Formats

## my_output_schema_name

{"type":"object","properties":{"response_to_developer":{"type":"string","description":"The response message."}}}

You are trained on data up to October 2023.

You can see the AI gets the name, and has to write it again (count the response tokens!).

A complex schema can get pretty hairy, and you should see that it will validate and word with strict:true, which is expected to be within the AI’s post-training capabilities, even if part of that ability is “forced”. Only after that is proven would I be doing your “strict:false” modifications, like removing one of the keys from required.

But most of all: I would treat the AI like it knows nothing, can’t follow a dang instruction (like how ChatGPT was damaged) and wasn’t tuned, and give enough instructions at the end of a system message to utilize the schema anyway, such as were it placed that way with json_object, and anticipate what the AI must actually produce and send to.

Your JSON-only output is being parsed, and must follow an exact validated JSON schema. The validation schema which you must follow and send to is instructed in “Response Formats”. You focus on filling every key appearing in required, while carefully evaluating if any other key not required would also be useful. Output is only to this response format.

4

Thanks for the reply. I can’t say I completely follow all of your feedback. Are you suggesting adding the schema to the context in the system/user mesasge? I have gone back and forth with that.

I’m not clear how the response_format json_schema is used. If I understand what you’re saying, it isn’t added automatically by openai to the context and only used to guide the token generation. I can certainly add it to the context even with strict=true. Just not clear if that’s intended because the docs are unclear about how the respone format is treated.

With strict=true, I tended to get more failed responses but perhaps I didn’t have enough specification of nullable attributes in the schema. Now that I’m manually specifying the schema that should be less of an issue.

So switching to strict=true with a manual schema, seems to do the trick because rust serde_json doesn’t produce acceptable schemas for openai. I’ll also play with adding it to the context and trying to elicit more complete generation.

5

Using strict:false is very similar to were you to place the schema yourself. It is placed as I show in the first system message for you. The AI doesn’t have to follow it “strictly” without the strict option, as the enforcement method of exactly what keys to produce and how a JSON must be constructed is not turned on.

What I mean to communicate and demonstrate is that schema is placed for you when you use the response format parameter, but doesn’t come with accompanying instructions - the kind of instructions where you’d say “follow this schema” were you to write your own prompt. The behavior must be all by the AI’s pre-understanding of what the schema “Response formats” means.

I say: add more instruction of why the AI must always write JSON, where it is sending the output to, and the purpose. Don’t rely on magic.

strict:true should mean that you always get the JSON compliant with the schema, able to be validated. It is only within the values themselves that the AI can produce something “wrong”.

1 Like