Invalid JSON response when using Structured Output

API Bugs
,
1

I’m using the assistant API to run a thread and get a response back in JSON format, using a specified schema (json_schema format). In some cases (maybe 1% of the time, and not reproducible except randomly), the response contains invalid JSON. For example, here is a simplified version of the response that contains invalid json:

{
  "summary": "blah blah",
  "details": [
  ]
},
"notes": "blah blah"
}

There is an extra } which is causing the issue. The response should have been:

{
  "summary": "blah blah",
  "details": [
  ],
  "notes": "blah blah"
}

… and I am using strict: true in my schema definitions.

Has anyone else experienced something similar? I assumed that when using the json_schema output type, the output would always be valid JSON - not just 99% of the time.

2 Likes
2

Yes 8 from the 10871 calls made, using the https://api.openai.com/v1/chat/completions API endpoint. To be a bit more specific, the invalid JSON was the stringified JSON data within the choices[0].message.content property

The workaround for me is to check if the stringified JSON data returned is actually parsable, if not do a retry.

"choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{ \"date\" : [ <RANDOM CUTOFF> }"
2 Likes
3

Thank you. I was going to implement a similar retry strategy. Also, looking at the documentation for structured outputs, it says this:

Structured Outputs can still contain mistakes. If you see mistakes, try adjusting your instructions, [etc…]

4

@CycleMost @dal1e

Are you prompting the model in the system message to respond with a valid JSON object?

2 Likes
5

@sps Currently not, but it is a trivial to add. But as I mentioned, only 8 of the 10871 calls, that is statistically reasonable, one could argue that it aligns with the expected rare event probability. Currently my system prompt - among other instructions only - instructs to stick to the schema provided.

[...]
- Strictly adhere to the described <XYZ> schema in the response.
[...]