Structured output generating meaningless data

API Bugs
1

We are currently testing structured output with GPT-5.2. Using the prompt “Invoice data,” we generated a JSON schema. Our system prompt is:

“Extract data from input. If you cannot extract meaningful data, leave all fields at their default values.”

However, when we provide empty or meaningless input, the model still returns a populated JSON object containing nonsensical or fabricated values (see attached screenshot).

Can anybody clarify whether this behavior is expected, or if there may be an issue with structured output handling in this scenario?

image
image1920×1200 296 KB

2

We just tested the same scenario using the API directly (not the playground) and we were not able to reproduce the scenario. Still, very strange behavior of the model/playground. Any idea?

3

A structured output FORCES the AI to fill out the JSON. The only decision it gets to make is after it is into “invoice_number” there, and then the continuation that comes in the remaining fields as the form being more completed then makes for more plausible input into fields instead of “BAD BAD BAD ABORT” or whatever you expect it to write.

It seems you did get default answers like you asked for: 1970.

What you will want to do, however, is have an initial object that has an anyOf, with two sub-schemas. Then the AI does have a choice. The second can be an “insufficient_info” or “missing_extracted_fields” type, or an error, a refusal, what have you.

Then you have a shape to “detect” in handling the output instead of your JSON being an indistinguishable “oopsie” because you didn’t send any user message or task.

2 Likes
4

In our case the “garbage” value is in a field typed as:

  • type: “string”

  • format: “email”

My understanding was that this combination is supported for OpenAI structured outputs, so I expected the model to only emit values that match the email format. The value we’re seeing is not a valid email address.

Is format: “email” treated as a strictly validated constraint during structured output generation, or is it only a “soft” hint/annotation? If it’s not enforced, is the recommended approach to use a stricter constraint (e.g., a pattern) and/or validate client-side?

5

I have no expectation that a string is anything but a string. A string is where the constraints of generating the strict form of the JSON are released so the AI can write whatever it wants until an unescaped double-quote is putting the enforcement back in play.

Consider that the AI produces word-like tokens one at a time. Enforcing generated logits of an email address would be near-impossible - is the token for an “@” symbol going to be forced into the output after “jack”, “jackblack”, “jackblack42”, “jackblack42+” etc.

Here’s a plausible JSON schema, that lets us talk to the AI while it also has been provided the key in question for an email address:

{
  "name": "output",
  "schema": {
    "type": "object",
    "properties": {
      "response_to_user": {
        "type": "string",
        "description": "Message to present to the user"
      },
      "extracted_email": {
        "type": "string",
        "description": "Email address extracted from user input",
        "format": "email"
      }
    },
    "required": [
      "response_to_user",
      "extracted_email"
    ],
    "additionalProperties": false
  },
  "strict": true
}

What is placed into AI context for its understanding in addition to the enforcement:

# Response Formats

## output

{"type":"object","properties":{"response_to_user":{"type":"string","description":"Message to present to the user"},"extracted_email":{"type":"string","description":"Email address extracted from user input"}}}

You will see no “format” - just one of many things you can ordinarily write in a draft schema to try to tune up or constrain values of a string or number, that in the subset supported, that at best become a comment, and here, are dropped.

If you were still curious: I told the AI to write “null” as the email and it did, and then a never-ending problem with the Responses API specifically: the output goes into a nonstop loop until max_output_tokens instead of a stop sequence terminating the output.

image
image568×340 24.2 KB

Or: are they waiting for an @ that will never come and ignoring the quote?

So like my description for email address in the schema, there alone, you’ll have to increase your prompting game so that the AI only produces seen reality and is instructed what is acceptable as a “none” string.

FYI: the same schema does do okay also:

image
image555×311 23.8 KB