Limiting json structured output

API
1

Hello, I’ve recently came across the feature in the API of json_schema allowing us to limit the model into a certain schema. Im limited by the time i get to response for a query therefore every unnecessary output token used is very painful. Currently I have a certain json output where not all attributes always has to take a rule. I know I can set the attribute to be “null” and the value of the attribute will be null if not needed but the key will still occur.
for example:
lets say this is the output type:
{
“a”:int,
“b”:int,
“c”:int
}
and for this a query the result can use only “a”.
the best case the LLM can return:
{
“a”: 3
}

but with the formal “null” option I get:
{
“a”:3,
“b”: null,
“c”: null
}

I tried “hacking” around using different json classes and making a list of union of types each representing a specific attribute which worked fine (meaning the irrelevant keys were not in the LLM results) but I couldn’t make it a clean JSON structure. Is there a formal solution to this?

2

When strict:true, all keys must be required. Also, you can’t try to trick your way around this by having an array of objects that the AI can arbitrarily name or grow.

The solution while strict:true and ANY keys given in a schema are required would seem to be to use a schema based on anyOf subschemas.

{
  "name": "single_key_choice_number",
  "schema": {
    "type": "object",
    "properties": {
      "output": {
        "anyOf": [
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number",
                "description": "Number value for key 'a'."
              }
            },
            "required": [
              "a"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "b": {
                "type": "number",
                "description": "Number value for key 'b'."
              }
            },
            "required": [
              "b"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "c": {
                "type": "number",
                "description": "Number value for key 'c'."
              }
            },
            "required": [
              "c"
            ],
            "additionalProperties": false
          }
        ],
        "description": "Object with exactly one key from 'a', 'b', or 'c', each holding a number."
      }
    },
    "required": [
      "output"
    ],
    "additionalProperties": false
  },
  "strict": true
}

image
image748×307 9.66 KB

The “container” is mandatory, as anyOf cannot be a top-level choice, but the total output being under 0.6s and containing only the key you want proves the technique. Then you can fill the AI with a bunch of choices to decide between.

If you really want to cut down on the generation at the token-optimized level, you can give the schema name and the keys one-token length names that work with and without leading space.

Amusing myself with application (where it is also fun to penalize the “right” answer key name to impossible with logit_bias and see the AI justify what it had to choose..)

image
image600×328 7.73 KB

1 Like
3

You can also try JSON mode.

It is less deterministic, so only guarantee is that you will get a JSON response. But you have more freedom to guide the model into your expected custom output.

1 Like
4

Maybe your solution is in a different area?

Have you thought about optimizing your workflow to simplify tasks and allow parallel processing by smaller/faster models? Depending on the app and use case, it’s often way more efficient than token counting.

Also simpler, and “rigid” schemas with good description fields often produce better results (especially on smaller models).

So before spending time in trimming tokens at the schema layer, I would try to look into the workflow, ideally with a fresh eye.

5

First of all, thank you for your reply. I think I may not have explained myself clearly. My goal is to allow the LLM to generate any combination of the 2^{# attributes} possible JSON outputs to answer a query. In other words, the LLM should decide whether to include each attribute or not—similar to allowing an attribute to be null, which effectively means it is omitted.

For example, using your schema definition, consider the following prompt:

You need to output a JSON object with at most three attributes: a, b, and c, following these rules:

  1. Generate a random number between 0 and 100 and assign it to attribute a.

  2. If a is greater than 10 but less than 20, include attribute b and generate a random number.

  3. If a is greater than 15, include attribute c and generate a random number.

With this logic, all of the following outputs would be valid:

  • {'a': 3}

  • {'a': 13, 'b': 125}

  • {'a': 17, 'b': 50, 'c': 60}

  • {'a': 25, 'c': 70}

The broader goal is to define a regex for each attribute to constrain its values. The LLM should then decide whether to include each attribute, but if it does, the value must comply with the corresponding regex.

6

Thank you for the suggestion, Im aware or this option. Currently our setup is using this feature but (I think) because we have a complex task + large context windows usage that the LLM doesn’t follow our output limitations precisely. So this is what draw me into the json_schema feature.

7

Thank you for the suggestion. We are aware that this could be an option and do implement parallel processing for some tasks using smaller, faster models. However, the main task remains complex and interdependent, requiring reasoning and problem-solving capabilities that we couldn’t get using smaller models.

1 Like
8

Personally, when I come across complex tasks, I try to see the following:

  1. How do humans solve that task? Here I look for the “tree” of steps as a whole approach to the workflow.
  2. Try to breakdown “branches” so that each task has as few “focus points” as possible (simplify tasks).
  3. For each of the tasks I define: a) “global goal”, b) task goal, c) output, d) focus, e) inputs. And walk backwards from the final goal to each branch points.
  4. Then, try to see if the branches can be reordered so that I can reuse outputs as inputs further down possibly with parallel processing.
  5. Normally, this is where you get the clear picture of what can be done by smaller models/in parallel.

Then, there is another consideration: the app approach itself. How do you deliver the value? Have you promised the “instant” solution or can it be delivered in asynchronous mode?

But then, all of the above is just speculations as I don’t know what your task is. Do you have at least the “tree” of operations defined with single-focus tasks? Do you have DTOs defined to be passed between the tasks?

9

Best

I think the solution here is simply to use the [“number”, “null”] union of output types.

You’re going to get more keys, but then at least each key is being evaluated for its need to be output instead of a premature “that was enough output”. A model like gpt-4.1-mini can produce 80 tokens a second, so unless you are using these keys like enums with a hundred, you won’t have much more latency by using null values.

You can present the key ordering so that any dependencies are output first, so the AI isn’t producing other information based on a choice that hasn’t been made yet.

What you want - but will confuse and cost in input

  • make a separate anyOf schema for every possible combination of final output. One for [a], one for [b], one for [a,b], one for [c], …

The number of subschemas blows up quickly…

number of options vs. number of schemas needed (any non-empty combination)

Options (n) Schemas needed (2^n − 1)
1 1
2 3
3 7
4 15
5 31
6 63
7 127
8 255
9 511
10 1023

so here, try the option with 5 ==> 31 anyOf schemas.

image
image796×306 9.13 KB

{
  "name": "five_key_combinations_numbers_a_to_e",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": {
      "output": {
        "anyOf": [
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              }
            },
            "required": [
              "a"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "b": {
                "type": "number"
              }
            },
            "required": [
              "b"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "c": {
                "type": "number"
              }
            },
            "required": [
              "c"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "d": {
                "type": "number"
              }
            },
            "required": [
              "d"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "e": {
                "type": "number"
              }
            },
            "required": [
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "b": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "b"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "c": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "c"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "d": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "d"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "b": {
                "type": "number"
              },
              "c": {
                "type": "number"
              }
            },
            "required": [
              "b",
              "c"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "b": {
                "type": "number"
              },
              "d": {
                "type": "number"
              }
            },
            "required": [
              "b",
              "d"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "b": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "b",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "c": {
                "type": "number"
              },
              "d": {
                "type": "number"
              }
            },
            "required": [
              "c",
              "d"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "c": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "c",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "d": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "d",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "b": {
                "type": "number"
              },
              "c": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "b",
              "c"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "b": {
                "type": "number"
              },
              "d": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "b",
              "d"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "b": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "b",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "c": {
                "type": "number"
              },
              "d": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "c",
              "d"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "c": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "c",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "d": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "d",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "b": {
                "type": "number"
              },
              "c": {
                "type": "number"
              },
              "d": {
                "type": "number"
              }
            },
            "required": [
              "b",
              "c",
              "d"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "b": {
                "type": "number"
              },
              "c": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "b",
              "c",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "b": {
                "type": "number"
              },
              "d": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "b",
              "d",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "c": {
                "type": "number"
              },
              "d": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "c",
              "d",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "b": {
                "type": "number"
              },
              "c": {
                "type": "number"
              },
              "d": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "b",
              "c",
              "d"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "b": {
                "type": "number"
              },
              "c": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "b",
              "c",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "b": {
                "type": "number"
              },
              "d": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "b",
              "d",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "c": {
                "type": "number"
              },
              "d": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "c",
              "d",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "b": {
                "type": "number"
              },
              "c": {
                "type": "number"
              },
              "d": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "b",
              "c",
              "d",
              "e"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "a": {
                "type": "number"
              },
              "b": {
                "type": "number"
              },
              "c": {
                "type": "number"
              },
              "d": {
                "type": "number"
              },
              "e": {
                "type": "number"
              }
            },
            "required": [
              "a",
              "b",
              "c",
              "d",
              "e"
            ],
            "additionalProperties": false
          }
        ],
        "description": "Object containing one or more of keys 'a'..'e', each a number. Only the listed keys for the chosen combination are permitted."
      }
    },
    "required": [
      "output"
    ],
    "additionalProperties": false
  }
}

A custom type of output where you get to provide a different schema to the AI than is enforced would be nice…

1 Like