Structured Outputs Infinite \n Newline Characters

API Bugs
,
1

We are using structured outputs in production with GPT-4o-2024-08-06 and seem to be encountering an issue where often one of the objects within the structured output will just return an infinite number of \n newline characters until the max token limit is hit.

Found that this seemed to be an issue in the JSON_Mode, wondering if anyone else was having the same issue and had found a solution?

Thanks!

5 Likes
2

Hi @joseph11 and welcome to the community!

Never seen this problem before. Am I correct in understanding that you are using JSON mode? i.e.

response_format: { type: "json_object" }

Have you tried Structured Outputs? i.e.

response_format: { type: "json_schema", json_schema: {"strict": true, "schema": ...} }

The difference is with the latter it uses constrained decoding of tokens so it adheres to your supplied schema, and the probability of getting these weird repeated tokens should be very low…

4 Likes
3

We are using the json_schema type in the response format.

Strangely enough, when we added, “please return the in the specified JSON structure”, all the errors went away.

So it looks like OAI has a gap in their documentation, as this is a suggestion for the json_object in their docs.

3 Likes
4

That’s a problem that’s been seen since before day 1 with the introduction of a JSON-mode model, activated by response_format: json_object.

If you don’t tell exactly what and how to produce anyway, with as much effort as it would take ANY AI to understand what your API takes as JSON input, it will go nuts.

That’s why OpenAI makes you include the word “JSON”, but you only need to say “never produce JSON” to get a repeating loop of tabs or newlines.

Don’t do this without using max_tokens:

Untitled
Untitled951×359 52.9 KB

The actual “structured output” of higher level enforcement requires a JSON schema be sent with the different setting for that: json_schema

The schema is placed in AI context, in a format similar to a function specification, where the AI should be able to follow it just like it would follow your own instruction.

3 Likes
5

@_j interesting stuff. Kind of a black box if you ask me.

We pass in our JSON Schema with Descriptions and when we added the instruction ‘please return the output in the JSON Schema provided’ it caused the errors to mostly go away. However it still happens from time to time.

Maybe I just need to more explicit in my instruction telling it to return JSON in the requested schema?

Thanks for the help and insight

1 Like
6

This is the format the newest gpt-4o has injected at the end of the first system message to inform of its expected response. You will note that it is not exactly the schema that is sent as API parameter.

You can try out the understanding and compliance on models not specifically trained for it.

The schema itself is the demonstraton format for my preset for an AI that makes…schemas.

# Response Formats

## response_reproducing_context

{"type":"object","properties":{"schemas":{"type":"array","items":{"type":"object","properties":{"schema_text":{"type":"string","description":"Any schemas requested, each in the form desired, such as the original text the AI received."},"schema_format":{"enum":["json","python","pydantic"],"type":"string","description":"The format of the schema, indicating how the schema is represented."},"destination_type":{"enum":["tool_function","response_format"],"type":"string","description":"Specifies whether the schema is intended for a tool function or a response format."}}}},"plain_text_response":{"type":"string","description":"A response to the user from the AI, providing a typical verbose response fulfilling the input."}}}

To really reinfore the JSON output by an algorithm actually affecting the tokens that can be produced, you would use json_schema, use "strict": true in the top level of the object, and set all the keys into the required field, placing “required” at each nesting level object. Only on “gpt-2024-08-06+” or mini. That then finally turns on structured outputs.

Increasing the repetition penalty can break up repetitive strings after a while of repeating and already being useless.

7

Hey, welcome to the community.

I’ve had a lot of success with getting structured output and response_format to work properly… but that doesn’t mean that the content (within your structure) itself is going to be properly structured.

In fact, I’ve had issues with \n myself, come to think of it—though nothing on the order of creating an infinite series.

Without jumping to fine-tuning, you can include “description” fields and include more information about that field should, or should not, include. You should also include examples in your instructions of what the final output should look like.

{
  "name": "structure_monster_info_response",
  "description": "Structure the monster's info and output it as a JSON.",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": {
      "monster_unique_id": {
        "type": "string",
        "description": "Unique ID for the monster in our database, e.g., 'dndgpt_mon_0001'. This is a unique field."
      },
      "monster_parent": {
        "type": "string",
        "description": "The general category or type of monster, e.g., 'Dragon'. This is a unique field in our database."
      },
      "monster_name": {
        "type": "string",
        "description": "The name of the monster."
      },
2 Likes