Structured Output Cached Response

We are working with structured output as shown in the following link.

https://platform.openai.com/docs/guides/structured-outputs

We are using the Python client as well. openai==1.55.0

We are passing in text from a PDF document and requesting structured output for specific data elements from the text.

It all seemed to be working great until I used a “dummy” PDF with the text of, “This is a dummy PDF.”

I’m using this for testing error handling and the structured output for this case looks like it is returning the previous structured output. I am guessing the structured output is cached and since this test has none of the fields we are looking for the structured output is not populated and we get back a cached response.

1 Like

Context caching only works on repeated inputs. The computation that can be reused is only up to the point where the AI begins to generate its own language. Context caching will not save expenses on output, even if that is output fed back in for the next turn.

If it was a strict structured response format, the AI would not be able to write anything but the exact keys provided in the schema. Nothing would be optional and nothing could be changed in the keys or object structure.

returning the previous structured output”? How does that manifest? Are you in a chat with a conversation where you have continued talking, perhaps showing the AI a different response than desired.

Here is building on and sending a schema to OpenAI (in Python). Observe each level and the keys that must be present for it to be strict, along with use of gpt-4o:

schema = {
        "type": "object",
        "properties": {
            "document_sentence_extraction": {
                "type": "string",
                "description": "Output the most important data element",
            },
            "document_topic": {
                "type": "string",
                "description": "Short summary of seen text",
            },
        },
        "required": ["document_sentence_extraction",
                     "document_topic"],
        "additionalProperties": False,
    }

    strict_schema = {"name": "response", "strict": True, "schema": schema}

    response_format = {"type": "json_schema", "json_schema": strict_schema}

Sorry about the new code color scheme…use keyword ```json on a code block to not have a puke green mess.

Actually it looks like my structured data response is filled with hallucinations. They are similar to my test data so I didn’t notice with a quick inspection.

Which model are you using to read the PDF?

I noticed in my own work that using 4o-Mini yielded a ton of hallucinations when using it to try to read a pdf and structure an output in a single step.

Separating concerns so the smarter model reads and restructure the data then handing it to a mini for structured output seemed to yield the most accuracy.

1 Like

I’m using gpt-4o. I actually found my problem which was a dumb mistake.

{“role”: “user”, “content”: f"{prompt}\n\n’The following text is the content of the document: '{document_text}"}

I have a prompt that specifies what we are looking for in the document text and then I am passing in the document text and the response is expecting it in a JSON format defined by a model I am passing in.

I had an error creating the document text that I didn’t notice so the AI was halucinating because the input text was not complete. It’s working fine now.

The only thing I am working through now is in my model I have required fields. When there is not data from the document text for those required fields I would like to the API to return ‘None’ and have JSON Schema validation throw an error due to a missing required field. Instead the API is filling a required filed with an empty string so the schema validation passes.

1 Like