Structure Outputs Not Working in evals

API Bugs
1

I’m running the following code to run an already defined eval I’ve created with some input data I’ve saved:

run = client.evals.runs.create(
    eval_obj.id,
    name="r1",
    data_source={
        "type": "completions",
        "model": "gpt-4.1-nano",
        "input_messages": {
            "type": "template",
            "template": [
                {"role": "developer", "content": MAIN_PROMPT},
                {"role": "user", "content": "{{ item.filtered_pdf }}"},
            ],
        },
        "response_format": {
            "type": "json_schema", 
            "json_schema": REPORT_SCHEMA
        },
        "source": {"type": "file_id", "id": file.id},
    },
)

The eval runs fine; however, the output is not formatted as specified in the json_schema. To test that my schema was working as intended I ran the following chat completion creation request using the same response format:

response = client.chat.completions.create(
    model="gpt-4.1-nano",
    messages=[
        {"role": "developer", "content": MAIN_PROMPT},
        {"role": "user", "content": ...},
    ],
    response_format= {
            "type": "json_schema", 
            "json_schema": REPORT_SCHEMA
        },
    temperature=0.0,
)

Here the output content looks exactly as specified in the schema, leading me to believe this a bug specifically with evals.

—UPDATE—
It wasn’t working because I wasn’t defining “reponse_format” within the “sampling_params” object. Doing so fixed the output.

2

Hi @jtwalter thanks for bringing this up! Can she share the final correct code? I can’t seem to get it working.

3

Just nest the response_format object within the sampling_params object as follows:

run = client.evals.runs.create(
    eval_obj.id,
    data_source={
        "type": "completions",
        "model": "o4-mini",
        "input_messages": {
            "type": "template",
            "template": [
                {"role": "developer", "content": MAIN_PROMPT},
                {"role": "user", "content": "{{ item.text }}"},
            ],
        },
        "sampling_params":{
            "temperature": 1.0,
            "response_format": {
                "type": "json_schema", 
                "json_schema": {
                     ...
                },
            },
        },
        "source": {"type": "file_id", "id": file.id},
    },
)
4

Thanks! I also dug through the documentation and found that this works too if you are using ‘responses’ instead of ‘completions’:

    run = client.evals.runs.create(
        eval_id,
        name=f"Sample Analysis Run {datetime.now().strftime('%Y%m%d_%H%M%S')}",
        data_source={
            "type": "responses",
            "input_messages": {
                "type": "template",
                "template": [
                    {"role": "system", "content": get_sample_analysis_prompt()},
                    {"role": "user", "content": "{{item.input}}"},
                ],
            },
            "sampling_params": {
                "temperature": 0,
                "max_completions_tokens": 1024,
                "text": {
                    "format": {
                        "type": "json_schema",
                        "name": "SampleAnalysis",
                        "schema": SAMPLE_ANALYSIS_SCHEMA,
                    }
                },
            },
            "model": model,
            "source": {"type": "file_id", "id": file_id},
        },
    )