I’m running the following code to run an already defined eval I’ve created with some input data I’ve saved:
run = client.evals.runs.create(
eval_obj.id,
name="r1",
data_source={
"type": "completions",
"model": "gpt-4.1-nano",
"input_messages": {
"type": "template",
"template": [
{"role": "developer", "content": MAIN_PROMPT},
{"role": "user", "content": "{{ item.filtered_pdf }}"},
],
},
"response_format": {
"type": "json_schema",
"json_schema": REPORT_SCHEMA
},
"source": {"type": "file_id", "id": file.id},
},
)
The eval runs fine; however, the output is not formatted as specified in the json_schema. To test that my schema was working as intended I ran the following chat completion creation request using the same response format:
response = client.chat.completions.create(
model="gpt-4.1-nano",
messages=[
{"role": "developer", "content": MAIN_PROMPT},
{"role": "user", "content": ...},
],
response_format= {
"type": "json_schema",
"json_schema": REPORT_SCHEMA
},
temperature=0.0,
)
Here the output content looks exactly as specified in the schema, leading me to believe this a bug specifically with evals.