ModelBehaviorError: Invalid output type

API
,
1

Hello everyone. I use OpenAI Agents SDK but I have a problem. Even if I define an outputType, model rarely gives outputs don’t align with the ouputType defined. This causes my backend service to go down because of an unhandled exception.

export const Answer = catchAsync(async (req: CustomRequestChatbot, res: Response) => {
  
...
...
...
 const runner = new Runner({
      workflowName: 'mas_storefront',
      groupId: chatSession.sessionId,
      traceId: trace_${message.id},
      tracingDisabled: false
    })

    const result = await runner.run(
      startAgent,
      message.message.content,
      {
        context: context,
        previousResponseId: chatSession.lastResponseId,
        stream: true,
      },

    );
    
    for await (const event of result.toTextStream()) {
      res.write(event);
    }

    res.end();
...
...
...
}

Unhandled rejection ModelBehaviorError: Invalid output type
    at executeToolsAndSideEffects (/app/node_modules/@openai/agents-core/src/runImplementation.ts:614:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Runner.#runStreamLoop (/app/node_modules/@openai/agents-core/src/run.ts:781:30) {

catchAsync doesn’t catch the error. I also tried to wrap .run() and .toTextStream() blocks with try/catch before, but it also didin’t work. I couldn’t find a way to handle this error. My backend is crushing every time API returns a response in the wrong format.

Also is it normal for API to give out of format response occasionally?

3 Likes
2

Hi, what is the format it must return?
Is your error intermittent or stable on specific formats?
Is it the same model that must provide this format?

Thanks.

4 Likes
3

Actually the format is always the same and defined by zod object as OpenAI Agents TS SDK suggests. However, sometimes the response is a string which causes the error.

export const AgentsResponseStorefront = z.object({
    content: z.string(),
    suggestive_answers: z.array(z.string()),
    recommended_products: z.array(z.string()),
    forms: z.array(z.object({
        type: z.string(),
        fields: z.array(z.object({
            name: z.string(),
            type: z.string(),
        }))
    }))
})
2 Likes
4

See GPT4.1 doesn't follow strict json schema

1 Like
5

Do you guys use field description in the schemas? I do because it helps focus the model on the exact values.

Also for the schema in the example looks like open door for hallucinations if not detailed in the system prompt (too loose as definition).

But having no other info makes me guess more than help.

2 Likes
6

I don’t think it is possible to give descriptions in zod schema. Their descriptions are in the system prompt. However, this is not important because it should give an output following the schema type. Descriptions may only effect the content’s relevance.

2 Likes
7

There are two key points of this issue:

The first one is a big issue and is being discussed here: GPT4.1 doesn't follow strict json schema

I created an issue for the second one and it is being fixed by OpenAI guys right now:

8

Well, in this case, the bulletproof technique I used before the JSON schemas were supported by llms would be reconstruct each field separately in parallel calls and then use regular code to build the json response.

9

I also had the exact same error with the Agents library. The way I got it to work was, instead of using the Zod object schema, I replaced it with the raw schema definition. Then it worked!

I assume it could be a compatibility issue with Zod sometimes.

2 Likes
10

GitHub - openai/openai-agents-js: A lightweight, powerful framework for multi-agent workflows and voice agents has a bug that makes it impossible to catch ModelBehaviorError: Invalid output type error emitted by @openai/agents-core/src/runImplementation.ts.

This point of the issue was resolved by @seratch in Fix a bug where some of the exceptions thrown from runImplementation.ts could be unhandled by seratch · Pull Request #148 · openai/openai-agents-js · GitHub

OpenAI API (GPT 4.1 in particular) sometimes provides unstructured output while a structure for the output is defined is still valid and should be discussed under GPT4.1 doesn't follow strict json schema

2 Likes