Unexpected Tool Call Behavior with Response Format and Tool Descriptions in GPT-4o-mini API

API Bugs
, , , ,
1

I’ve encountered an issue while using the GPT-4 API (model: gpt-4o-mini-2024-07-18) with a specified response_format and tool definitions. The model generates tool calls that do not adhere to the declared tool schema or specified parameters.

Sample Request Body:

{
  "model": "gpt-4o-mini-2024-07-18",
  "temperature": 0.3,
  "messages": [
    {
      "role": "system",
      "content": "save_answer tool should be called with param_1: '2', param_2: 'hello'"
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "save_answer",
        "parameters": {
          "type": "object",
          "properties": {
            "param_1": {
              "type": "string"
            },
            "param_2": {
              "type": "string"
            }
          },
          "required": [
            "param_1",
            "param_2"
          ]
        }
      }
    }
  ],
  "response_format": {
    "json_schema": {
      "name": "output_format",
      "schema": {
        "properties": {
          "message": {
            "title": "Response To User",
            "type": "string"
          }
        },
        "required": [
          "message"
        ],
        "additionalProperties": false,
        "title": "schema_to_convert",
        "type": "object"
      },
      "strict": true
    },
    "type": "json_schema"
  }
}

API Response:

{
    "id": "chatcmpl-AdCVU6d9OSDTFHtJBnxw28ozp3nU6",
    "object": "chat.completion",
    "created": 1733906232,
    "model": "gpt-4o-mini-2024-07-18",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": null,
                "tool_calls": [
                    {
                        "id": "call_Q7BGaeJxmZuvK4GRiPVZzetI",
                        "type": "function",
                        "function": {
                            "name": "FUNCTIONS",
                            "arguments": "{\"param_1\": \"two\", \"param_2\": \"hi\"}"
                        }
                    },
                    {
                        "id": "call_AO8FKTYX8LlJcJ8ppnfzmK7E",
                        "type": "function",
                        "function": {
                            "name": "FUNCTIONS",
                            "arguments": "{\"param_1\": \"two\", \"param_2\": \"hi\"}"
                        }
                    }
                ],
                "refusal": null
            },
            "logprobs": null,
            "finish_reason": "tool_calls"
        }
    ],
    "usage": {
        "prompt_tokens": 96,
        "completion_tokens": 58,
        "total_tokens": 154,
        "prompt_tokens_details": {
            "cached_tokens": 0,
            "audio_tokens": 0
        },
        "completion_tokens_details": {
            "reasoning_tokens": 0,
            "audio_tokens": 0,
            "accepted_prediction_tokens": 0,
            "rejected_prediction_tokens": 0
        }
    },
    "system_fingerprint": "fp_62208eb26c"
}

As seen in the example, func name is wrong and the params do not comply with the function definition. Sometimes the API returns timeout too.

This was functional up until a few hours ago. Anyone experiencing something similar?

2 Likes
2

Sometimes other unexpected responses are returned from the API too. Some examples are given below:

               "tool_calls": [
                    {
                        "id": "call_ynVaT1nQp1vtuzj5pJY9bkso",
                        "type": "function",
                        "function": {
                            "name": "fifth",
                            "arguments": "{}"
                        }
                    },
                    {
                        "id": "call_8Jwk1mybUoSGs0fVPoeZPHtz",
                        "type": "function",
                        "function": {
                            "name": "fifth",
                            "arguments": "{}"
                        }
                    }
                ]

               "tool_calls": [
                    {
                        "id": "call_WfCXcb9MdvtwUVTnFBxPwnT2",
                        "type": "function",
                        "function": {
                            "name": "f",
                            "arguments": "{\"param_1\": \"two\", \"param_2\": \"hi\"}"
                        }
                    }
                ],
3

Same problem.
Here an issue I ve just posted on Vercel AInSDK GitHub repo.

—-
Hello,
First, thanks for the lib.

I’m using AI SDK with OpenAI to generate structuredOutputs. Until this morning, it was working just fine.

  // Generate a JSON object and return it
  const { object } = await generateObject({
    model: openai('gpt-4o-mini-2024-07-18', { structuredOutputs: true }),
    messages: context.conversation,
    schema: Augmented_Query, // See `_schema.ts`
    temperature: 0.5
  })

However, this morning, I’ve updated AI SDK to latest version and I’ve got (sometimes) a validation schema whereas I didn’t change anything. And, indeed, value does not include location key. Hence the error.

Trying another time, the schema error was different because value returns an array where a string was asked…

Did you see the same problem?
Is it linked to AI SDK Core (I think not because I’ve tried to downgrade to previous version) or OpenAI (I didn’t change model though).

Thanks.

AI_TypeValidationError: Type validation failed: **Value: ....**

Error message: [
  {
    "code": "invalid_type",
    "expected": "object",
    "received": "undefined",
    "path": [
      "location"
    ],
    "message": "Required"
  }
]
      at new _AISDKError (/Users/charnould/GitHub/pierre/node_modules/@ai-sdk/provider/dist/index.mjs:19:5)
      at new _TypeValidationError (/Users/charnould/GitHub/pierre/node_modules/@ai-sdk/provider/dist/index.mjs:362:3)
      at wrap (/Users/charnould/GitHub/pierre/node_modules/@ai-sdk/provider/dist/index.mjs:373:1)
      at safeValidateTypes (/Users/charnould/GitHub/pierre/node_modules/@ai-sdk/provider-u
4

I’m using the exact same prompt and API query params that worked perfectly last week, but it started failing this week. Since nothing has changed on our end, it seems likely that the issue is on OpenAI’s side rather than with any library or setup on our part.

5

Seems like the issue has been resolved, the API responses got back to normal.

However, it is frustrating to encounter such unreliability with the API. If OpenAI has made changes to address this issue, it would be helpful to receive confirmation from them to ensure the problem is truly resolved. Otherwise, it means the issue may appear again at any time and the API is not stable.

1 Like