Calling chat completion parse with structured output and previous tool calls in the message history throws a 500

Hi, using the latest 1.40.0 version of python SDK and the new gpt-4o-2024-08-06 model, when I pass a previous message history that contains tool calls and the tool responses alongside a response format pydantic model, it throws a 500 error. When I remove the tool call it seems to work fine.

1 Like

I have a minimal reproduceable example:

from pydantic import BaseModel
from openai import OpenAI

client = OpenAI()

class Step(BaseModel):
    file_path: str
    repo_name: str
    type: str
    diff: str
    description: str
    commit_message: str


class CodingOutput(BaseModel):
    steps: list[Step]


completion = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=message_dicts,
    response_format=CodingOutput,
)

print(completion.choices[0].message.parsed)

Works:

message_dicts = [
    {
        "content": "You are an exceptional principal engineer that is amazing at finding and fixing issues in codebases.",
        "role": "system",
    },
    {
        "content": 'We need to enhance error handling in the `from_str` method.',
        "role": "assistant",
    },
    {
        "content": "Break down the task of fixing the issue into steps.",
        "role": "user",
    },
]

Does not work, throws a 500 HTTP error:

message_dicts = [
    {
        "content": "You are an exceptional principal engineer that is amazing at finding and fixing issues in codebases.",
        "role": "system",
    },
    {
        "content": "To address the issue of the malformed input string causing a `KeyError` in the `PriorityLevel` enum, we need to implement a comprehensive solution.",
        "role": "assistant",
        "tool_calls": [
            {
                "id": "call_wHhKUvGq8xMyB78PYQrqIqTp",
                "function": {
                    "name": "expand_document",
                    "arguments": '{"input": "some/document.py", "repo_name": "owner/repo"}',
                },
                "type": "function",
            },
        ],
    },
    {
        "content": 'class PriorityLevel(IntEnum):\n    LOW = 25\n    MEDIUM = 50\n    HIGH = 75\n\n    def to_str(self) -> str:\n        return self.name.lower()\n\n    @classmethod\n    def from_str(self, name: str) -> "PriorityLevel":\n        return self[name.upper()]\n',
        "role": "tool",
        "tool_call_id": "call_wHhKUvGq8xMyB78PYQrqIqTp",
    },
    {
        "content": 'We need to enhance error handling in the `from_str` method.',
        "role": "assistant",
    },
    {
        "content": "Break down the task of fixing the issue into steps.",
        "role": "user",
    },
]

from my experience, you do not add previous tool calls in the context.

You need to add the previous tool calls results in the context no? Our use case is to format only the last output.

hmm, you only add the tool call response and tool call output when you are processing it. but i believe (maybe i am wrong) that you do not add it to the main context itself. so in the sample code you gave, it is already processed and already behind two messages.

Each call with the context is independent from oneanother, it’s not processed and stored in a memory somewhere. You need to keep sending longer and longer message chains.

This works perfectly fine outside of the json schema mode.

In the assistant message that contains the “tool_calls” the content needs to be None for some reason.

Updated version:

message_dicts = [
    {
        "content": "You are an exceptional principal engineer that is amazing at finding and fixing issues in codebases.",
        "role": "system",
    },
    {
        "content": None,
        "role": "assistant",
        "tool_calls": [
            {
                "id": "call_wHhKUvGq8xMyB78PYQrqIqTp",
                "function": {
                    "name": "expand_document",
                    "arguments": '{"input": "some/document.py", "repo_name": "owner/repo"}',
                },
                "type": "function",
            },
        ],
    },
    {
        "content": 'class PriorityLevel(IntEnum):\n    LOW = 25\n    MEDIUM = 50\n    HIGH = 75\n\n    def to_str(self) -> str:\n        return self.name.lower()\n\n    @classmethod\n    def from_str(self, name: str) -> "PriorityLevel":\n        return self[name.upper()]\n',
        "role": "tool",
        "tool_call_id": "call_wHhKUvGq8xMyB78PYQrqIqTp",
    },
    {
        "content": "We need to enhance error handling in the `from_str` method.",
        "role": "assistant",
    },
    {
        "content": "Break down the task of fixing the issue into steps.",
        "role": "user",
    },
]

I don’t think it should be like this, it must be a bug.

1 Like

Hmm this doesn’t fix it for me.

Is it because the “required” and “additionalProperties” fields are needed? Docs say that this is the case.