ChatGPT occasionally reuses tool IDs in the same session

API Bugs
, ,
1

I can’t easily reproduce this, but this happens maybe 1 in 100 times so it is happening enough to be a problem.

I am building an autogpt-like experience where ChatGPT would call functions successively and execute on some task. I’m using the tool API, so if there is a tool ID referenced in an assistant’s function call, there must be an associated tool response with the same ID. As a toy example, if I use these kwargs:

kwargs = {
  "stream": False,
  "model": "gpt-3.5-turbo-1106",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant"
    },
    {
      "role": "user",
      "content": "Send so-and-so@gmail.com an email"
    },
    {
      "role": "assistant",
      "content": None,
      "tool_calls": [
        {
          "id": "call_gO7MBZKAevgoMoTpdmowmTcx",
          "function": {
            "name": "perform_action",
            "arguments": "some arguments"
          },
          "type": "function"
        }
      ]
    } #, Uncommenting this will fix the below error
    # {
    #   "role": "tool",
    #   "tool_call_id": "call_gO7MBZKAevgoMoTpdmowmTcx",
    #   "content": "some outputs"
    # }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "perform_action",
        "description": "Some action"
      }
    }
  ]
}

I will get the following error:

BadRequestError: Error code: 400 - {'error': {'message': "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_gO7MBZKAevgoMoTpdmowmTcx", 'type': 'invalid_request_error', 'param': 'messages', 'code': None}}

The problem I’m running into is that ChatGPT will occasionally reuse a tool ID in a tool call, and that causes a different error:

openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid parameter: messages with role 'tool' must be a response to a preceeding message with 'tool_calls'.", 'type': 'invalid_request_error', 'param': 'messages.[10].role', 'code': None}}

I can reproduce the error by repeating a function call & output with a duplicate Tool ID.

kwargs = {
  "stream": False,
  "model": "gpt-3.5-turbo-1106",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant"
    },
    {
      "role": "user",
      "content": "Send so-and-so@gmail.com an email"
    },
    {
      "role": "assistant",
      "content": None,
      "tool_calls": [
        {
          "id": "call_gO7MBZKAevgoMoTpdmowmTcx",
          "function": {
            "name": "perform_action",
            "arguments": "some arguments"
          },
          "type": "function"
        },
        {
          "id": "call_gO7MBZKAevgoMoTpdmowmTcx",
          "function": {
            "name": "perform_action",
            "arguments": "some arguments"
          },
          "type": "function"
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_gO7MBZKAevgoMoTpdmowmTcx",
      "content": "some outputs"
    },
    {
      "role": "tool",
      "tool_call_id": "call_gO7MBZKAevgoMoTpdmowmTcx",
      "content": "some outputs"
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "perform_action",
        "description": "Some action"
      }
    }
  ]
}

As a developer, it is annoying that the tool call IDs are not unique across the same session, and it seems like there may be bug that’s causing this to happen. I know that there are workarounds like issuing my own IDs, but this seems like it should be fixed on the API side.

3 Likes
2

Just discovered this same issue – though I would say not ChatGPT, but rather in models gpt-3.5-turbo and gpt-4-1106-preview through API calls (chat completions). Re-used call-identifiers across tool calls are proving to be problematic, and these are across sessions (different days, different tools, etc.).

I guess I assumed these would be unique.

3 Likes