GPT-4 ask for tool calls but never tells which one

API Bugs
4

This is happening on Azure to me as well, it broke yesterday. Consider this simple Chat completion request:

{
   "tool_choice":"auto"
  ,"tools":[
      {
        "type":"function"
      ,"function": {
        "name":"659eb9def985b66941c2c3c3",
        "description": "Tool to answer question 659eb9def985b66941c2c3c3",
        "parameters": {
            "properties":{
                "659eb9def985b66941c2c3c3":{
                    "type": "object",
                    "properties":{ 
                        "answer": {"type": "string"}
                      }
                    }
            }
           ,"required":["659eb9def985b66941c2c3c3"]
           ,"title":"Answers"
           ,"type":"object"
           }
      }
    }
  ]
  ,"temperature":0
  ,"stream":false
  ,"response_format":{"type":"json_object"}
  ,"model":"gpt-4"
  ,"messages":[
      {"content":"Use all available tools, reply in JSON.","role":"system"}
    ,{
        "content":"Answer question '659eb9def985b66941c2c3c3': Who is Leo Fender?"
      ,"role":"user"
    }
  ]
}

Using api version 2023-12-01-preview, this yields:

{
    "id": "chatcmpl-...",
    "object": "chat.completion",
    "created": 1704967194,
    "model": "gpt-4",
    "prompt_filter_results": [
        {
            "prompt_index": 0,
            "content_filter_results": {
                "hate": {
                    "filtered": false,
                    "severity": "safe"
                },
                "self_harm": {
                    "filtered": false,
                    "severity": "safe"
                },
                "sexual": {
                    "filtered": false,
                    "severity": "safe"
                },
                "violence": {
                    "filtered": false,
                    "severity": "safe"
                }
            }
        }
    ],
    "choices": [
        {
            "index": 0,
            "finish_reason": "tool_calls",
            "message": {
                "role": "assistant"
            },
            "content_filter_results": {}
        }
    ],
    "usage": {
        "prompt_tokens": 126,
        "completion_tokens": 127,
        "total_tokens": 253
    }
}

Unless I’m being completely dumb, the request looks fine to me, and requests like these used to work just fine before yesterday. Note that if I replace the above with the old function_calls, it works as intended:

{
   "function_call": {"name": "659eb9def985b66941c2c3c3" }
  ,"functions": [{
        "name":"659eb9def985b66941c2c3c3",
        "description": "Function for question 659eb9def985b66941c2c3c3",
        "parameters": {
            "properties":{
                "659eb9def985b66941c2c3c3":{
                    "type": "object",
                    "properties":{ 
                        "answer": {"type": "string"}
                      }
                    }
            }
           ,"required":["659eb9def985b66941c2c3c3"]
           ,"type":"object"
           }
      }
  ]
  ,"temperature":0
  ,"stream":false
  ,"response_format":{"type":"json_object"}
  ,"model":"gpt-4"
  ,"messages":[
      {"content":"Use function '659eb9def985b66941c2c3c3', reply in JSON.","role":"system"}
    ,{
        "content":"Who is Leo Fender?"
      ,"role":"user"
    }
  ]
}

Yielding:

{
    "id": "chatcmpl-...",
    "object": "chat.completion",
    "created": 1704967409,
    "model": "gpt-4",
    "prompt_filter_results": [
        {
            "prompt_index": 0,
            "content_filter_results": {
                "hate": {
                    "filtered": false,
                    "severity": "safe"
                },
                "self_harm": {
                    "filtered": false,
                    "severity": "safe"
                },
                "sexual": {
                    "filtered": false,
                    "severity": "safe"
                },
                "violence": {
                    "filtered": false,
                    "severity": "safe"
                }
            }
        }
    ],
    "choices": [
        {
            "index": 0,
            "finish_reason": "stop",
            "message": {
                "role": "assistant",
                "function_call": {
                    "name": "659eb9def985b66941c2c3c3",
                    "arguments": "{\"659eb9def985b66941c2c3c3\":{\"answer\":\"Leo Fender was an American inventor and entrepreneur who founded Fender Electric Instrument Manufacturing Company, now known as Fender Musical Instruments Corporation. He is widely recognized for creating some of the most iconic electric guitars and amplifiers, including the Telecaster and Stratocaster guitars, and the Bassman amplifier. His innovations significantly influenced the development of music, particularly rock and roll, and his instruments continue to be highly regarded by musicians worldwide.\"}}"
                }
            },
            "content_filter_results": {}
        }
    ],
    "usage": {
        "prompt_tokens": 138,
        "completion_tokens": 105,
        "total_tokens": 243
    }
}

From some comments I have read somewhere, I have the suspect this is a deliberate choice from OpenAI, because tools are not meant to be used this way anymore but rather via their (now BETA) Assistants and Threads API.

It would be nice to get some semi-official confirmation of this, because currently I switched my software away from the deprecated function calls thinking it was a no brainer, but it seems like the behaviour is starting to diverge?

It’s a bit of upsetting situation, because from one side we have some deprecated fields (that works as intended), on the other some BETA APIs which are not yet meant for production and in the middle a grey area of things that currently … do not work :slight_smile:

2 Likes