Function calling not returning the expected response structure

API
,
1

I recently implemented function calling in a project I am working on and am getting some weird output. My understanding is that when I call ‘chat.completion’ with a 0613 model such as gpt-3.5-0613 and I include function descriptions for the functions parameter, that it should return the name of the function to call here:

 response = openai.ChatCompletion.create(
                model=model,
                messages=messages,
                functions=functions,
                temperature=0
            )
response_message = response["choices"][0]["message"]["function_call"]["name"]

For some queries, even when calling chat completion from an appropriate model, and including functions = my_functions to populate the functions paramater, I am sometimes getting back the function selection in the content field, with no function_call field populated at all.
For example:

[{'name': 'generate_sql', 'description': 'function used to generate the relevant sql to fulfill user request. use when the user wants to query the database or reference information in the database', 'parameters': {'type': 'object', 'properties': {'user_prompt': {'type': 'string', 'description': "the user's query to the database ex. 'How many shipments are in the database? What was our profit for the month of may 2023?'"}}, 'required': ['user_prompt']}}, 
{'name': 'reference_memory', 'description': 'function that references previous hsitory of the conversation to answer user query. conversation history contains summaries about previous topics discussed', 'parameters': {'type': 'object', 'properties': {'user_prompt': {'type': 'string', 'description': "the user's query to the conversation history. ex. 'What did I just ask you? What does that data mean? Can you identify any trends there?'"}}, 'required': ['user_prompt']}}]
{
  "id": "chatcmpl-7W9ovBd5AhhY1K9dYooxeHuFZspl3",
  "object": "chat.completion",
  "created": 1687898601,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "reference_memory"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 285,
    "completion_tokens": 3,
    "total_tokens": 288
  }
}

As you can see the name of the function to call populates under ‘content’ and does not create a function_call key, even though I used the appropriate model, and included information for the functions parameter.

Any ideas what could be the cause of this? Is it a bug in the function-calling feature?

1 Like
2

It looks like you are defining two functions: generate_sql and reference_memory, and then have a JSON object which seems to be the output have you… Edit … I just compiled your splice and got 0 flaws on a dummy run… have you checked the json for syntax?? Possibly a stray . Somewhere??

1 Like
3

I also tried your functions and they work as expected.
Can you share what query are giving you weird response?

4

Hi, yes. Here is the structure of the function i am using up until the point of the api call.

def ask(
    query: str,
    messages,
    model: str = "gpt-3.5-turbo-0613",
    token_budget: int = 4096 - 500,
    print_message: bool = False,
    user_prompt: str = False,
    functions = False
) -> str:
    retry_limit = 3
    retry_interval = 2
    messages.append({"role": "user", "content": f"{query}"})
    # if num_tokens(str(messages)) > 4095:
    #     messages = summarize(messages)
    print("Functions " + str(functions))
    print("Messages:" + str(messages) + "\n")
    try:
        if functions:
            response = openai.ChatCompletion.create(
                model=model,
                messages=messages,
                functions=functions,
                temperature=0
            )
            print(functions)
            print(response)
            response_message = response["choices"][0]["message"]["function_call"]["name"]

This is what I sent for the messages:

[{'role': 'system', 'content': "You are an ai tasked with selecting whether to answer a user's query by referencing the previous conversation history between the user and the agent or by querying the database to return relevant data to help answer the query."}, {'role': 'user', 'content': 'Select the correct function based on the user message and the previous conversation history. \n\nUser_query: how many shipments are in the database?.'}]

Here are the functions I am submitting to the functions paramater:

[{'name': 'generate_sql', 'description': 'function used to generate the relevant sql to fulfill user request. use when the user wants to query the database or reference information in the database', 'parameters': {'type': 'object', 'properties': {'user_prompt': {'type': 'string', 'description': "the user's query to the database ex. 'How many shipments are in the database? What was our profit for the month of may 2023?'"}}, 'required': ['user_prompt']}}, 
{'name': 'reference_memory', 'description': 'function that references previous hsitory of the conversation to answer user query. conversation history contains summaries about previous topics discussed', 'parameters': {'type': 'object', 'properties': {'user_prompt': {'type': 'string', 'description': "the user's query to the conversation history. ex. 'What did I just ask you? What does that data mean? Can you identify any trends there?'"}}, 'required': ['user_prompt']}}]

I checked inside my ask() function call and saw the messages and functions got passed successfully to the ask() function, which is immediately sent to the API call. So I’m so confused about why its trying to answer the question by returning information in the content field, when I am passing information to the functions parameter…

I just don’t see how its possible for me to send a whole different data structure, to a parameter specifically called ‘functions’ (completely different from the messages parameter) and it interprets it as a request for an unstructured response. That really seems like a bug, no?

I’m curious if you guys get the same response with these model, messages, and functions arguments.

5

Here is a quickly reproducible example. Please let me know if you get the same results or if you notice anything wrong.

import os
import openai

os.environ['OPENAI_KEY'] = "YOUR_API_KEY"
openai.api_key = os.getenv('OPENAI_KEY')

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo-0613",
    messages=[{'role': 'system', 'content': "You are an ai tasked with selecting whether to answer a user's query by referencing the previous conversation history between the user and the agent or by querying the database to return relevant data to help answer the query."}, {'role': 'user', 'content': 'Select the correct function based on the user message and the previous conversation history. \n\nUser_query: how many shipments are in the database?.'}],
    functions=[{'name': 'generate_sql', 'description': 'function used to generate the relevant sql to fulfill user request. use when the user wants to query the database or reference information in the database', 'parameters': {'type': 'object', 'properties': {'user_prompt': {'type': 'string', 'description': "the user's query to the database ex. 'How many shipments are in the database? What was our profit for the month of may 2023?'"}}, 'required': ['user_prompt']}}, 
    {'name': 'reference_memory', 'description': 'function that references previous hsitory of the conversation to answer user query. conversation history contains summaries about previous topics discussed', 'parameters': {'type': 'object', 'properties': {'user_prompt': {'type': 'string', 'description': "the user's query to the conversation history. ex. 'What did I just ask you? What does that data mean? Can you identify any trends there?'"}}, 'required': ['user_prompt']}}],
    temperature=0
)
print(response)

Produces this output:

{
  "id": "chatcmpl-7WPxxZbdTJbMsC4GzTvloPutBZ0C7",
  "object": "chat.completion",
  "created": 1687960665,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The correct function to use in this case would be the `generate_sql` function, as the user is asking for information about the number of shipments in the database, which requires querying the database."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 243,
    "completion_tokens": 40,
    "total_tokens": 283
  }
}

As mentioned previously, function calling, presumably triggered by including a functions arguments for the functions parameter in the API call, is expected to return information accessible by response[“choices”][0][“message”][“function_call”][“name”] which in this case, it does not.

Edit: Found someone else having a similar problem here, and it doesn’t appear anyone found an exact solution. https://community.openai.com/t/function-call-included-in-content-not-in-function-call-response-param/272373/11

Edit2: I could technically force it to return a function by setting function_call= but I’m not sure that I can explicitly name multiple functions here. I just tried and it threw an error, but I am still investigating.

Edit3: My takeaway here is that you should be allowed to force function calling to do function calling in the case of multiple function definitions. As it is right now you can only set it to ‘None’, ‘Auto’, or name 1 specific function. This misses deterministic output for the case of multiple functions and is not an adequate implementation in its current form based on my experience.

6

I tried your code. I got the following results:

Using this prompt:

Select the correct function based on the user message and the previous conversation history.
User_query: how many shipments are in the database?

The response is similar to yours

{
  role: 'assistant',
  content: "The correct function to use in this case is the `generate_sql` function. This function is used to generate the relevant SQL query to fulfill the user's request to query the database."
}

If I remove the initial part of the prompt and just use:

how many shipments are in the database?

I get function call result

{
  role: 'assistant',
  content: null,
  function_call: {
    name: 'generate_sql',
    arguments: '{\n  "user_prompt": "How many shipments are in the database?"\n}'
  }
}

I tried normal conversation and seem to be getting the response correct.

Based on this, the initial part of the user prompt seem to be used as an additional instruction that affects the response. Do you really need to include it?

7

@supershaneski What finish_reason are you getting in your response? I am having an issue similar to @93INFINITY, where the finish_reason is stop, but in my case the response is otherwise in the correct format. The problem is I’m using the finish_reason test to decide how to parse the response. I wondered if perhaps it was returning “stop” because I was using “function_call” when calling.

Sure enough, it looks like using “function_call” overrides the finish_reason logic in the response. Self-evident, sure, but for now when deciding to treat the response as a message or as function output we have to check whether function_call was included in the call OR whether the finish_reason is “function_call”.

8

Actually a good point, yes the “finish_reason” does now have a fucntion result code, so that needs to be handled.

9

yes the finish_reason is stop.

i have not been monitoring finish_reason in my testing. it seems it does give additional context on the result that devs can use. thanks for the heads up.

1 Like