Model is streaming message content as function call arguments

[joshdschneider]

The gpt-3.5-turbo model is streaming message content as function_call arguments. This happens about 50% of the time. Here’s the request payload:

{
  model: 'gpt-3.5-turbo-0613',
  temperature: 0.5,
  messages: [
    {
      role: 'system',
      content:
        'You are an AI agent talking to a human over the phone. Today is Wednesday, October 25, 2023 at 7:11PM UTC. Your replies are short (1-2 sentences).',
    },
    { role: 'user', content: 'Hello?' },
    { role: 'assistant', content: ' Hi! How can I assist you today? ' },
    { role: 'user', content: 'Can you transfer me to sales?' },
  ],
  functions: [
    {
      name: 'end_call',
      description: 'End the call and hang up the phone.',
      parameters: { type: 'object', properties: {} },
    },
    {
      name: 'hold_call',
      description: 'Place the call on hold. Call this function when the human asks you to hold or wait.',
      parameters: { type: 'object', properties: {} },
    },
    {
      name: 'transfer_call',
      description: 'Transfer the call to sales.',
      parameters: { type: 'object', properties: {} },
    },
  ],
  function_call: 'auto',
  stream: true,
};

Logs of raw message chunks:

[Foxabilo]

Hi and welcome to the Developer Forum!

This seems correct, what part of it does not seem right to you?

[joshdschneider]

If the model is sending function_call chunks, shouldn’t the value for “arguments” be function call arguments, not message content?

Look at the screenshot of the logs. I’m sporadically getting “Sure, I can transfer you…” as function call arguments. If you look at the functions that I defined in the request payload, the transfer_call action has no properties so I’d expect arguments to be an empty object or null.

Let me know if that is in fact the expected behavior. Thanks!

[Foxabilo]

Well looking at the docs,

 # Step 2: check if GPT wanted to call a function
    if response_message.get("function_call"):

Which you are getting with a content null and then the assistant message contains “function_call” and then the various message deltas contains the json function parameters, which looks correct to me.

[joshdschneider]

Did you look at the message deltas in the screenshot? The function call arguments are not function call arguments. When you concatenate those strings it is message content, not function call arguments. I believe this is a bug.

[Foxabilo]

Ahh, ok , I get you. The replies can sometimes contain hallucinations, there is some advice in the documentation about specifying a system message to help with this.

So , in this case, it might be worth trying a system message along the lines of “Please ensure that the function parameters returned only contain fully formed json objects and not text descriptions”

I agree that this could be considered a bug.

[supershaneski]

Can I ask if the function definitions you wrote above is as is or you just removed the parameters part? I want to test it and see if I can get the same result as yours.

[joshdschneider]

Yes, those are the exact function definitions from the error. Please let me know if you can reproduce it.

[supershaneski]

Yes, I can confirm that it happens. Probably because of the function definition. If I will rewrite your functions:

{
            "name": "manage_call",
            "description": "Call management function that allows for various operations on tasks",
            "parameters": {
                "type": "object",
                "properties": {
                    "task": {
                        "type": "string",
                        "description": "The action to be performed, such as 'transfer_call', 'hold_call', or 'end_call'",
                        "enum": [
                            "transfer_call",
                            "hold_call",
                            "end_call"
                        ]
                    }
                },
                "required": ["task"]
            }
        }

It will get you something like this:

{
  index: 0,
  delta: {
    role: 'assistant',
    content: null,
    function_call: { name: 'manage_call', arguments: '' }
  },
  finish_reason: null
}
{
  index: 0,
  delta: { function_call: { arguments: '{\n' } },
  finish_reason: null
}
{
  index: 0,
  delta: { function_call: { arguments: ' ' } },
  finish_reason: null
}
{
  index: 0,
  delta: { function_call: { arguments: ' "' } },
  finish_reason: null
}
{
  index: 0,
  delta: { function_call: { arguments: 'task' } },
  finish_reason: null
}
{
  index: 0,
  delta: { function_call: { arguments: '":' } },
  finish_reason: null
}
{
  index: 0,
  delta: { function_call: { arguments: ' "' } },
  finish_reason: null
}
{
  index: 0,
  delta: { function_call: { arguments: 'transfer' } },
  finish_reason: null
}
{
  index: 0,
  delta: { function_call: { arguments: '_call' } },
  finish_reason: null
}
{
  index: 0,
  delta: { function_call: { arguments: '"\n' } },
  finish_reason: null
}
{
  index: 0,
  delta: { function_call: { arguments: '}' } },
  finish_reason: null
}
{ index: 0, delta: {}, finish_reason: 'function_call' }

{ name: ‘manage_call’, arguments: ‘{\n “task”: “transfer_call”\n}’ }

[jilp]

@joshdschneider

...
    },
  ],
  function_call: 'auto',
  stream: true, <==
};

Including stream: true in the request body will result in streaming 100% of time until removed.

@reginald.I shared a helpful solution for handling streaming arguments

    func_call = {
        "name": None,
        "arguments": "",
    }
    for res in response:
        delta = res.choices[0].delta
        if "function_call" in delta:
            if "name" in delta.function_call:
                func_call["name"] = delta.function_call["name"]
            if "arguments" in delta.function_call:
                func_call["arguments"] += delta.function_call["arguments"] <== Arguments are assembled here
        if res.choices[0].finish_reason == "function_call":
            # function call here using func_call
            return
        if not delta.get("content", None):
            continue
        sys.stdout.write(delta.content)
        sys.stdout.flush()

As pointed out by @supershaneski, you’ll end up with:

{ name: ‘manage_call’, arguments: ‘{\n “task”: “transfer_call”\n}’ }