Different behavior in streaming function calling

API Bugs
1 
FUNCTIONS = {
    "count_string": {
        "name": "count_string",
        "description": "Counts the number of characters in a string.",
        "parameters": {
            "type": "object",
            "properties": {
                "string_to_count": {
                    "type": "string",
                    "description": "The string whose characters you want to count.",
                },
            },
            "required": ["string_to_count"],
        },
    }
}
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "count_string",
            "description": "Counts the number of characters in a string.",
            "parameters": {
                "type": "object",
                "properties": {
                    "string_to_count": {
                        "type": "string",
                        "description": "The string whose characters you want to count.",
                    },
                },
                "required": ["string_to_count"],
            },
        }
    }
]

When I use functions to get response, it’s ok.

messages = [{"role": "user", "content": "Counts the number of characters in 'hello word'."}]
response = client.chat.completions.create(
    model="gpt-4-0613",
    messages=messages,
    functions=list(FUNCTIONS.values()),
    stream=True,
)

The response is as bellow, function_call is not None.

ChatCompletionChunk(id='chatcmpl-92xSEdSMDugkO3jGEkTbbDTNJGPai', choices=[Choice(delta=ChoiceDelta(content=None, function_call=ChoiceDeltaFunctionCall(arguments='{\n', name=None), role=None, tool_calls=None), ....

But I use tools to get response, it is not ok.

messages = [{"role": "user", "content": "Counts the number of characters in 'hello word'."}]
response = client.chat.completions.create(
    model="gpt-4-0613",
    messages=messages,
    tools=TOOLS,
    stream=True,
)

The response is as bellow, function_call is None.

ChatCompletionChunk(id='chatcmpl-92xhJurez25gW8BXLNldlIkVN4x9j', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role='assistant', tool_calls=None), finish_reason=None ...
ChatCompletionChunk(id='chatcmpl-92xhJurez25gW8BXLNldlIkVN4x9j', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role=None, tool_calls=None), ...
....
2

Include a system message. Even “You are a nice chatbot”. And write the user input like a person would…

3

Does it work if you disable streaming? Streaming mixed with tool/function use seems like a bad idea. The issue is you could get a tool/function predicted without the full set of arguments.

4

Here’s someone who hacked together streaming+tools support but it’s a complete hack:

5

It worked if you disabled streaming.

6

delta.content and delta.tool_calls are both None, so the code doesn’t work.