Help for function calls with streaming

API
2

I would look at how the chunks are actually coming in a tool call response, and immediately branch to a path that simply assembles the rest of the AI response into a list of tool strings when you get a tool_call. We don’t have to actually “branch”, but just gather something different, silently, for those chunks.

display raw chunks

c = client.chat.completions.with_raw_response.create(**params)

Leaving this in pydantic model format:

next(c.parse())
ChatCompletionChunk(id=‘chatcmpl-8xxx’, choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role=‘assistant’, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1707962400, model=‘gpt-4-0125-preview’, object=‘chat.completion.chunk’, system_fingerprint=‘fp_f084bcfc79’)

A typical “no use” or “we blocked you seeing what the AI actually produced” chunk as the first is what we see above.

Then, next, we get to – tool_calls with a name. That is the point where we can now exit any playing of content chunks and start the silent collection of the remainder of the response as a tool_call object.

next(c.parse())
ChatCompletionChunk(id=‘chatcmpl-8xxx’, choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id=‘call_TqhIpvAGcZOLG81yIhu59Kj0’, function=ChoiceDeltaToolCallFunction(arguments=‘’, name=‘get_random_float’), type=‘function’)]), finish_reason=None, index=0, logprobs=None)], created=1707962400, model=‘gpt-4-0125-preview’, object=‘chat.completion.chunk’, system_fingerprint=‘fp_f084bcfc79’)

Then arguments contents of that particular function to assemble (the start of “random…”).

next(c.parse())
ChatCompletionChunk(id=‘chatcmpl-8xxx’, choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=‘{"ra’, name=None), type=None)]), finish_reason=None, index=0, logprobs=None)], created=1707962400, model=‘gpt-4-0125-preview’, object=‘chat.completion.chunk’, system_fingerprint=‘fp_f084bcfc79’)

I haven’t seen them yet in “tools” for what I’ve written to the API, but you should write code expecting any “content” chunks until finish reason need to be printed.

Implement

Let’s write some neat code:

c = client.chat.completions.with_raw_response.create(**params)
reply=""
tools=[]
for chunk in c.parse():
    print(chunk.choices[0].delta)
    if chunk.choices[0].delta.content:
        reply += chunk.choices[0].delta.content        # gather for chat history
        print(chunk.choices[0].delta.content, end="")  # your output method
    if chunk.choices[0].delta.tool_calls:
        tools += chunk.choices[0].delta.tool_calls     # gather ChoiceDeltaToolCall list chunks
tools_obj = tool_list_to_tool_obj(tools)
print(reply)
print(tools_obj)

It’s only neat because I gather messy tool deltas and turn them back into a typical non-stream object with a function:

from collections import defaultdict

def tool_list_to_tool_obj(tools):
    # Initialize a dictionary with default values
    tool_calls_dict = defaultdict(lambda: {"id": None, "function": {"arguments": "", "name": None}, "type": None})

    # Iterate over the tool calls
    for tool_call in tools:
        # If the id is not None, set it
        if tool_call.id is not None:
            tool_calls_dict[tool_call.index]["id"] = tool_call.id

        # If the function name is not None, set it
        if tool_call.function.name is not None:
            tool_calls_dict[tool_call.index]["function"]["name"] = tool_call.function.name

        # Append the arguments
        tool_calls_dict[tool_call.index]["function"]["arguments"] += tool_call.function.arguments

        # If the type is not None, set it
        if tool_call.type is not None:
            tool_calls_dict[tool_call.index]["type"] = tool_call.type

    # Convert the dictionary to a list
    tool_calls_list = list(tool_calls_dict.values())

    # Return the result
    return {"tool_calls": tool_calls_list}

Output of running the code with some tools and messages as parameters:

{‘tool_calls’: [{‘id’: ‘call_44LLGv0lFEZeFAnRahnQx4H8’, ‘function’: {‘arguments’: ‘{“range_start”: 0, “range_end”: 66}’, ‘name’: ‘get_random_float’}, ‘type’: ‘function’}, {‘id’: ‘call_vjhf3oHMYbK0FyZYUpyNWimR’, ‘function’: {‘arguments’: ‘{“range_start”: 1, “range_end”: 33}’, ‘name’: ‘get_random_int’}, ‘type’: ‘function’}]}

2 Likes