GPT4o using tools + streaming?

Hi community - I have come to ask a question regarding the handling of tools whilst also streaming. - I cannot find anywhere saying how this is done in the docs and have spent a few days now writing code.

Essentially I am trying to handle a flow like this

  1. User asks question
  2. We check if the chat completion wants to use a tool

If yes - run tool function, run completion again, stream back to the user
If No - stream response back to the user

I have tried to condense my code process down as much as possible to help community readers - and anyone looking to do the same :slight_smile: :slight_smile:


const response = await fetch('https://api.openai.com/v1/chat/completions', {
      headers: {
        Authorization: `Bearer ${openAiKey}`,
        'Content-Type': 'application/json',
      },
      method: 'POST',
      body: JSON.stringify(completionOptions),
    })

    if (!response.ok) {
      const error = await response.json()
      throw new ApplicationError('Failed to generate completion', error)
    }

    // Create a readable stream to handle the response
    const [responseStream1, responseStream2] = response.body.tee();

    // Function to process the response stream for tool calls
    async function processStream(stream) {
      const reader = stream.getReader();
      const decoder = new TextDecoder();
      let toolCalls = [];
      let responseBody = '';
      let fullResponse = '';
      let assistantMessage = null;

      while (true) {
        const { value, done } = await reader.read();
        if (done) break;

        const chunk = decoder.decode(value);
        fullResponse += chunk;
        const matches = chunk.match(/data: (.+)/);

      
        if (matches && matches[1]) {
          try {
            const parsedChunk = JSON.parse(matches[1]);

            if (parsedChunk.choices[0].delta.tool_calls) {
              console.log('Tool call detected in chunk');
              toolCalls = toolCalls.concat(parsedChunk.choices[0].delta.tool_calls);
              assistantMessage = parsedChunk.choices[0]; // Capture the assistant's response
            } else if (parsedChunk.choices[0].delta.content) {
              responseBody += parsedChunk.choices[0].delta.content;
            }
          } catch (error) {
            console.log('Error parsing JSON:', error.message);
          }
        }
      }

      return { toolCalls, responseBody, fullResponse, assistantMessage };
    }

    const { toolCalls, responseBody, fullResponse, assistantMessage } = await processStream(responseStream1);

My first question is - is assistantMessage defined correctly/ pointing to the correct information? I know that the assistant response needs to be appended to my messages as seen via the docs: https://platform.openai.com/docs/guides/function-calling?lang=node.js

This is what I get after appending my assistant message plus function call result:

[
{
    role: "user",
    content: "Please tell me what the last activity i did was?"
  },
  {
    role: "assistant",
    content: null,
    tool_calls: [
      {
        index: 0,
        id: "call_u3mgzgRCrmIUHOH6oWpPH43C",
        type: "function",
        function: { name: "get_last_user_activity", arguments: "" }
      }
    ]
  },
  {
    tool_call_id: "call_u3mgzgRCrmIUHOH6oWpPH43C",
    role: "tool",
    name: "get_last_user_activity",
    content: '{"activity_id":13,"user_id":"1"}'
  }
]

My AI response after running this back through gpt-4o is “I am unable to comply with this request.”

Is there something that I am doing wrong - thanks for the help in advance!

Hi @calmer_blends0d, welcome to the forum.

If you are able to read Java code, you could abstract the logic from this fully functional demo:

This is a sample result:

Demo Results
Welcome! Write any message: Hi, can you help me with some quetions about Lima, Peru?
Of course! What would you like to know about Lima, Peru?

Write any message (or write 'exit' to finish): Tell me something brief about Lima Peru, then tell me how's the weather there right now. Finally give me three tips to travel there.
### Brief About Lima, Peru
Lima, the capital city of Peru, is a bustling metropolis that blends modernity with rich historical heritage. Founded by Spanish conquistador Francisco Pizarro in 1535, Lima is known for its colonial architecture, vibrant culture, and delicious cuisine, particularly its world-renowned ceviche. The city is also a gateway to exploring Peru's diverse landscapes, from the coastal deserts to the Andean highlands and the Amazon rainforest.

### Current Weather in Lima, Peru
I'll check the current temperature and the probability of rain in Lima for you.### Current Weather in Lima, Peru
- **Temperature:** Approximately 11.8°C
- **Probability of Rain:** Approximately 97.8%

### Three Tips for Traveling to Lima, Peru
1. **Explore the Historic Center:**
   - Visit the Plaza Mayor, the Government Palace, and the Cathedral of Lima. These landmarks offer a glimpse into Lima's colonial past and are UNESCO World Heritage Sites.

2. **Savor the Local Cuisine:**
   - Don't miss out on trying ceviche, a traditional Peruvian dish made from fresh raw fish marinated in citrus juices. Also, explore the local markets and try other Peruvian delicacies.

3. **Visit the Coastal Districts:**
   - Head to Miraflores and Barranco for stunning ocean views, vibrant nightlife, and cultural experiences. These districts are known for their beautiful parks, cliffs, and bohemian atmosphere.

Enjoy your trip to Lima! If you have any more questions, feel free to ask.

Write any message (or write 'exit' to finish): exit