Tool calls and streaming, error on second API call

API
, ,
1

I am trying to get both streaming and tool calls working in a nodeJS project. I’ve seen only a few posts sort of related to this, but none with my exact issue and setup. My use case is a user-facing chat bot where there tool calls will be used to inject context into chat history, for example, recommended restaurants from a curated list. Currently I have streaming working as long as a tool call is not returned. When a tool call is returned, I am able to capture the tool_call_id, tool call function name, and arguments in the returning stream.

After the first request is processed, I pushed the info into the context and call the API again, but get a very long response that I believe is an error, like:

 <ref *2> IncomingMessage {
  _readableState: ReadableState {
    objectMode: false,
    highWaterMark: 16384,
    buffer: BufferList { head: [Object], tail: [Object], length: 1 },
    length: 233,
    pipes: [],
    flowing: null,
    ended: false,
    endEmitted: false,
...

I think I could have a fundamental misunderstanding of some aspect of the sequence of events. I have yet to find a full working example with both streaming with tool calls that involve hitting the API the second time.

What’s unclear to me is:

  1. What does the error/response mean? I couldn’t find any documentation.
  2. Am I missing anything besides pushing the tool call message into my context, and re-calling the API?
  3. Am I missing any nuance to the functionality of tool calls that is making me misunderstand how it’s implemented?
  4. When it comes to streaming the chunks, I assume we need to wait for all chunks to be streamed, and for the stream to end before re-calling the API. Correct?

This is a high level view of how I’m processing the stream. I can confirm that the second call to the API is structured properly, if my understanding is correct.

const message = req.query.q;
const responseStream = await chatRequest(chatHistory, message); // my API call function

responseStream.on('data', async (chunk) => {
	// iterate through chunks
	// open front-end stream, stream content
	// if stop reason, stop streaming to front-end
	// if tool calls, capture tool function name, tool call ID, args
})

responseStream.on('end', async () => {
	// if tool calls were flagged
      chatHistory.push({
        tool_call_id: toolCallContent.id,
        role: "tool",
        name: toolCallContent.functionName,
        content: toolCallMessage,
      });

	const secondResponse = await chatRequest(chatHistory); // the function can handle context as the only arg
})