Assistants "tool_choice" finally available but ends in an infinite loop while streaming

API Bugs
1

Last night I checked the assistant api docs and found out that the tool_choice parameter is finally available. So now we can force the use of tool functions without adding retry loops and more and more instructions :wink:

I implemented this immediately and what can I say… :slight_smile: it works but I can never get the thread to run into ‘completed’ state.

My use case is to enter a request and force the tool function to dynamically request data. Via submitToolOutputs the thread should then be completed and a response message should be delivered by the assistant.

Do you have similar use cases and perhaps an idea on how to complete the run manually? one possibility, as I have now implemented, is to manually cancel the run, have the tool function generate a corresponding response and then manually add this to the thread as an assistant message. but it’s not the yellow from the egg :wink:

Cheers

2 Likes
2

FYI:
Today I played around in the playground and it seems there could be a bug or I misunderstood something.

When I use streaming activated in playground settings the above described behaviour still persists (tool_choice calls are looped and there is no way to end it).

But when I switch to non-streaming everything work as expected (first call with tool choice, second one devlivers an answer and completes the run).

** UPDATE **
I was able to confirm through further tests that there is obviously a bug in the submitToolOutputs node as soon as “streaming: true” is set. It always ends in an infinite loop of “requires_action”.

The solution approach described above worked, but it’s not really practicable. Accordingly, I have now solved it by starting the (first) run WITH “stream: true” until the “requires_action” event is triggered. Then I use “submitToolOutputs” WITHOUT “stream: true” and then use a while loop to call the run status until it goes to “completed”. This is because the run is completed at some point instead of triggering “requires_action” again.

Still not a nice solution, because I would have like to have an “observeStream” function running recursively, but well, maybe this will be fixed soon.

I’ll post my observations here in case anyone runs into the same problem.

Cheers

2 Likes
3

Hey! Sorry for the trouble here. I think we figured out what was wrong here – could you try again and let us know if it works now?

Thanks!

2 Likes
4

Hey @nikunj

thanks for your response! It looks like it’s still the same problem.

Attached you can find my (minified) code (node.js):

async function handleAssistantRun(threadId, assistantData, message) {
	const initialRunData = { run: null, runArguments: {} };
	if (!threadId?.length || !assistantData?.assistant_id?.length || !message?.length) return initialRunData;

	try {
		const runProps = {
			assistant_id: assistantData.assistant_id,
			additional_messages: [{ role: "user", content: message }],
			tool_choice: { type: "function", function: { name: "createNewResponse" } },
			temperature: assistantData.config.assist.temperature * 0.01,
			stream: true,
		};

		const runStream = await openai.beta.threads.runs.create(threadId, runProps);

        return await observeStream(runStream, assistantData, threadId);
	} catch (error) {
		console.error("Error in handleAssistantRun:", error);
		return initialRunData;
	}
}

async function observeStream(stream, assistantData, threadId, actionAlreadyRequired = false) {
    let runArguments = {};

    for await (const chunk of stream) {
        const { event, data } = chunk;

        if (event === 'thread.run.requires_action') {
            if (!actionAlreadyRequired) {
                const actionData = await handleRequiresAction(data, threadId);
                runArguments = actionData.runArguments;
                const result = await observeStream(actionData.actionRunStream, assistantData, threadId, true);

                return { run: result.run, runArguments };
            } else {
                return { run: data, runArguments };
            }
        } else if (['thread.run.completed', 'done', 'end'].includes(event)) {
            return { run: data, runArguments };
        } else if (['thread.run.failed', 'thread.run.cancelled', 'thread.run.expired', 'error'].includes(event)) {
            console.error('Unable to complete request. Data: ' + JSON.stringify(data));
            return { run: event === 'error' ? null : data, runArguments };
        }
    }
}

async function handleRequiresAction(run, threadId) {
    const initialActionRunStreamData = { runActionStream: null, runArguments: {} };

    if (!run?.required_action?.submit_tool_outputs?.tool_calls?.length) {
        console.error(`Unable to complete the request. No valid tool calls provided.`);
        return initialActionRunStreamData;
    }

    const toolCalls = run.required_action.submit_tool_outputs.tool_calls;
    const toolOutputs = [];
    let runArguments = {};

    try {
        for (const toolCall of toolCalls) {
            if (toolCall.function?.name !== 'createNewResponse') {
                toolOutputs.push({
                    tool_call_id: toolCall.id,
                    output: JSON.stringify({ success: true }),
                });

                continue;
            }

            runArguments = JSON.parse(toolCall.function?.arguments) || {};

            toolOutputs.push({
                tool_call_id: toolCall.id,
                output: JSON.stringify({response: "Some response"}),
            });
        }

        const actionRunStream = await openai.beta.threads.runs.submitToolOutputs(threadId, run.id, {
            stream: true,
            tool_outputs: toolOutputs,
        });

        return { actionRunStream, runArguments };
    } catch (error) {
        console.error('Error in handleRequiresAction:', error);
        return initialActionRunStreamData;
    }
}

It stops after the second call of observeStream via the else block (if (event === 'thread.run.requires_action')). I’m using "openai": "^4.33.1" npm module.

** UPDATE 1 **
I haven’t actually changed anything, but I can’t rule out the possibility that I haven’t updated something correctly somewhere. Anyway, it works now! Many thanks for the effort!

** UPDATE 2 **
@nikunj It looks like most of the time it works, but sometimes it still triggers an additional “requires_action” unfortunately. I think there is still a bug (only randomly reproducible after some runs).

5

Thanks - fixed a couple more things so let’s give it one more try. Let me know if it works! Really appreciate your help!

2 Likes
6

It looks like it is working as expected and consistently. Thanks again for your fast help!

1 Like