Hello there,
I want to create a custom assistant that helps with a building. The prompt of the user could lead to different results, e.g. answering the queston and highlighting specific elements in the 3D model of a building in the client.
That’s why each user prompt needs to do a file search to extract necessary info from the building elements in a file.
Afterwards the wished tool function should be executed, e.g. highlightElements or something similar.
My problem is that this is executed properly (and I can log the correct paramteres in my Backend call, e.g. with correct highlightIds) but the results of that toolOutputs aren’t stored on the thread/inside the messages.
So, when I render the messages in the frontend, I have no option to recall the tools output that was created together with the message.
return AssistantResponse(
{ threadId, messageId: createdMessage.id },
async ({ forwardStream }) => {
let runResult = await forwardStream(
openai.beta.threads.runs.stream(threadId, {assistant_id }),
);
while (
runResult?.status === 'requires_action' &&
runResult.required_action?.type === 'submit_tool_outputs'
) {
const toolOutputs = runResult.required_action.submit_tool_outputs.tool_calls.map(
(toolCall: any) => {
const parameters = JSON.parse(toolCall.function.arguments);
// Here the correct parameters and function name are logged e.g. highlightIds: [1,2,3,4]
console.log('Tool call:', toolCall.function.name, parameters);
return {
tool_call_id: toolCall.id,
output: JSON.stringify({ parameters }),
};
},
);
// here the correct toolOutpus are logged as well
console.log('Submitting tool outputs:', toolOutputs);
// This results in a message like "The elements have been highlighted." The highlightIds (output) are stored nowhere
runResult = await forwardStream(
openai.beta.threads.runs.submitToolOutputsStream(threadId, runResult.id, {
tool_outputs: toolOutputs,
}),
);
}
},
);
How could I store the toolOutput together with a message, so in my Frontend I can render a custom message for each tool output (with custom controls to highlight elements or similar)