Storing tool outputs on messages/thread

Hello there,
I want to create a custom assistant that helps with a building. The prompt of the user could lead to different results, e.g. answering the queston and highlighting specific elements in the 3D model of a building in the client.
That’s why each user prompt needs to do a file search to extract necessary info from the building elements in a file.
Afterwards the wished tool function should be executed, e.g. highlightElements or something similar.
My problem is that this is executed properly (and I can log the correct paramteres in my Backend call, e.g. with correct highlightIds) but the results of that toolOutputs aren’t stored on the thread/inside the messages.
So, when I render the messages in the frontend, I have no option to recall the tools output that was created together with the message.

return AssistantResponse(
      { threadId, messageId: createdMessage.id },
      async ({ forwardStream }) => {
        let runResult = await forwardStream(
          openai.beta.threads.runs.stream(threadId, {assistant_id }),
        );

        while (
          runResult?.status === 'requires_action' &&
          runResult.required_action?.type === 'submit_tool_outputs'
        ) {
          const toolOutputs = runResult.required_action.submit_tool_outputs.tool_calls.map(
            (toolCall: any) => {

              const parameters = JSON.parse(toolCall.function.arguments);
              // Here the correct parameters and function name are logged e.g. highlightIds: [1,2,3,4]
              console.log('Tool call:', toolCall.function.name, parameters);

              return {
                tool_call_id: toolCall.id,
                output: JSON.stringify({  parameters  }),
              };
            },
          );

          // here the correct toolOutpus are logged as well
          console.log('Submitting tool outputs:', toolOutputs);

          // This results in a message like "The elements have been highlighted." The highlightIds (output) are stored nowhere
          runResult = await forwardStream(
            openai.beta.threads.runs.submitToolOutputsStream(threadId, runResult.id, {
              tool_outputs: toolOutputs,
            }),
          );
        }
      },
    );

How could I store the toolOutput together with a message, so in my Frontend I can render a custom message for each tool output (with custom controls to highlight elements or similar)

You could consider (if not too many tools) storing the outputs in the thread metadata at the time you submit them?

Thanks for your answer, I also stumbled accross this metadata property of a message/thread. But it looks like it is quite limited:

https://platform.openai.com/docs/api-reference/messages/object:
Keys can be a maximum of 64 characters long and values can be a maximum of 512 characters long

This could be a quick solution but not a long-term solid solution for this use case.

Would be nice to attach additional structured information to each message but it looks like currently this is not supported

1 Like