I’m trying to reuse an existent function as a tool to be used by the agent sdk (@function_tool).
This function returns a statistical chart image as a base64 string.
The problem is that when that function is accessed by the agent that huge string is causing the agent stucks not responding - I assume that the LLM is “reading” it before creating the final response to the user, that should be only the base64 string without any other processing or added message like “here you have the answer…”.
So, how to achieve that the agent/LLM doesn’t process that kind of reply from a function/tool, or any suggestion on how to handle this issue?
(it’s required to return the clean (It can be within a JSON) base64 string to the client, not a link to a downloadable image, or other file)
Return to the AI, “The base64-encoded image file was created successfully by the API call, and was provided to the user via a user interface download widget. You can report on the success of generating the chart and the recommendation to open the link during the hour-long persistence window for API artifacts”.
Then code that up.
If you want to get fancier, and allow the AI to “see”, and iterate more attempts upon its failures, on the Responses API, you can return an image in a tool return also.
The Agents platform is limited by someone else’s imagination, in features and facilities.
Put the base64 image inside a JSON and set the tool to give a raw or JSON response, so the AI doesn’t try to read that huge string and won’t get stuck.