I have an application that mixes AI and python code for sending messages to the user. The model can choose to use a tool send_image to the user, and the tool will send an image to the user.
Right now, when the user interacts again with the agent, I serialize a placeholder assistant message with something like “See this image: [IMAGE SENT]”. Is it possible to properly serialize an image with the assistant role? maybe using the same format as the user’s image message?
The only role message that can contain image data for vision is “user”.
You would not want to give a conversation history with the assistant role message containing things impossible for the AI to produce.
You want to show the chat model how to properly use the tool from past messages, not for it to fantasize about showing you pictures of unknown origin.
You can come up with other clever techniques as long as they don’t confuse the user also. For example, after assistant, tool, assistant, and before you insert the most recent user message on the next chat input, you could insert a user message for vision, “here’s the image contents I just received from your send_image link {image}”.