In a conversation with a bot, you might ask it for a picture.
You may then ask it to refine the picture it shared.
This will lead to a natural set of messages where at least one or more assistant entries have an image URL.
This would be fine, but this is illegal in Completions:
{
"error": {
"message": "Invalid 'messages[5]'. Image URLs are only allowed for messages with role 'user', but this message with role 'assistant' contains an image URL.",
"type": "invalid_request_error",
"param": "messages[5]",
"code": "invalid_value"
}
}
Can we please make this legal so we don’t need to add workarounds?
It doesn’t make sense for me to convert the picture message the assistant shared into a user message, because that destroys the logic of the conversation.
The AI cannot actually make pictures for you. You would be producing a simulation that did not in fact happen.
Or rather, the multimodal AI model can make images, but this is not released, and in-context training on how to respond with them could activate this modality.
Tool returns at least should be able to include them. “Here’s the DALL-E image displayed in the user interface. Take appropriate action based on your visual understanding of realized output vs your tool call. ”
In a conversation with a well specced Chatbot it would be normal to have a feature whereby it can create pictures for you. It then owns a response with a picture.
The conversation history now contains a picture in an assistant owned response.
It is stupid to have to change the content or ownership of parts of the history just to accommodate a pointless limitation of the API.
What if you want to discuss the image with the chatbot?
The Chatbot did functionally create the image and owns it. So turning the assistant message into a user message is corrupting the conversation history.
”