Gpt4-o Support for Image URLS as tool responses

The previous Gpt4-turbo model supported image urls being part of a tool response which was useful for tools that responded back images to the models as a result of the tool call.

The new gpt4o models have apparently lost this functionality and can only receive images from messages with user role.

Is there a way to workaround this regression or is there any plan to fix the issue?

Example request:

[
	{
		"role": "user",
		"content": [
			{
				"type": "text",
				"text": "Navigate to google.com website."
			}
		]
	},
	{
		"role": "assistant",
		"content": null,
		"tool_calls": [
			{
				"id": "call_MsDL8L4dUgJwxW4UKMW9XK6c",
				"type": "function",
				"function": {
					"name": "navigate-to-website",
					"arguments": "{\"url\":\"https://www.google.com\",\"keywords\":[\"search\",\"Google\"],\"searchDescription\":\"Access the Google homepage to perform web searches and explore Google's services.\"}"
				}
			}
		]
	},
	{
		"role": "tool",
		"tool_call_id": "call_MsDL8L4dUgJwxW4UKMW9XK6c",
		"content": [
			{
				"type": "text",
				"text": "You have navigated to the website."
			},
			{
				"type": "image_url",
				"image_url": {
					"url": "data:image/png;base64,iVBORw0KGgoAA...==",
					"detail": "high"
				}
			}
		]
	}
]

API Response:

{
    "error": {
        "message": "Invalid 'messages[3]'. Image URLs are only allowed for messages with role 'user', but this message with role 'tool' contains an image URL.",
        "type": "invalid_request_error",
        "param": "messages[3]",
        "code": "invalid_value"
    }
}
1 Like

Hi,

That does seem to be deliberate, but I will ask the question, won’t be until next Friday though.

4 Likes

Let me know if you got a response from the team.
Thanks!

Hi, yes, I’ve asked the question. Hopefully get an answer soon.

2 Likes

Hi, any response please?

Thanks