Issue: Reasoning models for which OpenAI doesn’t support image input (which one has to discover themselves due to poor documentation), give a quite unhelpful message - insinuating that it might work if you were to send an image file as a base64 data URL. This occurs when using a web URL to be retrieved.
{
"role": "user",
"content": [
{"type": "text", "text": "Briefly: Describe the image contents"},
{
"type": "image_url",
"image_url": {
"url": "https://i.imgur.com/image_link.jpeg",
"detail": "low"
}
},
]
}
Testing model: o1-preview
o1-preview raised Error code: 400 - {'error': {'message': "Invalid image URL: 'messages[1].content[1].image_url.url'. Expected a base64-encoded data URL with an image MIME type (e.g. 'data:image/png;base64,aW1nIGJ5dGVzIGhlcmU='), but got a value without the 'data:' prefix.", 'type': 'invalid_request_error', 'param': 'messages[1].content[1].image_url.url', 'code': 'invalid_value'}}
Testing model: o1-mini
o1-mini raised Error code: 400 - {'error': {'message': "Invalid image URL: 'messages[1].content[1].image_url.url'. Expected a base64-encoded data URL with an image MIME type (e.g. 'data:image/png;base64,aW1nIGJ5dGVzIGhlcmU='), but got a value without the 'data:' prefix.", 'type': 'invalid_request_error', 'param': 'messages[1].content[1].image_url.url', 'code': 'invalid_value'}}
Testing model: o3-mini
o3-mini raised Error code: 400 - {'error': {'message': "Invalid image URL: 'messages[1].content[1].image_url.url'. Expected a base64-encoded data URL with an image MIME type (e.g. 'data:image/png;base64,aW1nIGJ5dGVzIGhlcmU='), but got a value without the 'data:' prefix.", 'type': 'invalid_request_error', 'param': 'messages[1].content[1].image_url.url', 'code': 'invalid_value'}}
Testing model: o1
It shows five adults standing side by side in a casual indoor setting. They are in front of a wall displaying multiple paintings of animals, and a couch is positioned behind them.
Testing model: gpt-4o
The image shows five people standing indoors in front of a wall displaying various animal paintings. They are posing for the photo, with three men and two women in casual clothing. They are standing on or near a brown sofa, and the room has a bright ceiling light.
Summary:
Instead of blocking by a “type”: “image_url”, the validator is continuing into the the contents and giving a plausible validation “expectation”.
Sending base64 gets a useful API error:
messages = [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Analyze this image using built in vision, extracting text.",
},
{
"type": "image_url",
"image_url": {"url": f"data:image/png;base64,{base64_encoded_image}",
"detail": "low"},
},
],
}
]
Testing o1-mini
openai.BadRequestError: Error code: 400 - {‘error’: {‘message’: ‘Invalid content type. image_url is only supported by certain models.’, ‘type’: ‘invalid_request_error’, ‘param’: ‘messages.[0].content.[1].type’, ‘code’: None}}
The only “authority” on whether images are supported is by the appearance of an image price calculator in the pricing list.