API Reasoning models without image support returning bad validation error message

Issue: Reasoning models for which OpenAI doesn’t support image input (which one has to discover themselves due to poor documentation), give a quite unhelpful message - insinuating that it might work if you were to send an image file as a base64 data URL. This occurs when using a web URL to be retrieved.

{
        "role": "user",
        "content": [
            {"type": "text", "text": "Briefly: Describe the image contents"},
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://i.imgur.com/image_link.jpeg",
                    "detail": "low"
                }
            },
        ]
    }

Testing model: o1-preview

o1-preview raised Error code: 400 - {'error': {'message': "Invalid image URL: 'messages[1].content[1].image_url.url'. Expected a base64-encoded data URL with an image MIME type (e.g. 'data:image/png;base64,aW1nIGJ5dGVzIGhlcmU='), but got a value without the 'data:' prefix.", 'type': 'invalid_request_error', 'param': 'messages[1].content[1].image_url.url', 'code': 'invalid_value'}}


Testing model: o1-mini

o1-mini raised Error code: 400 - {'error': {'message': "Invalid image URL: 'messages[1].content[1].image_url.url'. Expected a base64-encoded data URL with an image MIME type (e.g. 'data:image/png;base64,aW1nIGJ5dGVzIGhlcmU='), but got a value without the 'data:' prefix.", 'type': 'invalid_request_error', 'param': 'messages[1].content[1].image_url.url', 'code': 'invalid_value'}}


Testing model: o3-mini

o3-mini raised Error code: 400 - {'error': {'message': "Invalid image URL: 'messages[1].content[1].image_url.url'. Expected a base64-encoded data URL with an image MIME type (e.g. 'data:image/png;base64,aW1nIGJ5dGVzIGhlcmU='), but got a value without the 'data:' prefix.", 'type': 'invalid_request_error', 'param': 'messages[1].content[1].image_url.url', 'code': 'invalid_value'}}


Testing model: o1

It shows five adults standing side by side in a casual indoor setting. They are in front of a wall displaying multiple paintings of animals, and a couch is positioned behind them.


Testing model: gpt-4o

The image shows five people standing indoors in front of a wall displaying various animal paintings. They are posing for the photo, with three men and two women in casual clothing. They are standing on or near a brown sofa, and the room has a bright ceiling light.

Summary:

Instead of blocking by a “type”: “image_url”, the validator is continuing into the the contents and giving a plausible validation “expectation”.

Sending base64 gets a useful API error:

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Analyze this image using built in vision, extracting text.",
            },
            {
                "type": "image_url",
                "image_url": {"url": f"data:image/png;base64,{base64_encoded_image}",
                              "detail": "low"},
            },
        ],
    }
]

Testing o1-mini

openai.BadRequestError: Error code: 400 - {‘error’: {‘message’: ‘Invalid content type. image_url is only supported by certain models.’, ‘type’: ‘invalid_request_error’, ‘param’: ‘messages.[0].content.[1].type’, ‘code’: None}}


The only “authority” on whether images are supported is by the appearance of an image price calculator in the pricing list.