API ISSUE: "Responses" endpoint: using vision with user image is only errors (now fixed)

[_j]

In a recent task, I had to switch from Responses to Chat Completions in the playground (which works on the same context).

A user input message with an image, or “additional messages” with an image added (depicted), results in an error, replicated by sending the same to either “gpt-4.5-preview” or “gpt-4o-2024-11-20” or others.

The image was placed by pasting a clipboard of image contents into the text focus, where it appeared.

Instead linking right to a forum topic’s image URL with a paperclip press populates the UI with the image, but then sending fails - a quick timeout continues, even upon retries, same with another image source:

The timeout is happening with local Python also:

input: list[dict] = [
    {
      "role": "user",
      "content": [
        {
          "type": "input_image",
          "image_url": "https://images.unsplash.com/photo-1461988320302-91bde64fc8e4?ixid=2yJhcHBfaWQiOjEyMDd9"
        },
        {
          "type": "input_text",
          "text": "extract any image text from images"
        }
      ]
    }
  ]

event: error
data: {"type":"error","error":{"type":"invalid_request_error","code":"invalid_image_url","message":"Timeout while downloading https://images.unsplash.com/photo-1461988320302-91bde64fc8e4?ixid=2yJhcHBfaWQiOjEyMDd9.","param":null}}

Conclusion:

Looking at stuff with Responses is currently broken

[Diet]

Ah, this could be what this user might have been reporting:

What's going on with gpt4o today?

We’re getting the same issues too in relation to API calls… We’re using images and its throwing 400 Timeout errors. It was working yesterday? { "error": "400 Timeout while downloading https://[REDACTED]/[URL]/image/.....imagetokenhere.......?token=[REDACTED]." }

It seems to be an intermittent network issue on OpenAI’s end

[_j]

@Diet

this could be what this user might have been reporting

Those users having continuous use and now report problems are likely not on the Responses endpoint.

The timeout error on internet URL has been fixed, confirmed locally, but base64 data URL still has the same “An error occurred while processing your request”.

Chat Completions and Responses on the API Playground are sending the same base64 encoding. One just fails. The same trying base64 locally.

The Playground is forming the request properly, otherwise the validation would bounce it:

Chat completions: "content":[{"type":"image_url","image_url":{"url":"data:image/png;base64,iVBORw0...
Responses: "content":[{"type":"input_image","image_url":"data:image/png;base64,iVBORw0KGgoAAAANSU...

[Diet]

Ah, I assumed it might have had the same root cause, but maybe not

[_j]

This got working again. The type is not “image_input”.

Sending an image (in the SDK’s “simple input”) as part of the content of a message, a vision file ID is also supported on Responses:

image_part = {
    "type": "input_image",
    "detail": "low",
    "image_url": f"data:image/png;base64,{base64_encoded_image",
    # OR
    # "file_id": "file_123456",
}

And OpenAI really wants a model to know it can see, even with no image, jamming another system message on the context end right into the existing assistant writing prompt…

<200006>system<200008>Image input capabilities: Enabled<200007><200006>user<200008>Describe image<200007><200006>assistant<200006>system<200008>Image input capabilities: Enabled<200007><200006>user<200008>Describe image<200007><200006>assistant

That’s from only user: "Describe image" when not activating the safety text dump also.