"Invalid image" error in gpt-4-vision

I’m sending several URLs in a single request to GPT-4 Vision, and three out of five times I get not informative error message: ‘Invalid image.’, type: ‘invalid_request_error’. The URLs change dynamically, so predicting the set of these URLs is impossible.

I’ve collected the problematic images but still can’t identify the issue with them. I have checked all the links manually; they are all in PNG, JPEG, or WEBP format, as per the requirements. They vary in format, size, and domain, so no common parameters are evident. Moreover, the API doesn’t specify the exact problem.

Here are examples of the problematic URLs:

  1. https://s3.amazonaws.com/thumbnails.venngage.com/template/224aa448-5adf-4287-aca7-ce09cc177fd8.png
  2. https://info.methodlearning.com/hubfs/schedule.jpg

The most frustrating part is that if even one image causes an issue, the entire request fails, preventing the handling of other data. The API also doesn’t indicate which specific URL is problematic, does not skip it, so it’s impossible to dynamically handle the error.

Here’s an example of the request:

model: 'gpt-4-vision-preview',
messages: [
    {
        role: 'user',
        content: [
            { type: 'text', text: 'What image URL is the most relevant for the topic {topic}? Return the URL and describe what this picture is about.' },

            {
                type: 'image_url',
                image_url: {
                     'url': 'https://i.ytimg.com/vi/qRE0WicGz4I/maxresdefault.jpg',
                    'detail': 'auto'
                },
            },

            {
                type: 'image_url',
                image_url: {
              'url': 'https://jdadvising.com/wp-content/uploads/2014/12/Sample-Daily-Bar-Exam-Study-Schedule.png',
                    'detail': 'auto'
                },
            },
            {
                type: 'image_url',
                image_url: {
                     'url': 'https://shovelapp.io/wp-content/uploads/2022/07/Study-Schedule-needs-to-show-all-your-time-commitments-1024x576.png',
                    'detail': 'auto'
                },
            },

            {
                type: 'image_url',
                image_url: {
                    'url': 'https://info.methodlearning.com/hubfs/schedule.jpg',
                    'detail': 'auto'
                },
            },
            {
                type: 'image_url',
                image_url: {
                    'url': 'https://i.ytimg.com/vi/qRE0WicGz4I/hqdefault.jpg',
                    'detail': 'auto'
                },
            },
        ]
    },
],
temperature: 0.1,
max_tokens: 500,
},

How can I localise the problem and what does this error mean?

1 Like

Does it always fail on the same images?

I wonder if some form of hotlinking protection might be causing some to fail if they’re not being called directly from the domain in question?

Yes, they are the same images. When I removed them the problem disappeared.

I thought about hotlinking, but I’m not sure how it works. Does hotlinking protection mean that I cannot paste these image URLs on any other site? Because I can easily insert these problematic pictures, and they work fine on the Notion page Notion – The all-in-one workspace for your notes, tasks, wikis, and databases.

1 Like

A webmaster can set-up their webserver so that images will only load if called from the host domain (or whitelisted domains…) So, they might have Notion whitelisted for hotlinking (due to benefits they receive from it?) while all other domains (like OpenAI’s that are calling the image) get a bad response OR in a bad case, an image that’s NOTHING like the image shown on their website. This isn’t as big of a problem as it was in the early days of the web, but I have to wonder if it’s happening here. Test would be to try embedding/loading that image with an <img tag in an html page maybe?

2 Likes

Hi @lena

Welcome to the dev forum.

Does the error go away when you set detail to auto for all images?

2 Likes

I’ve tried these images with tags, they are working fine. It seems the problem is not in hotlinking protection but in OpenAI API.
The example ↓

Thank you, but no, all images have this parameter in my request and this doesn’t help. An example of a request in my first post.

Have you tried making 4 calls in parallel with one image each?

No, but I’ll try, there may be a problem with several images. However, in my case, I need to compare 5 images and find the most relevant one.

Upd. Actually, yes, I tried all of these images one by one, that is how I found the problematic ones. Also, I tried different variations with multiple images, and the request can work fine if there are no problematic images in the set.

I think the thing I’d do is cache the images locally, resize them to 512x512 and base64 and link them that way… no url funny business then, or at least any that did happen would be under my codes error detection and retry control.

1 Like

I did some quick tests on these 2 images and both work fine by directly uploading to OpenAI, so the image files should be fine. One possible reason is that OpenAI’s server IPs are blocked by the original image hosts.

2 Likes

I think there will be an increasing number of data pipes that refuse certain OpenAI associated IP’s… I think they are making a mistake by doing so, but that’s their business.

Just tried the problematic urls you shared and they work fine:

message=ChatCompletionMessage(content='The first image is a detailed template of a Daily Study Planner. It includes various sections to help organize a student\'s day, such as "Wake Up At" and "Sleep At" times, "Hours to Study" with planned and actual hours, and a comprehensive "Timetable" starting at 6:00 AM with preparation and ending at 22:00 with rest. The planner also lists "Today\'s Subjects," "Deadlines" for assignments, "Goals" for the day, and "Study Progress," indicating what has been accomplished.\n\nThe second image illustrates a colorful weekly schedule board. The board is divided into days of the week from Monday to Friday with different colored sticky notes attached to each day, indicating various tasks or appointments. Two hands are interacting with the board; one is placing a sticky note while the other holds a smartphone, possibly referring to coordination with a digital calendar. The board is mounted on a brick wall background, suggesting an informal or creative workspace, such as a startup office or a student’s study area.', role='assistant', function_call=None, tool_calls=None), finish_details={'type': 'stop', 'stop': '<|fim_suffix|>'})

Here’s my code:

response = client.chat.completions.create(
    model="gpt-4-vision-preview",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe the images."},
                {
                    "type": "image_url",
                    "image_url": "https://s3.amazonaws.com/thumbnails.venngage.com/template/224aa448-5adf-4287-aca7-ce09cc177fd8.png",
                },
                {
                    "type": "image_url",
                    "image_url": "https://info.methodlearning.com/hubfs/schedule.jpg",
                },
            ],
        }
    ],
    max_tokens=1500,
)

print(response.choices[0])
2 Likes

Wow, this is strange, I’ll double-check with your request, I see you didn’t pass the ‘detail’ parameters. Thank you for trying it!

Upd: It works for me as well. I removed the detail and temperature parameters and added max-tokens. I am just a bit confused as to why did you pass the string for “image_url” instead of an object. But nevertheless, for an object, the problem was fixed as well. Thank you all for help!

 model: 'gpt-4-vision-preview',
 messages: [
                {
                    role: 'user',
                    content: [
                        { type: 'text', text: 'Describe the images.' },
                        {
                            type: 'image_url',
                            image_url: {url: 'https://s3.amazonaws.com/thumbnails.venngage.com/template/224aa448-5adf-4287-aca7-ce09cc177fd8.png'},
                        },
                        {
                            type: 'image_url',
                            image_url: {url: 'https://info.methodlearning.com/hubfs/schedule.jpg'},
                        },
                    ],
                }
            ],
  max_tokens: 1500,
1 Like

@lena I had the old boilerplate code from docs saved in a file which I edited just to include multiple images. It looks like the docs were updated after I had copied the boilerplate from the docs that were originally published on DevDay. :joy:

@kjordan She’s not wrong, just following the documentation which is listed as such and happens to be correct:

2 Likes

right… I’m wrong. Then it’s a puzzle again. Thanks for pointing it out.

1 Like

@lena Even with the exact same request (same image and settings), ~20% of my requests fail. So I’m fairly confident it’s an issue on OpenAI end. I suspect some of their server instances might have issue downloading the image. I am converting my images into b64 encoded before requesting in hopes it would have a better success rate

1 Like

Do you have better results with base64 encoding ?

I’m seeing this too.

I’m using the same image files but with different URL’s as I’m coming from signedurl’s from supabase object storage & so the url is a file reference + the token for signed url access. As I’ve been testing over a number of days, the signedURL’s change.

The url works fine in a browser always, but every now and then OpenAI will reject a file I know it’s worked fine on previously

my code is

const chatCompletion = await openai.chat.completions.create({
messages: [{
role: ‘user’,
content: [
{ type: ‘text’, text: 'given the attachment, extract the data from this image in json in the format below. If you can't populate a field, put null. If you are unsure of the result, add another property for 'dirty' and make it true. { json structure here },

    {
      type: 'image_url',
      image_url: {
        url: cover_url,
        detail: 'low'
      }
    },
  ],
}],
model: 'gpt-4-vision-preview',
max_tokens: 150,

});

I was also about to try the base64 encoding but will also try omitting the detail property. I have a feeling it’s the detail one that will solve it as I reckon it’s the downscaling system that’s having the issue.

I’m also seeing the same issue. I am using the same image coming from S3 as a presigned url.

Sometimes an Invalid Image error response is returned even though calls milliseconds apart using the same image url successfully complete.

Did anyone figure out a fix for this issue?