"Invalid image" error in gpt-4-vision

lena · November 15, 2023, 11:28pm

I’m sending several URLs in a single request to GPT-4 Vision, and three out of five times I get not informative error message: ‘Invalid image.’, type: ‘invalid_request_error’. The URLs change dynamically, so predicting the set of these URLs is impossible.

I’ve collected the problematic images but still can’t identify the issue with them. I have checked all the links manually; they are all in PNG, JPEG, or WEBP format, as per the requirements. They vary in format, size, and domain, so no common parameters are evident. Moreover, the API doesn’t specify the exact problem.

Here are examples of the problematic URLs:

The most frustrating part is that if even one image causes an issue, the entire request fails, preventing the handling of other data. The API also doesn’t indicate which specific URL is problematic, does not skip it, so it’s impossible to dynamically handle the error.

Here’s an example of the request:

model: 'gpt-4-vision-preview',
messages: [
    {
        role: 'user',
        content: [
            { type: 'text', text: 'What image URL is the most relevant for the topic {topic}? Return the URL and describe what this picture is about.' },

            {
                type: 'image_url',
                image_url: {
                     'url': 'https://i.ytimg.com/vi/qRE0WicGz4I/maxresdefault.jpg',
                    'detail': 'auto'
                },
            },

            {
                type: 'image_url',
                image_url: {
              'url': 'https://jdadvising.com/wp-content/uploads/2014/12/Sample-Daily-Bar-Exam-Study-Schedule.png',
                    'detail': 'auto'
                },
            },
            {
                type: 'image_url',
                image_url: {
                     'url': 'https://shovelapp.io/wp-content/uploads/2022/07/Study-Schedule-needs-to-show-all-your-time-commitments-1024x576.png',
                    'detail': 'auto'
                },
            },

            {
                type: 'image_url',
                image_url: {
                    'url': 'https://info.methodlearning.com/hubfs/schedule.jpg',
                    'detail': 'auto'
                },
            },
            {
                type: 'image_url',
                image_url: {
                    'url': 'https://i.ytimg.com/vi/qRE0WicGz4I/hqdefault.jpg',
                    'detail': 'auto'
                },
            },
        ]
    },
],
temperature: 0.1,
max_tokens: 500,
},

How can I localise the problem and what does this error mean?

PaulBellow · November 15, 2023, 11:44pm

Does it always fail on the same images?

I wonder if some form of hotlinking protection might be causing some to fail if they’re not being called directly from the domain in question?

lena · November 15, 2023, 11:54pm

Yes, they are the same images. When I removed them the problem disappeared.

I thought about hotlinking, but I’m not sure how it works. Does hotlinking protection mean that I cannot paste these image URLs on any other site? Because I can easily insert these problematic pictures, and they work fine on the Notion page Notion – The all-in-one workspace for your notes, tasks, wikis, and databases.

PaulBellow · November 15, 2023, 11:57pm

A webmaster can set-up their webserver so that images will only load if called from the host domain (or whitelisted domains…) So, they might have Notion whitelisted for hotlinking (due to benefits they receive from it?) while all other domains (like OpenAI’s that are calling the image) get a bad response OR in a bad case, an image that’s NOTHING like the image shown on their website. This isn’t as big of a problem as it was in the early days of the web, but I have to wonder if it’s happening here. Test would be to try embedding/loading that image with an <img tag in an html page maybe?

sps · November 16, 2023, 12:41am

Hi @lena

Welcome to the dev forum.

Does the error go away when you set detail to auto for all images?

lena · November 16, 2023, 8:20am

I’ve tried these images with tags, they are working fine. It seems the problem is not in hotlinking protection but in OpenAI API.
The example ↓

lena · November 16, 2023, 8:22am

Thank you, but no, all images have this parameter in my request and this doesn’t help. An example of a request in my first post.

Foxabilo · November 16, 2023, 8:32am

Have you tried making 4 calls in parallel with one image each?

lena · November 16, 2023, 8:42am

No, but I’ll try, there may be a problem with several images. However, in my case, I need to compare 5 images and find the most relevant one.

Upd. Actually, yes, I tried all of these images one by one, that is how I found the problematic ones. Also, I tried different variations with multiple images, and the request can work fine if there are no problematic images in the set.

Foxabilo · November 16, 2023, 8:56am

I think the thing I’d do is cache the images locally, resize them to 512x512 and base64 and link them that way… no url funny business then, or at least any that did happen would be under my codes error detection and retry control.

kjordan · November 16, 2023, 9:02am

I did some quick tests on these 2 images and both work fine by directly uploading to OpenAI, so the image files should be fine. One possible reason is that OpenAI’s server IPs are blocked by the original image hosts.

Foxabilo · November 16, 2023, 9:03am

I think there will be an increasing number of data pipes that refuse certain OpenAI associated IP’s… I think they are making a mistake by doing so, but that’s their business.

sps · November 16, 2023, 10:56am

Just tried the problematic urls you shared and they work fine:

message=ChatCompletionMessage(content='The first image is a detailed template of a Daily Study Planner. It includes various sections to help organize a student\'s day, such as "Wake Up At" and "Sleep At" times, "Hours to Study" with planned and actual hours, and a comprehensive "Timetable" starting at 6:00 AM with preparation and ending at 22:00 with rest. The planner also lists "Today\'s Subjects," "Deadlines" for assignments, "Goals" for the day, and "Study Progress," indicating what has been accomplished.\n\nThe second image illustrates a colorful weekly schedule board. The board is divided into days of the week from Monday to Friday with different colored sticky notes attached to each day, indicating various tasks or appointments. Two hands are interacting with the board; one is placing a sticky note while the other holds a smartphone, possibly referring to coordination with a digital calendar. The board is mounted on a brick wall background, suggesting an informal or creative workspace, such as a startup office or a student’s study area.', role='assistant', function_call=None, tool_calls=None), finish_details={'type': 'stop', 'stop': '<|fim_suffix|>'})

Here’s my code:

response = client.chat.completions.create(
    model="gpt-4-vision-preview",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe the images."},
                {
                    "type": "image_url",
                    "image_url": "https://s3.amazonaws.com/thumbnails.venngage.com/template/224aa448-5adf-4287-aca7-ce09cc177fd8.png",
                },
                {
                    "type": "image_url",
                    "image_url": "https://info.methodlearning.com/hubfs/schedule.jpg",
                },
            ],
        }
    ],
    max_tokens=1500,
)

print(response.choices[0])

lena · November 16, 2023, 11:12am

Wow, this is strange, I’ll double-check with your request, I see you didn’t pass the ‘detail’ parameters. Thank you for trying it!

Upd: It works for me as well. I removed the detail and temperature parameters and added max-tokens. I am just a bit confused as to why did you pass the string for “image_url” instead of an object. But nevertheless, for an object, the problem was fixed as well. Thank you all for help!

 model: 'gpt-4-vision-preview',
 messages: [
                {
                    role: 'user',
                    content: [
                        { type: 'text', text: 'Describe the images.' },
                        {
                            type: 'image_url',
                            image_url: {url: 'https://s3.amazonaws.com/thumbnails.venngage.com/template/224aa448-5adf-4287-aca7-ce09cc177fd8.png'},
                        },
                        {
                            type: 'image_url',
                            image_url: {url: 'https://info.methodlearning.com/hubfs/schedule.jpg'},
                        },
                    ],
                }
            ],
  max_tokens: 1500,

sps · November 16, 2023, 12:16pm

@lena I had the old boilerplate code from docs saved in a file which I edited just to include multiple images. It looks like the docs were updated after I had copied the boilerplate from the docs that were originally published on DevDay.

@kjordan She’s not wrong, just following the documentation which is listed as such and happens to be correct:

kjordan · November 16, 2023, 12:22pm

right… I’m wrong. Then it’s a puzzle again. Thanks for pointing it out.

zhang.linghan62 · November 21, 2023, 3:58pm

@lena Even with the exact same request (same image and settings), ~20% of my requests fail. So I’m fairly confident it’s an issue on OpenAI end. I suspect some of their server instances might have issue downloading the image. I am converting my images into b64 encoded before requesting in hopes it would have a better success rate