GTP-4o no longer analyses images

API
,
1

I know there are other similar topics but maybe if this issue becomes more obvious it might get solve, I’ve been using gtp-4o to analyse images via the API to return a desciption in JSON format.

The code has been working for 4-5 weeks then all of a sudden it stopped working yesterday. I now just get this response.

“I can no longer analyse images”, this is NOT helpful. It makes it extremely worrying to launch a commercial product that relies on an OpenAI API if it can just stop working with no warning.

I’ve switched to gtp-4o-mini and the analysis is iffy at best. Sometimes the descriptions are completely off yet re-run on the same image and the analysis is correct.

4

Let’s test out the model for you.

Here is an app that is rather personal, returning eye colors of a person or people (another forum member’s idea). I got immediate failure even though running top_p:0.01 – then successes.

human_eye_colors=[‘unknown’] confidence_1_to_10=-1
human_eye_colors=[‘blue’] confidence_1_to_10=5
human_eye_colors=[‘blue’] confidence_1_to_10=5
human_eye_colors=[‘blue’] confidence_1_to_10=5

This is too small to cache.

Shoot off 20 of them at top_p=0.0001 with a field for the AI to report on image failure:

image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘unknown’] confidence_1_to_10=-1 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘unknown’] confidence_1_to_10=5 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘unknown’] confidence_1_to_10=-1 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘unknown’] confidence_1_to_10=-1 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘unknown’] confidence_1_to_10=-1 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘unknown’] confidence_1_to_10=-1 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘unknown’] confidence_1_to_10=-1 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘blue’] confidence_1_to_10=8 fp_a20a4ee344

It seems I just found a particular case of two close-ranked logprobs that switch because of non-determinism in inference; a different image is not returning any uncertainty:

image_received_successfully=True human_eye_colors=[‘brown’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘brown’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘brown’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘brown’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘brown’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘brown’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘brown’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘brown’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘brown’] confidence_1_to_10=8 fp_a20a4ee344
image_received_successfully=True human_eye_colors=[‘brown’] confidence_1_to_10=8 fp_a20a4ee344

You should leave no place for the AI to write something like “I can no longer analyse images” in your structured output schema. Also, simply add “GPT-4 computer vision for images: enabled” to the system message. See if that helps!

5

This is the message returned from the API call:

“Raw response content: I’m sorry, I cannot assist with this request.”

My out schema was working perfectly for 5-6 weeks with no issues.

I’ve added GPT-4 computer vision for images: enabled and it WORKED, thank you so much. No idea why I should have to though when it worked perfectly previously.

1 Like