Gpt-4o api giving wrong response for image type questions

[mathu]

I am facing problem with chat completion api. I am using gpt-4o for image type question. when I gave the wrong answer for the question, I got the response as correct. I have tried to fix it by changing the value in prompt but it is returning the same. I there any way to prevent this issue.

The prompt I used,

{
  "model" => "gpt-4o",
  "messages" => [{
           "role" => "system",
           "content" => "You are a tutor. Ask questions and check if the student's answers are correct according to the image attached. Provide feedback accordingly. Give a chance for the student to reply. Teach about #{title}.Show the equations in mathematical format. Display the following LaTeX equation without using any backslashes in the final output."
         },
         {
           "role" => "user",
           "content" => [{
                           "type" => "text",
                           "text" => "Don't give the student the answer immediately have a conversation with the student and ensure the student understands the concept. The image for the question is attached. #{title}."
                         },
                         {
                           "type" => "image_url",
                           "image_url" => {
                             "url" => "url"
                           },
                         }]
         },
         {
           "role" => "assistant",
           "content" => "Great! Let's analyze this line graph together to determine between which two consecutive days the peak temperature decreased the most.First, let's identify the high 
                        temperatures for each day Sunday to Saturday Could you please tell me the high temperature for each of these days as indicated on the graph?"
         },
         {
          "role" => "user",
          "content" => "sunday - 62"
         },
         {
          "role" => "assistant",
          "content" => "Excellent, Sunday has a high temperature of 62°F.Let's continue. Can you find the high temperatures for Monday, Tuesday, Wednesday, Thursday, Friday, and Saturday?"
         },
         {
          "role" => "user",
          "content" => "what is the temperature on sunday"
         },
         {
          "role" => "assistant",
          "content" => "The high temperature on Sunday is 66°F. Now, let's move on to Monday. What is the high temperature for Monday?"
         }],
  "temperature" => 0.8
}

Please check the respone and suggest a way to prevent this?

[dignity_for_all]

@mathu

high temperature for each of these days

The way the question is posed by the model seems to be a bit of a trick question.

The highest temperature for each day should not be read from the graph, but the model has turned this into a problem.

The last part, “Excellent, Sunday has a high temperature of 62°F,” is not an incorrect answer. Even if you change the last part to a different model, the model will still respond as correct.

I will share GPT-4 Turbo’s answer for comparison.
https://platform.openai.com/playground/p/dLCloIuiid62RPnWBQoFHw6g

For Sunday, the interpretation would be that the high temperature is 62.

Omitting “High” from the graph title seems to make the model ask a meaningful question.

[mathu]

In the place of title I have mentioned to find the high temperature.
title = “Refer to the line graph, which tracks the high temperature in a town over one calendar week.
Of the days shown, between which two consecutive days did the peak temperature decrease the most?”

[_j]

@mathu

suggest a way to prevent this?

Provide the data in a manner that does not require reliance on vision.

---

User: Use your vision skill to extract the daily high temperatures from the attached image. Hint: you’ll need to pay close attention to the values along the x and y axis.

Sunday: 66°F
Monday: 70°F (should be 69)
Tuesday: 64°F(should be 63)
Wednesday: 61°F (should be 60)
Thursday: 58°F (should be 59)
Friday: 62°F (should be 61)
Saturday: 60°F

[dignity_for_all]

You can keep the image title as it is, but you need to set the title for the model as “#{Find the highest temperature of the week}”.
Or
“#{Refer to the line graph that showing the temperature in a city over a calendar week.
Of the days shown, between which two consecutive days did the temperature decrease the most?}”.

It depends on what kind of questions you want the model to ask.