Can ChatGPT analyze images using the API?

API
1

I’ve seen conflicting info about this. I know that ChatGPT can analyze images that are manually uploaded into the chatgpt web interface, but is there a way to analyze images with API prompts?

I am using a Javascript node project and I have been able to successfully upload image files to ChatGPT and retrieve their IDs using the API, but when I ask ChatGPT to analyze the images I get responses like “I’m sorry, but I don’t have the capability to analyze images directly. However, I can help you with a description or analysis of the content if you provide a detailed description of the image.”.

I’ve tried using different models and I get the same response with all of them. Is this feature unavailable, or am I approaching it wrong? Here is a snippet of the code that I’m using with the latest openai node package:

const messages = [
  { role: 'system', content: 'Analyze the following images.' },
  { role: 'user', content: 'Image ID: file-12345' } // Placeholder file ID
];

const completion = await openai.chat.completions.create({
  messages: messages,
  model: "gpt-4",
});

Thanks!

2

You need to use the vision format for the messages and use gpt-4-turbo, gpt-4o or gpt-4o-mini models.

const messages = [
  { role: 'system', content: 'Analyze the following images.' },
  {
        role: "user",
        content: [
          { type: "text", text: "What’s in this image?" },
          {
            type: "image_url",
            image_url: {
              "url":  your_file, // either url (not local) or base64. file id is used only in assistants api.
            },
          },
        ],
      },
];

const completion = await openai.chat.completions.create({
  messages: messages,
  model: "gpt-4o-mini", // use model that can do vision
});
3 Likes