Issue with gpt-4-vision-preview API when passing base64 encoded image

API
, ,
1

Hi,

I’m trying to use the gpt-4 vision API and using the Completions API in a Next.js project but it throws an error that the content has no overload for accepting type and image_url. Could anyone please help what’s wrong with this and how to fix this?

I have the below call tp the openai API

const { data: chatCompletion, response: raw } = await openai.chat.completions.create({
      messages: [
        { role: 'system', content: "You are a latex code generator, directed to convert the image to latex code. You must process the image and return the content as latex code. Make sure to import any packages when you use a command. Return only the latex code." },
        { role: 'user', content: {
          type: "image_url", 
          image_url: 
            {
              "url": "data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAA…"
            }
          }
        },
      ],
      model: 'gpt-4-vision-preview',
    }).withResponse();

The issue seems to be with Node.js version of the API doesn’t accept complex JSON type input for content but I am not sure how to fix it. I would really appreciate any help.

Thanks!!

1 Like
2

Your problem is here:

{ role: ‘user’, content: {

When making a multi-modal request, the message content must be formatted as an array, not an object.

    {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What’s in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
            }
          }
        ]
1 Like
3

Hi and welcome to the Developer Forum!

I don’t think your messages object is correct.
I made this modifications following the documentation. (https://platform.openai.com/docs/guides/vision/quick-start)

const { data: chatCompletion, response: raw } = await openai.chat.completions.create({
      messages: [
        { role: 'system', content: "You are a latex code generator, directed to convert the image to latex code. You must process the image and return the content as latex code. Make sure to import any packages when you use a command. Return only the latex code." },
        { role: 'user', content: [
          {
              type: "image_url", 
              image_url: 
              {
                "url": "data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAA…"
              }
            }
          ]
        },
      ],
      model: 'gpt-4-vision-preview',
    }).withResponse();
1 Like
4

Thank you @danny-avila @HenriqueMelo!!

Yeah in fact the error was that. The content was malformed which was causing it to fail. The error goes away but the API does not work and throws a CORS error followed by an Internal server error.

I’m also using the Completions API with the gpt-4-1106-preview model for text completion use case.

Is the CORS error expected and leading to the internal server error? Any suggestions on how this could be handled, thanks!

1 Like
5

I could be mistaken, but I believe the CORS (Cross-Origin Resource Sharing) error might relate to Next.js. In my experience with Next.js, this error often occurs when attempting to make a request to an API hosted on a different domain, which hasn’t been configured in the ‘next.config.js’ file.

If you share the specific error I can try to help.

6

These are errors I see

Access to fetch at ‘https://api.openai.com/v1/chat/completions’ from origin ‘http://localhost:3000’ has been blocked by CORS policy: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. If an opaque response serves your needs, set the request’s mode to ‘no-cors’ to fetch the resource with CORS disabled.
core.mjs:326

POST https://api.openai.com/v1/chat/completions net::ERR_FAILED 500 (Internal Server Error)

I currently have the Open AI API key as the environment variable and nothing else. I am not sure how we add the CORS allow to the header for all requests in Next.js and would really appreciate any pointers.

7

You can find detailed instructions in How to Enable CORS on Vercel

Your ‘next.config.js’ should look something like this:

module.exports = {
  async headers() {
    return [
      {
        source: "/api/:path*",
        headers: [
          { key: "Access-Control-Allow-Credentials", value: "true" },
          { key: "Access-Control-Allow-Origin", value: "https://api.openai.com/v1/chat/completions" }, //  You can also use '*' instead of the specific endpoint but I am not sure if it's best practice 
          { key: "Access-Control-Allow-Methods", value: "GET,OPTIONS,PATCH,DELETE,POST,PUT" },
          { key: "Access-Control-Allow-Headers", value: "X-CSRF-Token, X-Requested-With, Accept, Accept-Version, Content-Length, Content-MD5, Content-Type, Date, X-Api-Version" },
        ]
      }
    ]
  }
};
8

Thank you so much @HenriqueMelo for helping me with this.

I tried adding these CORS configurations to the next.config.js and unfortunately, I still get the same CORS error followed by an internal server error with the below message.

{
    "error": {
        "message": "You didn't provide an API key. You need to provide your API key in an Authorization header using Bearer auth (i.e. Authorization: Bearer YOUR_KEY), or as the password field (with blank username) if you're accessing the API from your browser and are prompted for a username and password. You can obtain an API key from https://platform.openai.com/account/api-keys.",
        "type": "invalid_request_error",
        "param": null,
        "code": null
    }
}

I even tried to relax as you suggested and used “*” ad the allowed values but it didn’t work. It only throws this for the Vision preview model however the same key works for the regular gpt-4 model. I’m not sure if there’s a difference between the two.

9

It seems you might not be including the API Key in your request. Could you please verify this? Also, to clarify, are you using Next.js 13 and attempting to fetch data from the OpenAI API within the API folder of your project?

10

I am on the NextJs version 13.5.6. Also, I have been using the API key when creating an OpenAI object and setting the apiKey property. I am using the same object for the completion API with just text and that works seamlessly.

However, the same when done with the vision preview model and sending base64 image fails with the above error. I am not sure why would it throw API key missing error. Also, just to confirm, we have a common API key for different models, correct?

11

Yes, same API key. Can’t hurt to generate a new one that has all GPT-4 rights you get after making a qualifying payment.

You can go as far as simply changing the model name to standard gpt-4 preview and putting a normal user message in to verify your header for authorization: bearer sk-xxxx is working.

If the model was not available to you, you’d just have a “no such model” message returned.

1 Like
12 
import { OpenAI } from 'openai';

const openai = new OpenAI(
    {
        apiKey: process.env.OPEN_AI_KEY,
        dangerouslyAllowBrowser: true
    }
);

export default openai;

I’m using openai node module where I specify the API key and have verified that it works with the completions API when just using text. I have verified that I am passing the correct API key but still getting CORS error.

I have also tried to relax CORS restrictions but it still gives the same error.

13

@_j @HenriqueMelo Please let me know if there are any other pointers/things I could try to get this working. I’m still stuck on this and cannot understand why?

I tried changing the model to ‘gpt-4-1106-preview’ and the error went away. It is just with the vision preview model when I try to pass base64 image.

    { role: 'user', content: [
          {
              type: "image_url", 
              image_url: 
              {
                "url": imageContent
              }
           }
        ]
     },
14

I believe that clarifies the issue.

Could you provide more details on the structure of “imageContent”? Is it simply a base64 encoded string, or does it have a different format?

Itt should resemble something like the following structure:
"image_url": { "url": f"data:image/jpeg;base64,{base64_image}" }