Error Sending Base64 Image Data in API Request to GPT-4-vision Model

I keep encountering a 400 Bad Request error with the message: “Invalid chat format. Content blocks are expected to be either text or image_url type.”

Here’s the summary of the issue:

  • API Endpoint: https://api.openai.com/v1/chat/completions
  • Model: gpt-4-vision-preview
  • Request Method: POST
  • Headers: Content-Type: application/json, Authorization: Bearer [API_KEY]
  • Payload Structure: Including both text and base64 encoded image data as specified in the documentation.
  • Error Message: “Invalid chat format. Content blocks are expected to be either text or image_url type.”

I’ve tried several payload structures, ensuring the ‘content’ field is an array of content objects, including the base64 encoded image string under various keys as per documentation (e.g., image, image_base64, data). Each attempt results in the aforementioned error. The documentation indicates that base64 images are supported, but the API response suggests otherwise.

Could anyone provide insight into the correct format for sending base64 images to the GPT-4 Vision API or point out what might be going wrong in my requests?

I appreciate any help or guidance on the issue. Thank you!

Below is the JSON structure of my latest attempt:

{
“model”: “gpt-4-vision-preview”,
“messages”: [
{
“role”: “system”,
“content”: “You are an assistant…”
},
{
“role”: “user”,
“content”: [
{“type”: “text”, “text”: “What’s in this image?”},
{“type”: “image”, “data”: “iVBORw0KGgoAAAANSUhEUgAA…”}
]
}
],
“max_tokens”: 3000
}

1 Like

You’re using the wrong schema for the image object, instead of

{
“type”: “image”, 
“data”: “iVBORw0KGgoAAAANSUhEUgAA…”
}

Use:

{
"type": "image_url", 
"image_url": 
{
      "url": "data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAA…"
    }
}

When I was implementing GPTV into my own project, I found this to be interesting.
On one hand, the naming structure is confusing. On the other, keeping the same schema regardless of the call maintains polymorphic behavior. Write one GPTV call function and it can easily handle either url or base64 input without any conditional logic.

I think that your naming structure is superior, personally, because it better represents the agnostic behavior of the route. But I’m not sure who to ping to get the conversation started ha.

2 Likes

Thank you @JDKratos for taking the time to respond. I appreciate your suggestion regarding the use of the "type": "image_url" field for sending base64-encoded images.

Following your advice, I updated my payload to include the base64-encoded image as an image_url like so:

"type": "image_url",
"image_url": {
    "url": "data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAA…"

However, I encountered the same error message as before, indicating that the content blocks are expected to be either text or image_url type:

API Response Error: { “error”: { “message”: “Invalid chat format. Content blocks are expected to be either text or image_url type.”, “type”: “invalid_request_error”, “param”: null, “code”: null } }

Based on this, it seems there might still be a misunderstanding or perhaps a discrepancy in the API’s expected format for images. The documentation suggested the ability to send base64 images, yet the API response doesn’t respond to what I am sending?.. i don’t know.

I’ll mess with it some more tomorrow. Thank you once again for your assistance!!

Are you sure that you actually updated type to image_url from image? Cause that seems to be saying that the type given to it is something other than “image_url”. I have a dart function that I have been using all day, I’m certain this is the way it works.

If that’s not it, ping me tomorrow, I should be around.

@JDKratos thanks again for responding, and your willingness to help out.

I am pretty sure I’ve done this as you suggested properly, but I could have messed something up (my apologies, I am learning as I go here).

Here is how i have the payload structured:

async function sendMessageToGPTWithImage(message, base64Image) {

let imageDataUrl = `data:image/jpeg;base64,${base64Image}`;

const data = {
    model: "gpt-4-vision-preview",
    messages: [
        systemMessage,
        {
            role: "user",
            content: [
                {
                    type: "text",
                    text: message
                },
                {
                    type: "image_url",
                    image_Url:{
                        URL: imageDataUrl
                    }
                }
            ]
        }
    ],
    max_tokens: 3000
};

Given that you’ve done this successfully using a similar approach in Dart, I’m wondering if there might be something I am missing?

I greatly appreciate your assistance. Cheers

@JDKratos I’m an idiot. I checked it 900 times before i realized i was using “image_Url” instead of “image_url” - it is working now. this solution worked:

async function sendMessageToGPTWithImage(message, base64Image) {
loadingSpinner.hidden = false;

const data = {
    model: "gpt-4-vision-preview",
    messages: [
        systemMessage,
        {
            role: "user",
            content: [
                {
                    type: "text",
                    text: message
                },
                {
                    type: "image_url",
                    image_url: imageDataUrl
                    
                }
            ]
        }
    ],
    max_tokens: 3000
};

Thanks again very much!!

As this topic has a selected solution, closing topic.