Payload Structure: Including both text and base64 encoded image data as specified in the documentation.
Error Message: “Invalid chat format. Content blocks are expected to be either text or image_url type.”
I’ve tried several payload structures, ensuring the ‘content’ field is an array of content objects, including the base64 encoded image string under various keys as per documentation (e.g., image, image_base64, data). Each attempt results in the aforementioned error. The documentation indicates that base64 images are supported, but the API response suggests otherwise.
Could anyone provide insight into the correct format for sending base64 images to the GPT-4 Vision API or point out what might be going wrong in my requests?
I appreciate any help or guidance on the issue. Thank you!
Below is the JSON structure of my latest attempt:
{
“model”: “gpt-4-vision-preview”,
“messages”: [
{
“role”: “system”,
“content”: “You are an assistant…”
},
{
“role”: “user”,
“content”: [
{“type”: “text”, “text”: “What’s in this image?”},
{“type”: “image”, “data”: “iVBORw0KGgoAAAANSUhEUgAA…”}
]
}
],
“max_tokens”: 3000
}
When I was implementing GPTV into my own project, I found this to be interesting.
On one hand, the naming structure is confusing. On the other, keeping the same schema regardless of the call maintains polymorphic behavior. Write one GPTV call function and it can easily handle either url or base64 input without any conditional logic.
I think that your naming structure is superior, personally, because it better represents the agnostic behavior of the route. But I’m not sure who to ping to get the conversation started ha.
Thank you @JDKratos for taking the time to respond. I appreciate your suggestion regarding the use of the "type": "image_url" field for sending base64-encoded images.
Following your advice, I updated my payload to include the base64-encoded image as an image_url like so:
However, I encountered the same error message as before, indicating that the content blocks are expected to be either text or image_url type:
API Response Error: { “error”: { “message”: “Invalid chat format. Content blocks are expected to be either text or image_url type.”, “type”: “invalid_request_error”, “param”: null, “code”: null } }
Based on this, it seems there might still be a misunderstanding or perhaps a discrepancy in the API’s expected format for images. The documentation suggested the ability to send base64 images, yet the API response doesn’t respond to what I am sending?.. i don’t know.
I’ll mess with it some more tomorrow. Thank you once again for your assistance!!
Are you sure that you actually updated type to image_url from image? Cause that seems to be saying that the type given to it is something other than “image_url”. I have a dart function that I have been using all day, I’m certain this is the way it works.
If that’s not it, ping me tomorrow, I should be around.
@JDKratos I’m an idiot. I checked it 900 times before i realized i was using “image_Url” instead of “image_url” - it is working now. this solution worked:
async function sendMessageToGPTWithImage(message, base64Image) {
loadingSpinner.hidden = false;