You are right, apologies. Let me gather response samples / examples.
Since I am facing this same issue consistently, one site works and one site does not, I thought this might be a good example to solve what is going on. This seems to be out there for a long time.
I am using gpt-4o (not mini as in some threads) and the JSON seems to be almost the same. One works and one does not. I am curious whether the model is actually getting changed in the back end because of something in the request. I have reduced the instruction text and removed the images for readability.
(If I use this JSON and remove the Image_url in the user content, the JSON works fine. Of course the results are undesirable.)
Here is the JSON that consistently gets, “Image_url is only support by certain models”:
{
"model": "gpt-4o",
"messages": [
{
"role": "developer",
"content": "Please create a JSON object according to the included JSON Schema. Use an empty string for any data that is missing. You are an inventory manager. The user will be sending instructions and possibly some images to update their household inventory. The image might be of text, such as a receipt, or a picture of the items to address. If the user message is empty, assume they want to add items in the image to inventory. Always create a 'Summary' string element in paragraph form providing a short summary of the overall user request. Construct the text as if speaking to the user.If the person mentioned their own first name, Populate the 'FirstName' element with the person's first name in the text. If the person mentioned their own last name, Populate the 'LastName' element with the person's last name in the text. If the person mentioned their own email, Populate the 'Email' element with the person's email in the text. If the person has asked to add to their Inventory, For each occurance of an item in the user instructions or images uploaded, Add any item mentioned in the text and any items identified in the uploaded images to the Items_to_Add array. If the person has asked to remove items from their Inventory, For each occurance of an item in the user instructions or images uploaded, Add any item mentioned in the text and any items identified in the uploaded images to the Items_to_Remove array. Once the items have been identified to add or remove from the Inventory, update their current inventory and, For each occurance of an item in the user resulting inventory, Add the items to the Inventory_List array. If the person has asked about their Account, Populate the 'Account' element with a summary of their inquiry about their account. If the person has asked for help, Populate the 'Help' element with a summary of their inquiry about the help they need. If the person has asked about their Account usage, Populate the 'Usage' element with a summary of their inquiry about their account usage. If the person has asked about your Services, Populate the 'Services' element with a summary of their inquiry about your Services. If the person has asked about Seglamater, Populate the 'Seglamater' element with a summary of their inquiry about Seglamater. If the person has asked questions or requests that are not related to Seglamater, Help, Account, Usage,or Services, Populate the 'InquiryResponse' element with a short answer to each of their questions or requests in paragraph form. Construct the text as if speaking to the user."
},
{
"role": "assistant",
"content": "Here is a list of your current inventory: "
},
{
"role": "user",
"content": [
{
"type": "text",
"text": " Add "
},
{
"type": "image_url",
"url": "data:image/jpeg;base64,<Image>"
}
]
}
],
"max_completion_tokens": 5000,
"frequency_penalty": 0,
"presence_penalty": 0,
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "Irene",
"schema": {
"type": "object",
"properties": {
"Summary": {
"type": "string"
},
"FirstName": {
"type": "string"
},
"LastName": {
"type": "string"
},
"Email": {
"type": "string"
},
"Account": {
"type": "string"
},
"Help": {
"type": "string"
},
"Usage": {
"type": "string"
},
"Services": {
"type": "string"
},
"Seglamater": {
"type": "string"
},
"InquiryResponse": {
"type": "string"
},
"Items_To_Add": {
"type": "array",
"items": {
"type": "object",
"properties": {
"Title": {
"type": "string"
},
"Desctription": {
"type": "string"
}
},
"required": [
"Title",
"Desctription"
],
"additionalProperties": false
}
},
"Items_To_Remove": {
"type": "array",
"items": {
"type": "object",
"properties": {
"Title": {
"type": "string"
},
"Desctription": {
"type": "string"
}
},
"required": [
"Title",
"Desctription"
],
"additionalProperties": false
}
},
"Inventory_List": {
"type": "array",
"items": {
"type": "object",
"properties": {
"Title": {
"type": "string"
},
"Desctription": {
"type": "string"
}
},
"required": [
"Title",
"Desctription"
],
"additionalProperties": false
}
}
},
"required": [
"Summary",
"FirstName",
"LastName",
"Email",
"Account",
"Help",
"Usage",
"Services",
"Seglamater",
"InquiryResponse",
"Items_To_Add",
"Items_To_Remove",
"Inventory_List"
],
"additionalProperties": false
},
"strict": true
}
},
"temperature": 1
}
Here is the similar JSON that always works:
{
"messages": [
{
"role": "developer",
"content": "Please provide the response in a JSON string. \r\n We are going to construct a JSON string with contents of a Purchase Order from the included images. \r\nThe images are of a pdf converted to jpg pages. \r\n Be sure not to add any extra spaces in the JSON. Double check that the format is correct. "
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Please provide the response in a JSON string. Use the included jpg images. Please construct the JSON output. "
},
{
"type": "image_url",
"image_url": {
"url": "data:image/jpeg;base64,<Image>"
}
},
{
"type": "image_url",
"image_url": {
"url": "data:image/jpeg;base64,<Image>"
}
}
]
}
],
"model": "gpt-4o",
"max_completion_tokens": 1000,
"frequency_penalty": 0,
"presence_penalty": 0,
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "ProEnergyPO",
"schema": {
"type": "object",
"properties": {
"PO": {
"type": "string"
},
"PO_Date": {
"type": "string"
},
"PO_Lines": {
"type": "array",
"items": {
"type": "object",
"properties": {
"PO_Line_Item": {
"type": "string"
},
"Part_Number": {
"type": "string"
},
"Serial_Number": {
"type": [
"string",
"null"
]
},
"Work_Scope": {
"type": "string"
},
"Quantity": {
"type": "string"
},
"Unit_of_Measure": {
"type": "string"
},
"Unit_Price": {
"type": "number"
},
"Extended_Price": {
"type": "number"
},
"Part_Description": {
"type": "string"
},
"Instructions": {
"type": [
"string",
"null"
]
},
"Required_Date": {
"type": "string"
}
},
"required": [
"PO_Line_Item",
"Part_Number",
"Serial_Number",
"Work_Scope",
"Quantity",
"Unit_of_Measure",
"Unit_Price",
"Extended_Price",
"Part_Description",
"Instructions",
"Required_Date"
],
"additionalProperties": false
}
}
},
"required": [
"PO",
"PO_Date",
"PO_Lines"
],
"additionalProperties": false
},
"strict": true
}
},
"temperature": 1
}
I hope we can get to the bottom of this …
1 Like
This one was user error.
Incorrect snippet:
{
“type”: “image_url”,
“url”: “data:image/jpeg;base64,”
}
Correct snippet:
{
“type”: “image_url”,
“image_url”: {
“url”: “data:image/jpeg;base64,”
}
It is still an odd error for this misconstructed JSON.
Hi @platypus , I’m also wondering if a text only finetuned gpt-4o-mini-2024-07-18 model supports image_url?
I succeed to work well on the original gpt-4o-mini-2024-07-18 api. But with a text only finetuned gpt-4o-mini-2024-07-18 model api, the same input does not work anymore… (said bad request input)
But I cannot confirm such information (text only finetuned gpt-4o-mini does not support image_url) online so far. Does anyone succeed on this?
I just tried sending an image to a -mini fine-tuning. Received: Error: this model does not support vision.
This may be that the use of images against a fine-tuning that had no images gives no opportunity for filtering or predicting. You make a model that only says “hot” or “not hot”, it’s not going to have a fine-tune moderation error until you include the preceding pictures.
Also, the normal learning rates without modified hyperparameters quickly make an over-specialized AI.
1 Like