Out of a million plus API users, of those who use assistants and vision, I suspect the majority.
If you want an AI model that will work well in assistants, start with gpt-4-turbo-0125 or gpt-4-turbo-1106 for English only, and only after success should you try cheaper models.
Hereās the full expansion of the API reference showing how to send the contents of a message into a thread in an API request:
Create messages within threads
Endpoint
POST https://api.openai.com/v1/threads/{thread_id}/messages
Create a message.
Path parameters
- thread_id (string, Required): The ID of the thread to create a message for.
Request Body
-
role (string, Required): The role of the entity that is creating the message. Allowed values include:
user: Indicates the message is sent by an actual user and should be used in most cases to represent user-generated messages.assistant: Indicates the message is generated by the assistant. Use this value to insert messages from the assistant into the conversation.
-
content (string or array, Required): The content can be either a simple string or an array of content parts, where each part can be of the following types:
- Text content (string): The text contents of the message.
- Array of content parts (array): An array where each element can be:
- Text: Pure text elements.
- Image URL: References an image URL in the content of a message, which must be one of the supported types: jpeg, jpg, png, gif, webp.
- url (string, Required): The external URL of the image.
- detail (string, Optional, default: auto): Specifies the detail level of the image, options are
low, high, or auto.
- Image file: References an image file in the content of a message. This is used with files that have been uploaded to the API storage with āpurposeā of āvisionā.
- file_id (string, Required): The File ID of the image in the message content. Set
purpose="vision" when uploading the File if you need to later display the file content.
- detail (string, Optional, default: auto): Specifies the detail level of the image, options are
low, high, or auto.
-
attachments (array or null, Optional): A list of files attached to the message, which can be added to tools.
-
metadata (map, Optional): Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format. Keys can be a maximum of 64 characters long and values can be a maximum of 512 characters long.
Sending images, examples
Here are the extended usage examples showing how to include an image in a message using the OpenAI API, covering both scenarios where the image is referenced via a URL and when itās uploaded as a file. The examples will be provided in both Python and Node.js for completeness.
Including an Image via URL
Python Example
from openai import OpenAI
client = OpenAI()
thread_message = client.beta.threads.messages.create(
"thread_abc123",
role="user",
content=[
{
"type": "text",
"text": "Here's an image example using a URL."
},
{
"type": "image URL",
"url": "https://example.com/path_to_image.jpg",
"detail": "high" # optional, can be 'low', 'high', or 'auto'
}
]
)
print(thread_message)
Node.js Example
import OpenAI from "openai";
const openai = new OpenAI();
async function main() {
const threadMessages = await openai.beta.threads.messages.create(
"thread_abc123",
{
role: "user",
content: [
{
type: "text",
text: "Here's an image example using a URL."
},
{
type: "image URL",
url: "https://example.com/path_to_image.jpg",
detail: "high" // optional, can be 'low', 'high', or 'auto'
}
]
}
);
console.log(threadMessages);
}
main();
Including an Image via Uploaded File
Python Example
from openai import OpenAI
client = OpenAI()
thread_message = client.beta.threads.messages.create(
"thread_abc123",
role="user",
content=[
{
"type": "text",
"text": "Here's an image example using an uploaded file."
},
{
"type": "image file",
"file_id": "file_abc123",
"detail": "auto" # optional, can be 'low', 'high', or 'auto'
}
]
)
print(thread_message)
Node.js Example
import OpenAI from "openai";
const openai = new OpenAI();
async function main() {
const threadMessages = await openai.beta.threads.messages.create(
"thread_abc123",
{
role: "user",
content: [
{
type: "text",
text: "Here's an image example using an uploaded file."
},
{
type: "image file",
file_id: "file_abc123",
detail: "auto" // optional, can be 'low', 'high', or 'auto'
}
]
}
);
console.log(threadMessages);
}
main();
These examples cover both methods of including images in messages using the OpenAI API, both via URLs and uploaded files, with optional detail level specifications.
Uploaded files are not altered or re-encoded when using SDK libraries that do the work of sending from local file.
