Asisstant API for querying image

With Assistant API, document can be uploaded for RAG. How can I use image as in vision with Assistant API?

In ChatGPT Plus, I created a custom chat where documents and images can be uploaded, and can be queried later. How can that achieved using API?

You can do that by:

  1. uploading the image to the API file storage, with purpose “vision”
  2. including the file ID as a part of a user message you add to a thread.

The API reference for assistants → messages has the format for passing user messages when you expand the content section.

Thanks Jay. It works well for me.
This is the code in Nodejs - may be beneficial for others too.

const imageFileId = “file-xxxxxxxxxxxxxx”

const actualContent = [{ type: ‘text’, text: prompt }, { type: ‘image_file’, image_file : {file_id: imageFileId} } ]
const threadMessages = await openai.beta.threads.messages.create(
myThreadId,
{ role: “user”, content: actualContent },

);

1 Like