Attaching image directly to user message using Assistant API

[Plutes]

Checking in to see whether anything has changed at all over last several months.

Last I checked, it wasn’t possible to attach an image directly to a user message in Assistant API (say, using base64). My understanding is that while this is possible using completions API it isn’t possible via Assistants API.

At the time I checked (Fall-ish 2024) one would put the image file to file storage and then attach the file ID to a user message, or post the image to some public store, then provide that URL to the assistant, then delete the image from the store after the assistant responded.

The create message still seems to require that the image be referred to by file ID or URL.

All the code examples I find use the completions API. All web searches indicate that Assistants cannot do what I need just yet. If I don’t hear from anyone to the otherwise I’ll assume that this still isn’t possible via Assistants API.

[familymember]

I have the same experience: Assistant is misleading by allowing attaching files in user message.
In my case, I am attaching PDF with instruction to process the PDF, similar to how I did this in regular user’s interface chat.
The misleading result from Assistant - It ignores the attachments at User message.

[_j]

It is not misleading, it is just poorly documented, leading us on the forum to continue clarifying:

Upload image to assistant via API

The linked “vision” guide is not applicable for Assistants, where you: upload to file storage with purpose “vision”, receive ID create a user message with the file ID as part of the message content then manage and maintain the link from file to chat so your platform can clean up after itself after chat deletion or expiration.

Here’s how to construct a “vision” message for assistants’ thread with either file ID or a URL:

Upload image to assistant via API

Sending a message is easy when it is just text. The content field is more complex when it is “array”, which is required for images. Here is a new API reference for you, since the current is so useless, neither human nor AI can understand. Assistants Message Content You have two different ways of sending content: content - string - Just the text you send as user message, no additional parts. or content - array - An array (list) of content parts with a defined type, each can be of ty…

[Plutes]

Super helpful, thanks for the pointer.