I’ve built an assistant using functions and retrieval tools using the gpt-4-1106-preview. It works fine but not for images. I know images in user messages isn’t supported yet but from what I see in the docs, the assistant should be able to include images:
A message created by an Assistant or a user. Messages can include text, images, and other files. Messages stored as a list on the Thread.
However, I get this when I try to get the assistant to reply with an image:
User: can you show me a picture of a cat?
Assistant: I’m sorry, but my current capabilities don’t include displaying or accessing images, including those of cats. My role is to assist you with text-based information and guidance. If there’s anything else you’d like help with, feel free to ask!
Does anyone have any insights on this?
Yes, maybe it’s me not reading the documentation properly, but I got the understanding that messages with role=“user” does not yet support images, but messages with role=“assistant” indeed does.
But anyway, if the case is that images aren’t yet supported despite role I don’t do anything wrong, and just have to wait for it to become available. Thanks!
1 Like
Here is a sample of using Assistants API with image included in the messages.
My function:
{
"name": "get_cat_picture_of_the_day",
"description": "Get picture of cat given a date",
"parameters": {
"type": "object",
"properties": {
"date": {
"type": "string",
"description": "The date for which the cat picture is requested. Format should be YYYY-MM-DD."
}
},
"required": [
"date"
]
}
}
Output:
{ status: 'success', message: 'Here is your cat picture for the day', image: { src: 'https://...', alt: 'cute cat' }}
1 Like
Thanks, but afaiu your example doesn’t include image content in the message. Your function will retrieve a cat image somehow (like via The cat randomizer | Random cat photo generator) and return it to the assistant. The assistant will include the the link in a text message which is then showed in the output.
I was trying to get the assistant to give me an image via the message.image object (after retrieving the image from image files I already had provided as files to the assistant).
Yes it works like that. So your use case is, you provided image files in the retrieval and you want to show it in the message, right? Not just part of the image but the image itself, or both case?
That’s right, thats pretty much the use case. A simple example would be that I upload two files, cat.jpg and dog.jpg. In the assistant instructions I can tell the assistant that cat.jpg contains a cat and dog.jpg contains a dog.
If I then ask the assistant to show me a cat I would like it to include the cat image in the message, that is the basic case (which I tried and failed on).
If (or when) assistants have better support for image processing I expect I do not need the explicit instructions and I would also expect (maybe 
) that I can ask the assistant to show me a picture of animals and it would combine the cat and the dog into a singe image for me.
Hi Andersson, from what I understand, png and jpeg files are not yet supported in the retrieval assistant API.
I’ve been trying to include image processing using the retrieval assistant too, but it’s not working.
Since you can call external APIs using the assistant, you probably could just call the Vision API and return what it sees.
