I am wondering how ChatGPT generates a image and then can see it, describe it, reason about it. Even I asked it many way if I uploaded the image, suspecting the system behind the scene upload the generated image as from me (user). But it denied. It says that it generated the image via tool, but it can see it. Then how does it see the image? Is it possible to send a image via the role assistant or anything else than user?
Hello @kingkong-dev,
Welcome to the community.
When you want the model to see an image and reason about it (e.g., “What is in this picture?”), the image data (via a URL, Base64, or a file_id) is placed within the content array of a message with the role: "user".
The roles like assistant, system, and tool generally do not accept images or files as primary input for the model to analyze in the same way.
-
assistantrole: This role is for the model’s output. While an assistant can output an image generation request or a reference to a file it has generated/accessed, the input structure of anassistantmessage in the API is typically for text, tool calls, or the model’s final response content. You cannot use it to provide an image/file to the model for processing in the chat history. -
systemrole: This is for high-level instructions on how the model should behave. OpenAI has historically restricted image inputs to theuserrole, and attempts to use images insystemmessages have often been disallowed or ignored by the models.
In a conversational flow, the user is the one providing the initial input, which is where images and files belong for the model to process.
2 Likes
Then I want to know how ChatGPT reasons about it generated image. I am a new learner. Now I want to know how does it get it’s generated image. I was actually creating a chatbot just like it. I implemented a image generation tool call also. But I can’t figure out how to feed it back to context!
Please someone help. OpenAI officials are welcomed earlier.
@Innovatix Can you help please?