Unable to directly analyze or view the content of files like (local) images

[_j]

What you are attempting to send is significantly wrong.

You tag this “assistants”, but assistants does not support uploading of files in messages, even if encoded to base64.

Yet you make a requestBody with model and messages, giving hint that you are using chat.completions and not Assistants.

Besides not encoding an image properly and sending it as part of contents of a message, you are trying to put simply an image into a system message where it is not permitted, where instead you should be putting instructions that the AI CAN look at images.

This previous forum post has simple application code showing how to makes a system message for a task, and how a user provides the images, and how they must be encoded.

How can I get gpt-4o or gpt-4o-mini to analyze user images?

Hi all of you. The above doesn’t overcome any issues. If you get refusals, it may be because your “user” is deliberately asking about identifying people, or that your “system” message hasn’t overcome pretraining against this use. The AI also must be encouraged to have and use its own image computer vision. It is given to you in a state of “not knowing”. Having a structured output and a singular task makes it less likely to receive a refusal API return, which you should handle, along with oth…

Or how to create the user message alone

Wrong type on image_url in chat completation api documentation

There are multiple methods to pass images, and also methods exclusive within Assistants. Take for example this Python code, which gives alternating strings or base64 image (with size and capability under your control) as user message contents, without JSON-like “type”: image_paths = ["./img1.png", "./img2.png"] file_names = [os.path.basename(path) for path in image_paths] base64_images = [] for path in image_paths: with open(path, "rb") as image_file: base64_image = base64.b64enco…

Or how those messages would individually appear as content, with the user providing instructions

How to do few shot prompting with images in GPT-4 vision api structure? Can someone provide a code to do so?

It would be essentially the same as sending other few-shot examples: you give an input, and you demonstrate the way the AI responds, so that it can begin following a pattern. These latest models, such as the 1106 version of gpt-4-turbo that vision is based on, are highly-trained on chat responses, so previous input will show far less impact on behavior. After the system message (that still needs some more demonstration to the AI), you then pass example messages as if they were chat that occurr…

So the fault is not “gpt-4o API unable to directly analyze”, but rather “user unable to directly analyze API reference documentation.” Hopefully the links I provided serve as a better documentation.