How to send base64 images to Assistant API?

[henning_cf]

One way to send images to the Chat API is via encoding it to base64 and creating a conversation turn as shown below, with the field {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}.

@henning_cf

image_bytes = io.BytesIO()
image.save(image_bytes, format=format)
base64_image = base64.b64encode(image_bytes.getvalue()).decode(
        "utf-8")

prompt_turns: list[dict[str, str]] = []
[...]
turn = {
        "role": "user",
        "content": [
            {"type": "text", "text": instruction},
            {
                "type": "image_url",
                "image_url": {
                    "url": f"data:image/jpeg;base64,{base64_image}",
                    "detail": "low"
                },
            },
        ],
prompt_turns.append(turn)

However, that doesn’t seem to work for the Assistant API.

Based on your previous comment (not the reference to the API docs), I would assume that file upload is the way for using images in the Assistant API like this

oai_file = oai_client.files.create(file=open(image_path, "rb"), purpose="vision")
turn = {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt.instruction},
                    {"type": "image_file", "image_file": {"file_id": oai_file.id}},
                ],
            }