Assistant API - Code Interpreter Image File Creation (files.retrieve_content) Bug

API Bugs
1

When retrieving a file that was created by an Assistant via the Code Interpreter as a tool (Matplotlib) the retrieve_content request seems to return a string (cast_to=str).

The same file can be downloaded via the Playground but when using the API to write to a local file using files.retrieve_content it fails to create the file correctly.

To reproduce this error, try asking an assistant to create a graph for you. It will use the tool Code Interpreter and will create the file which will be returned in the thread messages. This file_id can then be used to view the file via the Playground. However, if you want to render this file in the application you are building, you can not.

I hope this is enough definition to correct what I believe is a bug as I have tried various encoding to no avail.

2 Likes
2

Having the same error here. Here is some code that I used to recreate this:

assistant = client.beta.assistants.create(
  name="Data visualizer",
  description="You are great at creating beautiful data visualizations. You analyze data present in .csv files, understand trends, and come up with data visualizations relevant to those trends. You also share a brief text summary of the trends observed.",
  model="gpt-4-1106-preview",
  tools=[{"type": "code_interpreter"},{"type": "retrieval"}]
)

thread = client.beta.threads.create()
client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content="Can you please give me a simple example of a scatter plot. Please save as a png"
)

run = client.beta.threads.runs.create(
  thread_id=thread.id,
  assistant_id=assistant.id
)

messages = client.beta.threads.messages.list(
  thread_id=thread.id
)

messages

SyncCursorPage[ThreadMessage](data=[ThreadMessage(id='...', assistant_id='...', content=[MessageContentImageFile(image_file=ImageFile(file_id='my_file_id')....

Then when retrieving the contents of the file and encoding

content = client.files.retrieve_content('my_file_id')
Image(bytes(content,'utf-8'))

Turns out to be a corrupted image. Like @davenorris , tried a whole bunch of methods and encodings.

2 Likes
3

Just adding here that the request to the assistant instructions to save as a PNG is not needed for this to happen.

1 Like
4

Okay, so @ andreas over on Discord was able to chime in and help us figure this out : Discord

Here is the solution:

# handle image file
api_response = client.files.with_raw_response.retrieve_content(r.image_file.file_id)

if api_response.status_code == 200:
  content = api_response.content
  with open('image.png', 'wb') as f:
    f.write(content)
  print('File downloaded successfully.')
4 Likes
5

Can confirm that this works as well.

BTW r in r.image_file.file_id is the model’s response object for those wondering. Once saved, you’ll be able to use Image from Pillow to display the file.

from PIL import Image

im = Image.open('image.png')
im.show()
6

Could someone have a github link or entire code for the image file creation ?
I’m getting this error BadRequestError: Error code: 400 - {'error': {'message': 'Not allowed to download files of purpose: assistants', 'type': 'invalid_request_error', 'param': None, 'code': None}}

1 Like
7

@elroy How IS r in r.image_file.file_id supposed to be defined in the code exactly?

1 Like
9

Here’s how we get to r

prompt = input()

message = openai_client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content=prompt,
    file_ids=[file.id]
)

run = openai_client.beta.threads.runs.create(
  thread_id=thread.id,
  assistant_id=assistant.id,
  instructions="Please be professional. The user has a premium account.",
)

r = messages.data[0]
2 Likes
10

Thank you, I now have a different error. Any clue on this?
AttributeError: 'ThreadMessage' object has no attribute 'image_file'

Here is my entire code:

from openai import OpenAI
client = OpenAI(api_key='sk-...')
file = client.files.create(
    file=open("balancesheet.pdf", "rb"),
    purpose='assistants'
)

assistant = client.beta.assistants.create(
  name="Financial Visualizer",
  description="You are great at creating beautiful data visualizations. You analyze data present in .csv files, understand trends, and come up with data visualizations relevant to those trends. You also share a brief text summary of the trends observed.",
  model="gpt-4-1106-preview",
  tools=[{"type": "code_interpreter"},{"type": "retrieval"}],
  file_ids=[file.id]
)

thread = client.beta.threads.create()
client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content="Can you please give me a simple example of a scatter plot"
)

run = client.beta.threads.runs.create(
  thread_id=thread.id,
  assistant_id=assistant.id
)

messages = client.beta.threads.messages.list(
  thread_id=thread.id
)

#messages


r = messages.data[0]

What the next argument I should pass after r ?

# handle image file
api_response = client.files.with_raw_response.retrieve_content(r.?.?)

if api_response.status_code == 200:
  content = api_response.content
  with open('image.png', 'wb') as f:
    f.write(content)
  print('File downloaded successfully.')

Here is my r output

ThreadMessage(id='msg_09C8CSuPEXQvP39kjj2wz7pp', assistant_id=None, content=[MessageContentText(text=Text(annotations=[], value='Can you please give me a simple example of a scatter plot'), type='text')], created_at=1700040507, file_ids=[], metadata={}, object='thread.message', role='user', run_id=None, thread_id='thread_Q1iPMjIT0iGC9uoSDoBVHBTj')

messages output

SyncCursorPage[ThreadMessage](data=[ThreadMessage(id='msg_09C8CSuPEXQvP39kjj2wz7pp', assistant_id=None, content=[MessageContentText(text=Text(annotations=[], value='Can you please give me a simple example of a scatter plot'), type='text')], created_at=1700040507, file_ids=[], metadata={}, object='thread.message', role='user', run_id=None, thread_id='thread_Q1iPMjIT0iGC9uoSDoBVHBTj')], object='list', first_id='msg_09C8CSuPEXQvP39kjj2wz7pp', last_id='msg_09C8CSuPEXQvP39kjj2wz7pp', has_more=False)
1 Like
11

You can essentially use an if statement to see whether the content_type is an image_file or text. You have a few options.

If you know you are going to be getting an image_file response you could use:

r.content[0].image_file.file_id

However, I will note that this has been fixed via the Python library as of v1.2.1.

The new

So you would effectively do:

client.files.content(r.content[0].image_file.file_id)
12

This post was flagged by the community and is temporarily hidden.

13

I strongly encourage you to take down that collab. You have exposed your API key. Do the following immediately:

Remove access to the collab.

Revoke access to the OpenAI API key here: OpenAI Platform

Hit me up on Discord and we can talk through your code if you would like: davidnorris

1 Like
14

Is there a way i can display the image without saving it?

15

Negative. You will need to save the file to your local machine or server and then display that as the source.

16

I was able to display images generated by the assistant without saving using the below function:

import requests
from io import BytesIO

def displayFile(file_id):
file_id = file_id
headers = {
‘Authorization’: f"Bearer {‘YOUR_API_KEY’}"
}
response = requests.get(f’https://api.openai.com/v1/files/{file_id}/content’, headers=headers)
img = Image.open(BytesIO(response.content))
display(img)

1 Like
17

I am curious as to why you would want to mimic an object when you have the object there to begin with. This particular way of doing it is memory intensive. Why not store the file and serve it? This seems like a lot of extra work if you were to want to retrieve the binary data again too.

Or was this just a response that displaying an image is possible without saving it? Many things are doable, whether you should do them or not is a different story lol.

18

Current solution for JS (in Deno) that works unlike openai.files.retrieveContent that at the moment returns broken file.

async function getFileContent(id: string) {
  const response = await fetch(`https://api.openai.com/v1/files/${id}/content`, {
    headers: {
      'Authorization': `Bearer ${OPENAI_API_KEY}`
    }
  });
  if (!response.ok) {
    throw new Error(`HTTP error! status: ${response.status}`);
  } else {
    const buffer = await response.arrayBuffer();
    return new Uint8Array(buffer);
  }
}