BUG: Responses API with structured output AND Code Interpreter does not provide annotations?

[jlvanhulst]

I am excited to have code interpreter in Responses, finally allowing ‘full’ migration from Assistants API. I am running into a problem that it seems that UNLESS the ouput is text there is no consistent way to retrieve the files code generator creates.

The easiest way to test/debug is this prompt “Create a chart that plots 1,2,4, 8 against equal x steps. Output an annotated message that has the the file name in it.” (Enable Code interpreter and output ‘text’) - > this works fine. You will see the file show up in as part of the code interpreter message and also in the text message that follows.

Now if you change the output to ‘json’ (And add the output directive) there is no file attached to either messages (not in the final output and not in the interpreter session message) (Prompt: Create a chart that plots 1,2,4, 8 against equal x stept. Output the file as chart.png and create a nice title in the output JSON attribute “title”)

I have retested this with several models - only 4o is able to do this correct most of the time. All other models, including 4.1 and o3 will fail. I have flagged a lot of playground outcomes as either ‘good’ or ‘bad’ based on the outcome - hope that helps. For now the solution is to simply choose text output, in which case it almost always works across models, with gpt-4o still creating the best response.

[Sina_Azizi]

I am having a similar issue. I am not able to find the output of the executed code regardless of how I ask for it. This is my code:

  stream=True,
  tools=[{
      "type": "code_interpreter",
      "container": {
          "type": "auto",
          "file_ids": ["file-TXT3RH5yycr7MAX2H8kLvq"]
      }
  }],
  input=[{
      "role": "user",
      "content": [{ "type": "input_text", "text": "Run python code that shows exactly how many columns this spreadsheet has and what the name for each column is. " }]
  }],
  reasoning={"effort": "medium", "summary": "auto"},
  text={"format": {"type": "text"}},

Do you know how to access the file content and the output of the executed code? @jlvanhulst

[jlvanhulst]

I go through all the output messages to pick out the container responses. Then with the container_id I do list files.

    self.output_files = []
        # collect the files from the code interpreter calls and the text response object if it exists
        for output in self.raw_response.output:
            if output.type == "message":
                content = output.content
                for item in content:
                    if item.type == "output_text":
                        self.text_response_object = item
            elif output.type == "code_interpreter_call":
                files = await client.containers.files.list(container_id=output.container_id)
                # get files
                for file in files.data:
                    self.output_files.append({"file_id": file.id, "container_id": output.container_id})

(I just copy pasted, the ‘self’ is my Prompt class) and then use
await client.containers.files.retrieve() to get the files.

https://platform.openai.com/docs/api-reference/container-files/listContainerFiles
https://platform.openai.com/docs/api-reference/container-files for the details

[Sina_Azizi]

Thank you for that.

There seems to be an easier way to do this by enabling

include=[“code_interpreter_call.outputs”],

[jlvanhulst]

Where would you add that?

[Sina_Azizi]

Checkout the issue I opened.

BUG: Code Interpreter in Responses API is broken

When using the Responses API with code interpreter tools in streaming mode, the actual code execution results (stdout/stderr) are not delivered through any streaming events. Expected Behavior: Code execution results should be streamed as they become available, similar to how code deltas are streamed via response.code_interpreter_call_code.delta events. Actual Behavior: Streaming mode: ResponseOutputMessage.outputs is None in response.output_item.done events

You simply add it to your request like:

  reasoning={"effort": "medium", "summary": "auto"},
  text={"format": {"type": "text"}},
  include=["code_interpreter_call.outputs"],
  max_output_tokens=32000

[jlvanhulst]

Interesting - looking in the source only these would work (for Responses API)

ResponseIncludable: TypeAlias = Literal[
    "file_search_call.results",
    "message.input_image.image_url",
    "computer_call_output.output.image_url",
    "reasoning.encrypted_content",
]

[shanth]

Thanks for the report! I see the problems, we’ll work on fixing the citations to be more reliable regardless of text / json modes.

[jlvanhulst]

Great! Make sure to add testing for running two consecutive code interpreter sessions - 4o does best but has yet to return more than one file ever (with two expected one from each session)

[fitzjalen]

Confirming this problem. Do you plan to release some fixes for that?

[shanth]

We’ve made the following fixes:

• Increased odds of output files included in annotations array.
  • In case you find that doesn’t happen with certain json schemas, please consider prompting to include the name of the file as a separate field.
• Ensure all the generated files get uploaded to /v1/containers path.
  • So even if the annotations are missing with some json schemas, you should be able to list the files and download them at the /v1/containers endpoint.

If you see behavior different from above, could you please share a repro!

[jlvanhulst]

Great to hear - will certainly play with this!

[fitzjalen]

import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')

from agents import Agent, Runner
from agents.model_settings import ModelSettings, Reasoning
from agents import CodeInterpreterTool

from coscience.abstract.response import ModelResponse

from pydantic import Field

from openai import AsyncOpenAI
from openai.types.responses import ResponseCodeInterpreterToolCall

client = AsyncOpenAI()

class AssistantResponse(ModelResponse):
    response: str = Field(..., description="Assistant response")

agent = Agent(
    name="Assistant",
    instructions=(
        "You are a helpful agent."
    ),
    model='gpt-5-mini',
    model_settings=ModelSettings(
        reasoning=Reasoning(
            effort="low",
        ),
    ),
    tools=[CodeInterpreterTool(
        tool_config={"type": "code_interpreter", "container": {"type": "auto"}}
    )],
    output_type=AssistantResponse,
)


resp = await Runner.run(agent, (
    "Create a tiny CSV file named 'hello.csv' with header 'name,score' and two rows:\n"
    "Alice,10\nBob,9\n"
    "Also create a tiny XLSX file named 'hello.xlsx' with the same content.\n"
    "Then create a plot of the data from the CSV file using matplotlib (include download link for the file)."
))

container_ids = [
    output.container_id
    for item in resp.raw_responses
    for output in item.output
    if isinstance(output, ResponseCodeInterpreterToolCall)
]

output_files = []
for container_id in container_ids:
    files = await client.containers.files.list(container_id=container_id)
    for file in files.data:
        output_files.append({
            "file_id": file.id,
            "container_id": container_id,
            "path": file.path,
            "source": file.source,
        })

print(resp.new_items)
[ReasoningItem(agent=Agent(name='Assistant', handoff_description=None, tools=[CodeInterpreterTool(tool_config={'type': 'code_interpreter', 'container': {'type': 'auto'}})], mcp_servers=[], mcp_config={}, instructions='You are a helpful agent.', prompt=None, handoffs=[], model='gpt-5-mini', model_settings=ModelSettings(temperature=None, top_p=None, frequency_penalty=None, presence_penalty=None, tool_choice=None, parallel_tool_calls=None, truncation=None, max_tokens=None, reasoning=Reasoning(effort='low', generate_summary=None, summary=None), verbosity=None, metadata=None, store=None, include_usage=None, response_include=None, top_logprobs=None, extra_query=None, extra_body=None, extra_headers=None, extra_args=None), input_guardrails=[], output_guardrails=[], output_type=<class '__main__.AssistantResponse'>, hooks=None, tool_use_behavior='run_llm_again', reset_tool_choice=True), raw_item=ResponseReasoningItem(id='rs_68b0b45192f08190ab54d4e82e72180e0cd29e5dd2e39013', summary=[], type='reasoning', content=None, encrypted_content=None, status=None), type='reasoning_item'), ToolCallItem(agent=Agent(name='Assistant', handoff_description=None, tools=[CodeInterpreterTool(tool_config={'type': 'code_interpreter', 'container': {'type': 'auto'}})], mcp_servers=[], mcp_config={}, instructions='You are a helpful agent.', prompt=None, handoffs=[], model='gpt-5-mini', model_settings=ModelSettings(temperature=None, top_p=None, frequency_penalty=None, presence_penalty=None, tool_choice=None, parallel_tool_calls=None, truncation=None, max_tokens=None, reasoning=Reasoning(effort='low', generate_summary=None, summary=None), verbosity=None, metadata=None, store=None, include_usage=None, response_include=None, top_logprobs=None, extra_query=None, extra_body=None, extra_headers=None, extra_args=None), input_guardrails=[], output_guardrails=[], output_type=<class '__main__.AssistantResponse'>, hooks=None, tool_use_behavior='run_llm_again', reset_tool_choice=True), raw_item=ResponseCodeInterpreterToolCall(id='ci_68b0b452dabc819087480e30cb547d470cd29e5dd2e39013', code='# Create CSV and XLSX files and plot using matplotlib\r\nimport pandas as pd\r\nimport matplotlib.pyplot as plt\r\n\r\n# Data\r\ndata = {"name": ["Alice", "Bob"], "score": [10, 9]}\r\ndf = pd.DataFrame(data)\r\n\r\n# File paths\r\ncsv_path = "/mnt/data/hello.csv"\r\nxlsx_path = "/mnt/data/hello.xlsx"\r\nplot_path = "/mnt/data/hello_plot.png"\r\n\r\n# Save CSV and XLSX\r\ndf.to_csv(csv_path, index=False)\r\ndf.to_excel(xlsx_path, index=False, engine=\'openpyxl\')\r\n\r\n# Create plot\r\nplt.figure(figsize=(4,3))\r\nplt.bar(df[\'name\'], df[\'score\'], color=[\'#4C72B0\', \'#55A868\'])\r\nplt.title(\'Scores by Name\')\r\nplt.ylabel(\'Score\')\r\nplt.ylim(0, max(df[\'score\']) + 1)\r\nfor i, v in enumerate(df[\'score\']):\r\n    plt.text(i, v + 0.05, str(v), ha=\'center\')\r\nplt.tight_layout()\r\nplt.savefig(plot_path, dpi=150)\r\nplt.show()\r\n\r\n# List created files\r\nimport os\r\nos.listdir(\'/mnt/data\')[:10]  # show files in the data directory\r\n\r\n# Output paths so assistant can provide download links\r\ncsv_path, xlsx_path, plot_path', container_id='cntr_68b0b45138208193ab27705c284819e3044aee3ff3d2b34b', outputs=None, status='completed', type='code_interpreter_call'), type='tool_call_item'), MessageOutputItem(agent=Agent(name='Assistant', handoff_description=None, tools=[CodeInterpreterTool(tool_config={'type': 'code_interpreter', 'container': {'type': 'auto'}})], mcp_servers=[], mcp_config={}, instructions='You are a helpful agent.', prompt=None, handoffs=[], model='gpt-5-mini', model_settings=ModelSettings(temperature=None, top_p=None, frequency_penalty=None, presence_penalty=None, tool_choice=None, parallel_tool_calls=None, truncation=None, max_tokens=None, reasoning=Reasoning(effort='low', generate_summary=None, summary=None), verbosity=None, metadata=None, store=None, include_usage=None, response_include=None, top_logprobs=None, extra_query=None, extra_body=None, extra_headers=None, extra_args=None), input_guardrails=[], output_guardrails=[], output_type=<class '__main__.AssistantResponse'>, hooks=None, tool_use_behavior='run_llm_again', reset_tool_choice=True), raw_item=ResponseOutputMessage(id='msg_68b0b45f1e50819092d43c08a39ab7700cd29e5dd2e39013', content=[ResponseOutputText(annotations=[], text='{"response":"Files created:\\n\\n- CSV: [Download hello.csv](sandbox:/mnt/data/hello.csv)\\n- XLSX: [Download hello.xlsx](sandbox:/mnt/data/hello.xlsx)\\n- Plot image: [Download hello_plot.png](sandbox:/mnt/data/hello_plot.png)\\n\\nThe CSV contains:\\nname,score\\nAlice,10\\nBob,9\\n\\nI also created and displayed a bar plot of the CSV data."}', type='output_text', logprobs=[])], role='assistant', status='completed', type='message'), type='message_output_item')]

print(output_files)

[{'file_id': 'cfile_68b0b46194cc8191bbdad7ef44a4c333', 'container_id': 'cntr_68b0b45138208193ab27705c284819e3044aee3ff3d2b34b', 'path': '/mnt/data/hello.csv [/mnt/data/hello_plot.png](https://file+.vscode-resource.vscode-cdn.net/mnt/data/hello_plot.png)', 'source': 'assistant'}, {'file_id': 'cfile_68b0b46194b881919aee409facc6c407', 'container_id': 'cntr_68b0b45138208193ab27705c284819e3044aee3ff3d2b34b', 'path': '/mnt/data/cfile_68b0b45ebfa88191a151996a6c052e0b.png [/mnt/data/hello.xlsx](https://file+.vscode-resource.vscode-cdn.net/mnt/data/hello.xlsx)', 'source': 'assistant'}, {'file_id': 'cfile_68b0b45ebfa88191a151996a6c052e0b', 'container_id': 'cntr_68b0b45138208193ab27705c284819e3044aee3ff3d2b34b', 'path': '/mnt/data/cfile_68b0b45ebfa88191a151996a6c052e0b.png', 'source': 'assistant'}]

The issue still exists: the annotation is empty in the final answer, and I still see this bug with the path.

files = await client.containers.files.list(container_id=container_id)

files

AsyncCursorPage[FileListResponse](data=[FileListResponse(id='cfile_68b0b46194cc8191bbdad7ef44a4c333', bytes=None, container_id='cntr_68b0b45138208193ab27705c284819e3044aee3ff3d2b34b', created_at=1756410977, object='container.file', path='/mnt/data/hello.csv [/mnt/data/hello_plot.png](https://file+.vscode-resource.vscode-cdn.net/mnt/data/hello_plot.png)', source='assistant'), FileListResponse(id='cfile_68b0b46194b881919aee409facc6c407', bytes=None, container_id='cntr_68b0b45138208193ab27705c284819e3044aee3ff3d2b34b', created_at=1756410977, object='container.file', path='/mnt/data/cfile_68b0b45ebfa88191a151996a6c052e0b.png [/mnt/data/hello.xlsx](https://file+.vscode-resource.vscode-cdn.net/mnt/data/hello.xlsx)', source='assistant'), FileListResponse(id='cfile_68b0b45ebfa88191a151996a6c052e0b', bytes=29754, container_id='cntr_68b0b45138208193ab27705c284819e3044aee3ff3d2b34b', created_at=1756410974, object='container.file', path='/mnt/data/cfile_68b0b45ebfa88191a151996a6c052e0b.png', source='assistant')], has_more=False, object='list', first_id='cfile_68b0b46194cc8191bbdad7ef44a4c333', last_id='cfile_68b0b45ebfa88191a151996a6c052e0b')

[Dus-DB]

Hello @shanth , now I also got this problem:

It was working fine, but now it starts to get this error in my app making the api call that obtains the file generated by Code Interpreter tool :
ERROR - Could not find a file citation annotation in the model’s response.

Then I checked at the api playground with some small request and also there is impossible to download the generated file. It does shows the link to download the file (like “Download the excel File”), but nothing happens when click that link.

[vladimir.blackmore]

I confirm that the annotation fields are empty. The bot says it created the file, but the annotations are empty. Please fix the issue.

[christian.peters]

+1, we are also experiencing this issue! The model is not saving any generated files to disc (or not annotating them). We are not able to retrieve any files via containers/files

[Martin_Mirakyan]

Looks like the model has stopped generating any annotations output over the last 2 days. Is there any workaround or a way to fix this?

[Sina_Azizi]

I am seeing a similar issue.

[Dus-DB]

Hello there.

OpenAI support is quite lazy and doing nothing to solve this issu of blank annotations in the response of Code Interpreter.

So I have this workaround:

1. Create a container and get its id.

   container_response = await client.containers.create(name=“my_container”)
container_id = container_response.id

2. If needed input files, upload them to that container, and get their ids.

   with open(file_path, “rb”) as f:
uploaded = await client.containers.files.create(container_id=container_id, file=f)

   uploaded_file_id = uploaded.id
   

3. Execute the request to the llm model (better if you tell the llm the desired name of output file to be assigned) . Example:

   response = await client.responses.create(
model=“gpt-4.1”,
input=[{“role”: “system”,
“content”: [{“type”: “input_text”, “text”: your_prompt}]
}],
tools=[{“type”: “code_interpreter”, “container”: container_id}],
tool_choice={“type”: “code_interpreter”},
temperature=0.1,
max_output_tokens=8000,
)

4. Obtain the generated / output file:

   1. List the container files (to check the one you want):

      files_list = await client.containers.files.list(container_id=container_id)

   2. Get the id of output file (generally the first in the list):

      output_file_id = files_list.data[0].id

   3. Retrieve the file content:

      output_file_content = await client.containers.files.content.retrieve(container_id=container_id,file_id=output_file_id )

   4. Save to local file:

      with open(output_file_path, “wb”) as f:
f.write(output_file_content.content)

5. Delete the container (if no longer needed):

   await client.containers.delete(container_id)

In this way I feel that i have more control of the items (container, uploaded files and generated files)