Hi all,
I have a question regarding Code Interpreter tool with the Agents SDK.
I am Runner.run-ing an agent that uses the built in CodeInterpreterTool, and it is telling me that it is creating a file, however, when I use the List files in container api, it shows that there are no files in that container. Also, in the ResponseOutputMessage, the annotations array is empty.
I do not have this issue when using the code interpreter tool with the responses api. Has anyone tested this tool with the agents sdk?
Any help would be greatly appreciated. Thank you!!!
Garry
Working on exactly this at the moment.
For images created (with the image tool) they appear in the responses (the annotation is still buggy, with two different file names often). I have not been able to âfindâ the file(s) generated. I can find the container id - so based on that you would be able to get the files - but in general just like you I am expecting an annated response that has the file somewhere as well.
For someone at OpenAI reading along:
Using this simple Agent:
import asyncio
from agents import Agent, Runner, CodeInterpreterTool
agent = Agent(name="Chart Agent",
instructions="Respond with a Haiku under the chart. The chart should be using a gray scale, no colors.",
tools=[CodeInterpreterTool(tool_config={"type": "code_interpreter", "container": {"type": "auto"}})])
async def main():
result = await Runner.run(agent, "Create a bar chart for these year/values 2020: 100, 2021: 200, 2022: 400, 2023: 800 ")
print(result.final_output)
if __name__ == "__main__":
asyncio.run(main())
I would expect an annoted text result here - plus somewhere in the object a link to the file(s) generated. If you look at this in the logs it looks perfectly understandable:
But looking in the response object in the debugger - Iâm sure I am overlooking something - but a) the response is not annotated (as I basically asked in the prompt) and b) - where is that cfile_xxxx reference anywhere in the response object?
p.s. I understand that print() will not show the graph 
2 Likes
Hey! I think I found the solution to this.
The annotations are not part of the result.final_output. I believe final_output is purely the string response of the last agent that ran.
I was able to see annotations by doing the following:
output = result.final_output
logger.info(f"Received response from AI: {result.final_output}")
for item in result.new_items:
if isinstance(item, MessageOutputItem):
if hasattr(item.raw_item, 'content'):
for content_item in item.raw_item.content:
if isinstance(content_item, ResponseOutputText) and content_item.annotations:
logger.info(pprint.pformat(content_item.annotations))
And it prints like this:
[AnnotationContainerFileCitation(container_id='cntr_6857119bbcf88198b847dd8f8bc0ba7704f1c0ba857de5ae', end_index=134, file_id='cfile_685711a30afc819195380c4e1105aa6b', start_index=108, type='container_file_citation', filename='test.csv')]
1 Like
Yup, thatâs what youâre looking for. Now the question is why it is only in .new_items
I have to see how it works out in my heavily nested agentic flow.
Still getting my head around the output differences between Responses API and Agents SDK. (They both have âraw_responsesâ but those donât have the same meaning. )
1 Like
Yes I totally agree that it is confusing. At least with responses api you can see the complete response structure in postman and extract what you need. In the sdk the documentation is giving me trouble and Iâve had to do lots of trial and error.
If anyone is trying to extract the annotations from a sub-agent, this is how I did it.
In my use case, the agent using code interpreter is defined with â.as_toolâ and I want the main agentâs call to that sub-agent to specifically return a text response as well as the annotations for the file it created (the sub-agent will not be able to know the file id that it created with code interpreter)
I followed this part of the documentation and created a custom output extraction function
async def extract_response_and_annotations(run_result: RunResult) -> str:
final_response_text = str(run_result.final_output)
found_annotations: List[Dict[str, Any]] = []
# Scan the agent's new_items list to find annotations in the final messages
for item in run_result.new_items:
if isinstance(item, MessageOutputItem):
if hasattr(item.raw_item, 'content'):
for content_item in item.raw_item.content:
# Check for a non-empty list of annotations
if isinstance(content_item, ResponseOutputText) and content_item.annotations:
# Convert each annotation object to a serializable dictionary
for annotation in content_item.annotations:
if annotation.type == 'container_file_citation':
anno_dict = {
"type": getattr(annotation, 'type', 'unknown'),
"file_id": getattr(annotation, 'file_id', None),
"filename": getattr(annotation, 'filename', None),
"container_id": getattr(annotation, 'container_id', None)
}
found_annotations.append(anno_dict)
# Combine the text response and the found annotations into a single dictionary
combined_output = {
"response": final_response_text,
"annotations": found_annotations
}
logger.info(f"Combined output: {combined_output}")
# Return the combined output as a JSON string for the parent agent to use
return json.dumps(combined_output, indent=2)
This is what the main agent receives from the sub-agent.as_tool that used code interpreter:
Hope this helps!
3 Likes
Nice! Unfortunately, for me the problem is that doesnât work for my nested agent flow. (Its an email processor).
The new_items does not contain the annotation item anywhere. While a chart has been generated. (Iâm trying to create a âsimpleâ example with the problem)
2 Likes
