Code interpreter with streaming enabled ends the runs too early

[jhakulin]

Repro steps

• Create assistant (I was using gpt-4-turbo)
• Give csv file to code interpreter containing data that can be used to create charts
• Enable code interpreter
• Enable streaming
• Request assistant to create bar chart of the data in csv file

Result

Code interpreter starts to generate chart like output below, but ends the run too early, without providing the bar chart

user: please provide bar chart on METALS & MINING sector on operating profit and provide link to me?

assistant_v22: The dataset contains financial data for various companies listed under different sectors. The fields include the company's name, sector, revenue, operating expenses, operating profit, and more.

Now, I will filter the dataset for companies in the 'METALS & MINING' sector and create a bar chart showing their Operating Profit.

The output of code interpreter final on_tool_call_done is as follows:

2024-05-01 16:59:36,547 - INFO - on_tool_call_done - on_tool_call_done called, tool_call: CodeInterpreterToolCall(id=‘call_OBEo01SGUahHdVR8kysbW55F’, code_interpreter=CodeInterpreter(input=“import matplotlib.pyplot as plt\n\n# Filter data for ‘METALS & MINING’ sector\nmetals_mining_data = data[data[‘sector’] == ‘METALS & MINING’]\n\n# Convert ‘operating_profit’ to float after removing commas\nmetals_mining_data[‘operating_profit’] = metals_mining_data[‘operating_profit’].replace(‘,’, ‘’, regex=True).astype(float)\n\n# Create a bar chart\nplt.figure(figsize=(10, 6))\nplt.bar(metals_mining_data[‘name’], metals_mining_data[‘operating_profit’], color=‘teal’)\nplt.xlabel(‘Company Name’)\nplt.ylabel(‘Operating Profit (in million)’)\nplt.title(‘Operating Profit of Companies in METALS & MINING Sector’)\nplt.xticks(rotation=45, ha=‘right’)\nplt.tight_layout()\nplt.show()”, outputs=[CodeInterpreterOutputLogs(logs=“/tmp/ipykernel_12/2141688280.py:7: SettingWithCopyWarning: \nA value is trying to be set on a copy of a slice from a DataFrame.\nTry using .loc[row_indexer,col_indexer] = value instead\n\nSee the caveats in the documentation: Indexing and selecting data — pandas 2.2.2 documentation metals_mining_data[‘operating_profit’] = metals_mining_data[‘operating_profit’].replace(‘,’, ‘’, regex=True).astype(float)\n”, type=‘logs’, index=0)]), type=‘code_interpreter’, index=0)

2024-05-01 16:59:36,802 - INFO - on_end - on_end called, run_id: run_2sO3wgChvX8IAMIQ4cjwOceD, is_submit_tool_call: False

Additional information

thread_ukAj3CMwb4TCEJS99JzK5q7I
Repro rate: 100%

I wonder, is there a need to submit tool call outputs for code interpreter?

By looking the API constructs, I see submit tool outputs is only for functions.

[_j]

@jhakulin

import matplotlib.pyplot as plt

# Filter data for 'METALS & MINING' sector
metals_mining_data = data[data['sector'] == 'METALS & MINING']

# Convert 'operating_profit' to float after removing commas
metals_mining_data['operating_profit'] = metals_mining_data['operating_profit'].replace(',', '', regex=True).astype(float)

# Create a bar chart
plt.figure(figsize=(10, 6))
plt.bar(metals_mining_data['name'], metals_mining_data['operating_profit'], color='teal')
plt.xlabel('Company Name')
plt.ylabel('Operating Profit (in million)')
plt.title('Operating Profit of Companies in METALS & MINING Sector')
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()

The AI is trying to display a picture. It didn’t write code to save it or provide links.

Also, I dare you to find a “data” object input. Maybe that was already in the notebook from prior calls, but session and that data would expire after an hour.

It’s overtrained on being ChatGPT.

You’ll have to tell the AI it can’t .show() or .display() anything, and every code snippet emitted to near-stateless python notebook must include reading the source data again. It will have to produce and save .PNG files, and provide user download links with file id.

[jhakulin]

When not using streaming, the code interpreter always provides the answer at the end, something like

assistant_v22: The bar chart above displays the operating profit for companies in the METALS & MINING sector. You can download the chart using the link provided below:

Download the bar chart
output\Metals_Mining_Operating_Profit.png

So there is a difference in streaming and seems like processing ends before the final step, where concluding the results.

[_j]

The response should be the same on API, just with annotation links and file IDs in your stream in different parts of the response object data: that your code will have to handle (and you’ll have to make it robust enough to grab whatever assistants_output is produced).

I fire up a pre-created Python programming pal and paste Python for purvey. A bit of instruction tells the AI what I want and what it can’t do.

For download links either at temporary blob storage in Playground or your API file store.

Since you can’t repair the code interpreter specs or model tuning in Assistants (like were a seasoned solutions architect to create one for themselves using Chat Completions), you’ll have to make it understand by instructions alone – an informed user or unmistakable assistant instructions.

[enriqueag26]

A solution for me was to look at the runs every time I received a message was done event and looked for the last filled response.

Not ideal and requires checking for several cases but it worked