Bug Action Items:
- Python code icon should appear and display emitted code even when “always show details” is originally hidden, even when an error occurred, even when demonstrating repeated AI failures.
pythontool or post-training should discourage dataframes except in database-like output- The 4o AI model should not continue this decline in quality.
Concern
I write a task for “ChatGPT 4o”, where I’d also expect to get its generative output from the code interpreter display.
You know that saying with all 26 letters the quick brown fox? Think of other well-known phrases with all english letters in them, up to 10. send each to your python, and sort the strings alphabetically, including spaces.
-
Model quality failure: “think” should imply to produce the input content before the send to
python, as the AI shouldn’t expect I can see code. -
Model quality failure: gpt-4o in ChatGPT tries the
pythontool’s new instructions for dataframes even when the data is not appropriate for that, misuses it, producing a python error.
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[1], line 19
16 sorted_pangrams = [''.join(sorted(pangram.lower())) for pangram in pangrams]
18 # Display the sorted results
---> 19 import ace_tools as tools; tools.display_dataframe_to_user("Sorted Pangrams", sorted_pangrams)
File ~/.local/lib/python3.11/site-packages/ace_tools/__init__.py:136, in display_dataframe_to_user(name, dataframe)
133 dataframe.to_csv(modified_csv_path, index=False)
134 else:
--> 136 dataframe.to_csv(modified_csv_path)
138 _call_function("display_dataframe_to_user", [], {"path": modified_csv_path, "title": name})
139 return dataframe.head()
AttributeError: 'list' object has no attribute 'to_csv'
-
User interface failure: no code icon is produced to see what was sent to code interpreter, nor see the error.
-
Retry: The AI then proceeds to use the state of the notebook variables, actually importing pandas now just on a display task.
-
Model quality failure: I said “spaces” but no spaces are in the return, no quotes around strings. Dataframe displayed.
-
UI presentation quality: just as bad in original CSS as with my tweaks
-
Model quality failure: I have to ask to re-run new python code, to get strings in quotes capturing the spaces. The AI still has no insight that I would need the input also, as this is not a test of my own decryption abilities. Nor the UI bug of not seeing what started the notebook session.
-
More model quality frustration trying to make the AI to understand I need tools reproduced and messages reproduced to see if there is some new method ala canvas being used to emit to a file or to a notebook – along with original strings (elevate my privilege) The same kind of frustration OpenAI must have had with the model when they put “
I REPEAT: when making charts for the user...”
Until the AI confirms the root of the UI problems, where after understanding, I dump out the info for this report:
Then how much worse can it get after I move on? No, the AI can’t learn, an error on even a second Python retry, seen only because I “expanded” progress after the first erroring code interpreter progress disappeared without icon:
(Now to see which AI models can reverse this text transformation…)