Empty-answer bug in ChatGPT instead of running code (solved: ChatGPT confused between "tool use" and "Python code execution")

ChatGPT Bugs
, ,
1

EDIT: I found the solution (see below). I am leaving this post since it might be useful for other folks.

I have been observing this bug - ChatGPT giving an empty answer - for a while. This is happening inside a Custom GPT that uses code interpreter.

The GPT is an interactive fiction game with an instruction that tells it to run some Python code (input_parser) right after each User’s input. The code takes as input a dictionary and does stuff; details are irrelevant for now.

The problem is that most of the time the code doesn’t run - no call to code interpreter. Instead I get an empty answer (see example below; I have dozens of these), followed by a textual continuation. I had to address many issues when writing this GPT, but this one emerged recently.

image
image1922×761 70.9 KB

  • This is not an error of running the code (I do get those occasionally, and they are very different). Plus, I can get GPT to run code upon triggering of certain conditions. And if I explicitly ask the GPT to run input_parser, most of the time it does it correctly. The code itself is fine.
  • This is also not about the GPT forgetting to run the code. Forgetting happens all the time - trust me, I have been dealing with plenty of that - but: (1) for normal forgetting I don’t get that weird empty line; and (2) I know the GPT is not just forgetting this because sometimes it explicitly writes things like: “It seems I could not execute the input parser. I will proceed…”
  • For the record, in the rare case that I do not get this bug, then everything works fine, i.e. the GPT will remember to run input_parser first thing at every subsequent exchange with the user, and everything goes as planned.
  • This is not a temporary network thing. I get those, but again, they are very different (and they are connected to an explicit error that I can ask ChatGPT about). Instead, this seems to trigger a very robust behavior from ChatGPT.
  • If I ask ChatGPT to explain what happened or report the error message, it doesn’t answer with anything meaningful; just confabulates some hypothetical inability to execute input_parser or some “technical problems”, but there is no real error message.

This is driving me crazy… Note that I have plenty of other code that runs fine in various circumstances. It’s this input_parser thing which is somehow breaking the GPT.

I know my problem is fairly specific, but to try to understand this - has anyone observed the “empty answer” behavior? What was it linked to?

2

Same thing happening to me. Thought it was a me problem

3

Thanks, glad not to be alone…

Can you share some info about your case, to see if there is a pattern?

What’s driving me crazy is that I don’t generally get this problem when asking the GPT to call functions (even “by itself”, triggered by some conditions), just now with this one.

Edit: Actually, I found the solution, see below - perhaps see if it works for you too.

4

Alright, while going again through my instructions, I had a eureka moment.

What’s going on is a confusion in ChatGPT between generic function calling (also known as tools) and executing Python code. The latter is done via the python tool (for example, see the all_tools system prompt)

My instructions were telling ChatGPT to “call functions” and provided Python examples. Until now, in previous versions of this same GPT, ChatGPT was able to infer from the context that I actually meant “use the python tool to execute this Python function”.

However, for some reason, this parser_input Python function which I recently added was ambiguous enough that ChatGPT regularly mistook it as a tool. The empty answer was likely due to a failed attempt to internally call a non-existent parser_input tool (as opposed to calling the python tool to run a function in the Python environment).

What’s less clear is why ChatGPT fails silently (with the empty answer) as opposed to giving an error; that looks like a bug.

Anyhow, after realizing this I modified my instructions by adding a lot of “via python tool” whenever I specified to call a (Python) function. I have done a few tries now, and everything seems fixed and working correctly.

1 Like
5

You can look at finish reason. See if it is stop…or alternately content_filter.

6

Solved it, see above. The issue was with ChatGPT confusing tool use with Python function calling.