i’ve had to tell ChatGPT 5.6 High thinking “don’t re-run the 20 minute unit tests when you have a tooling timeout of 45 seconds”
here’s the weird thing: sometimes it will notice that the tests time out, and it will break the test suite down further and further, keeping me informed every step of the way “i’m isolating the one last long test, now”, and other times as happened now it not only runs them, not only hangs but becomes unrecoverable without terminating and requires interruption, often losing critical state information and work, as in the background the timeout was long enough for fuse/cache backend filesystems to have the rug pulled out from underneath the container.
I started to write a manual reply but just kept running into open questions so I asked ChatGPT to help with the technical details and possibly create a new reply. This is ChatGPTs reply, which I have read and I do agree it could be of benefit in your case.
Note: I am not an OpenAI employee.
There is not much information here yet to determine whether this is a ChatGPT bug, a tooling/runtime limitation, or something specific to the way the tests are being invoked.
One thing I would clarify first is exactly where the tests are running. The report template says:
Surface: API / SDK / Playground / Platform
but your description refers to ChatGPT 5.6 High thinking and to a container/filesystem. Which ChatGPT feature or tool is actually executing the Python tests?
That distinction may be important.
Also, telling ChatGPT:
don’t re-run the 20 minute unit tests
can help, but only if ChatGPT can reliably identify which tests those are. If that information exists only from earlier observations in a long conversation, I would not rely on it as the sole safeguard.
I would make the constraint explicit and preferably machine-readable. For example, identify the long-running tests by name/path/marker and give the AI a command that runs only the short test set. Depending on your test framework, that might mean separating slow tests, marking them, or providing separate commands such as a quick validation suite and a full validation suite.
Something along the lines of:
During normal iterations, run only the fast test suite. Do not run <specific command/test names>, because it takes approximately 20 minutes. Only run the full suite when I explicitly request it.
I would also consider whether Codex is a better fit for this workflow than a normal ChatGPT conversation. The Codex desktop app, CLI, and IDE extension are designed to work directly with a code repository, read project files, execute tests, and follow repository-level instructions. OpenAI documents using an AGENTS.md file to give Codex persistent project instructions such as which test commands to run. The Codex desktop app is also intended for longer-running software-development work.
For example, the project instructions could explicitly state which tests are safe to run routinely and which must only be run on request.
However, I would not assume that moving to Codex by itself fixes a 20-minute command. If there is a hard execution timeout in the environment you are currently using, the better solution may be to run the long test suite outside that constrained tool invocation and have the AI inspect a saved result/log afterward.
A few additional details would make this much easier to investigate:
Which ChatGPT surface/tool is executing the tests?
What exact command starts the 20-minute suite?
Which Python test framework are you using?
Does the command reliably take about 20 minutes when run manually?
What is the exact timeout/error shown by ChatGPT?
After the timeout, is the test process actually terminated, or is it still running somewhere?
Can you reproduce the problem in a new chat with a minimal repository/test?
What exactly is meant by “unrecoverable” — does the UI stop responding, does the tool stop accepting commands, or is only the execution environment lost?
Can you provide a Feedback ID from a reproduction?
The last point about the filesystem would also be worth separating from the test-duration issue. If cancelling or timing out the command causes mounted/cache-backed filesystems to disappear and leaves the session unusable, that may be the more significant bug than ChatGPT deciding to run an expensive test in the first place.
OpenAI’s Codex documentation:
Codex desktop app:
Codex GitHub issue tracker, if this turns out to reproduce in Codex:
i don’t wish to give out personal information however that is not in any way practical. i am running 3 simultaneous sessions minimum for between 12 and 14 hours a day continuous. just the cost alone, aside from sitting at a desk continuously, would be immense.
you’re going to love this: i have AGENTS.md - C.A.T forgets to read them all the more i get deeper into these complex projects, the more my Software Engineering training kicks in, and realises that just asking an AI to read instructions is not enough: you have to write unit tests (or get it to write them) that demonstrate exactly when it hasn’t read the instructions. example: i use python 3.9 in a debian 11.5 container. it’s written an AST parser that explicitly checks for non-python-3.9 violations. it’s now a cross-project standard unit test
the symptom is back - it resolved cleanly (went away) for a few days and has just resurfaced. looks like the devteam tried about… three different server-side rollouts in the past… ten(?) days?
ah. we have a new symptom since last night: long-running threads at the UI limit cannot download files at all. they are directed to “access denied” as a JSON object.
so yeah, the backend devs are trying their best to return things to a known former state, but not quite succeeding… y’know… there’s this thing called “git” and it has “tags” and submodules?
chatgpt explicitly runs “pytest” or other unit tests on its own initiative
the entire suite is thousands, well over 25 minutes now.
no error: it hangs and has to be terminated so there is nothing left to investigate.
C.A.T has occasionally noted with “ps -aux” that the process is actually still running and has had to kill it
no because “minimal” means “too small and not running long enough to become a problem”.
UI is unresponsive, requires termination with the “square” button.
ah. i have a hunch.
devs: can you check that the container-recycling-detection timeout is shorter than the poll-interval by ChatGPT?
what i suspect is happening is that ChatGPT gets “so focussed” on waiting for a long-standing process that is never going to return, that you actually manage to pull both the filesystem(s) and in extreme cases the entire container out from under it.
i did mention previously that the tooling is single-threaded and that therefore if one socket is not properly polled with timeouts, one single faulty child program (Popen’d) can mess absolutely everything up.
look at fastcgi and nginx for how to handle this situation. known pattern.
--
ah darnit, additional datapoint: the limit on “tooling usage” is currently much lower. now, when the threads are not close to the UI chat stream limit that’s “fine”, but that becomes a serious problem later. i’ve presently got C.A.T doing regular git bundle checkpoints which it extracts from the container and chuck them at me over the UI stream, but sometimes it doesn’t get a chance to create them. bottom line this is adversely impacting working on the Riemann Hypothesis.
yeah this brhaviour past 2 days is very different. prior to 2 days ago ChatGPT would catch a timeout on long-running tests (sympy going ballistic on exponents of 10x10 matrices). now, the app retracts all “Thinking”, replaces it with “system is taking a little longer” which i now know is code for “it’s all about to go to hell in a handbasket”, and fortunately i caught this one and was able to stop the thread with the square button and get it to hand me a checkpoint bundle before getting it to carry on.
If pytest is still alive while the ChatGPT turn/UI has stopped making progress, then at least in that specimen child-process lifetime and ChatGPT-turn lifetime are not the same thing. I would keep four things separate:
the pytest process;
the container/filesystem lifetime;
the backend ChatGPT worker/turn;
the frontend stream/UI that is supposed to stay attached to that turn.
I have an independent long-running ordinary-Chat trace with the same general lesson, although it is a different workload: the frontend eventually lost delivery while the backend continued reporting the original turn as STREAMING, and native resume temporarily carried real events afterward. There was no final/cancel at the point the frontend considered the turn lost. So a UI that looks dead does not by itself prove the worker, container, or child process died.
That is why I would treat the container-recycling idea as a hypothesis for now, not the conclusion.
For the next reproduction, the highest-value evidence would be:
exact timestamp + timezone;
DevTools HAR with Preserve log covering the hang if practical;
PID + container hostname before the hang and immediately after;
whether pytest continues consuming CPU / exists after the UI stops progressing;
whether a reload/new tab sees the turn as still active, terminal, or recoverable;
whether the PID/hostname changes and local files/variables disappear, which would be evidence of a separate runtime reset.
If the process stays alive and the same container state survives, that weakens the theory that the filesystem/container was simply pulled out from underneath the turn. If PID/hostname/state all change, then you have a much cleaner runtime-reset specimen.
I also would not treat “system is taking a little longer” as a reliable coded warning that the turn is doomed. It is a useful UI symptom to timestamp, but the network/backend state underneath it is what matters.
The nasty failure class is exactly the one you are circling: work can still be happening somewhere while the control/delivery path that lets the user recover it has gone bad.
As you noted in the other topic, is Lean needed for the “Long-running unit tests” problem discussed in this topic?
You mentioned C.A.T. here. Is C.A.T. needed for this?
One of the reasons I am not giving more detailed feedback is that it seems you are making better progress by explaining this to us and answering our questions than we are by trying to give you direct help.
Consider adding this to a prompt when it seems appropriate—especially when you are unsure what else to try:
Are there any gaps I may be missing? Are there any pain points or potential issues I should consider?
hi Eric, yes i need this kind of prompting, really appreciated, my memory is quite fragmented now.
no Lean is not needed, i started using it 5ish days ago and this has been ongoing and persistent in one form or another for months.
no C.A.T is just my name for the assistant, it’s presently ChatGPT 5.6 “high thinking”
and yes birth cyanosis 56 years ago left behind a type of… hybrid-dyslexia-and-vagueness that has to be compensated for by people asking me clear unambiguous questions.
krasnik, good to see you run into this type of issue as well.
yes absolutely: my experience in SW-Eng of 31 years allows me to piece together OpenAI’s infrastructure. ChatGPT is baaically no different from any other “person” sitting behind an IT help desk, receiving messages from users and then logging in remotely into a Virtualised Container and doing stuff on that user’s behalf. absolutely nothing that Kubernetes, Docker, and RT tracker with a lot of tweaking couldn’t handle/replicate.
once you think of ChatGPT in terms of being an IT sysadmin / helpdesker then the weirdness of the backend anomalies and interactions start to make sense.
Guys you are running excessive tasks as a plus user in chat and you admit work/codex would be too expensive
Then why would anyone else upgrade then if they could use chat
Sure chat works but is not great and there are trade offs you can’t expect chat to replace work/codex and it’s only fair re fair usage limits and compute
Small caution on the infrastructure analogy: the evidence here lets us separate observable lifetimes, but it does not identify OpenAI’s internal implementation.
What we can support from the traces is narrower:
in this thread, pytest can remain alive while the ChatGPT UI/turn stops progressing;
PID/hostname/files/state can distinguish child-process survival from an actual runtime reset;
in my separate trace, frontend delivery was lost while the backend still reported the original turn as STREAMING.
Those observations are useful precisely because they are architecture-agnostic. Kubernetes/Docker/helpdesk-style orchestration is one possible mental model, but these traces do not prove that stack or mechanism.
I’d keep the public claim at the observable boundaries: process alive / runtime state survives / frontend delivery lost / backend turn state observed. Then future traces can confirm or kill specific architecture hypotheses instead of us baking them into the evidence.
it’s a fair point, that i am almost reluctant to point out relies on a linear relationship balancing the value of the service provided (which is, frankly, absolutely immense given that my productivity as a software engineer goes up at least two orders of magnitude thanks to what OpenAI provide), and availability of funds. in my specific case i am now critically reliant on another individual for even basic amenities.
translation: the fundamental asumption is that businesses can afford $200 a month.
yes - the ChatGPT runtime, running on a separate host and connected via sockets to the web frontend and the Virtualised Container backend, hard-crashes and gets replaced. you term that “runtime reset”.
it seamlessly takes over, uses the exact same socket(s) both to the Virtualised Container and to the web frontend: the conversation seems a little odd as if it is rediscovering the state (because its predecessor hard-crashed, taking criticsl context down with it), but people do not notice unless they are paying very close attention.
over that socket it performs “ps -aux” and other investigations, just like any IT sysadmin would do over ssh, to any remote server.
watch the “Thinking” logs very very closely, you’ll soon see what i mean.
That is still stronger than the evidence supports.
When I use runtime reset, I mean an observed state transition such as PID/hostname/files/live variables changing or disappearing. I do not mean that we have proved a particular OpenAI “ChatGPT runtime host” hard-crashed, that a replacement process took over, or that it reused the exact same sockets.
Those are architecture hypotheses. The visible Thinking text cannot establish transport topology by itself.
To promote that hypothesis into evidence, we would need native before/after identifiers that actually show the handoff, for example connection/stream identifiers, worker/turn identifiers, host/process identity, or captured network/log state demonstrating that the old runtime disappeared and a new one inherited the same backend/frontend bindings.
Until then the defensible claim is narrower: some state can survive a UI/turn failure; some genuine runtime resets change PID/hostname/state; and frontend delivery can fail while backend work remains observable.
That distinction matters because it lets the next trace falsify a mechanism instead of treating a plausible mental model as already established.
a fair point, one that i recognise much more keenly having worked continuously with ChatGPT for eight months straight, where it won’t let me make conflationary assertions
the more i get deeper into these complex projects, the more my Software Engineering training kicks in, and realises that just asking an AI to read instructions is not enough: you have to write unit tests (or get it to write them) that demonstrate exactly when it hasn’t read the instructions. example: i use python 3.9 in a debian 11.5 container. it’s written an AST parser that explicitly checks for non-python-3.9 violations. it’s now a cross-project standard unit test
