Subject: Severe knowledge loss in long-running ChatGPT Projects despite extensive handover and recovery safeguards
I would like to report what I believe is a serious reliability issue when ChatGPT is used for long-running, technically complex project work.
This is not a complaint about ChatGPT forgetting one isolated detail. In my case, substantial amounts of previously established technical project knowledge disappeared from later consolidated working states, even though we had deliberately introduced extensive safeguards against exactly this problem.
Project context
I am using ChatGPT for a private engineering project involving the PLC/HMI control system and technical documentation of an expedition vehicle.
The project contains multiple specialist workstreams, including PLC programming, HMI structure, hydraulics, water treatment, humidification, heating, acoustic warnings and technical/user documentation.
Because preserving technical decisions is important, we established a structured architecture with:
- a central technical master chat responsible for technical truth, decisions and releases;
- separate specialist chats;
- explicit source hierarchies;
- status classifications such as DISCUSSED, AGREED, COMMISSIONED, OPEN/GATE, DO NOT CHANGE, IMPLEMENTED and TESTED;
- formal handover documents between chats;
- master checkpoints;
- delta registers;
- supersession tracking;
- explicit “not yet written back” states;
- uncompressed knowledge backups;
- forensic recovery registers;
- a rule that technical decisions must not exist only in chat history;
- and explicit instructions that summaries must not replace the underlying detailed knowledge.
Despite all of these precautions, significant knowledge was lost during successive chat transitions and consolidations.
Concrete example 1: a complete PLC function effectively disappeared
A previously agreed function for the hydraulic leveling supports had been documented in an earlier PLC handover:
- the operator normally stands outside the vehicle while retracting the supports;
- the final retracted position is visible on the HMI inside the vehicle but is difficult to verify from outside;
- when the support system is active and the “supports retracted” end position remains valid for 2 seconds, the external support LED lighting should switch OFF three times for 0.5 seconds;
- a 60-second lockout should then prevent repeated signaling;
- a new valid end-position event is required before the indication can occur again.
This was not merely an idea. It had already been included in an earlier consolidated PLC handover.
Nevertheless, the function later disappeared from the central operator/PLC review list.
It was only recovered by searching the original ChatGPT conversation export and reconstructing the original discussion.
Concrete example 2: the function remained, but most of its engineering meaning was lost
Another hydraulic optimization survived in a later summary only as something similar to an optional “automatic rear support adjustment.”
Forensic recovery of the original conversations revealed that the original knowledge was much richer:
- an internal hydraulic cross-effect in the manufactured hydraulic block caused the rear supports to extend approximately 2–3 cm during cab operation;
- this was explicitly not caused by an incorrect PLC command;
- the operator could later forget that the supports had moved;
- a software compensation sequence had therefore been discussed in detail;
- the historical trigger accumulated cab-lowering activity within a three-minute window;
- after the threshold, the PLC should wait 60 seconds and automatically retract both rear supports until the existing end condition or a maximum runtime;
- importantly, the PLC cannot actually detect that the cab is fully lowered;
- the historical timing was derived from the operating sequence that existed at that time.
Most of this rationale and context had disappeared through consolidation.
Recovering the original rationale later became essential because the operating sequence changed and the historical timing could no longer simply be copied into the current PLC specification.
The problem is therefore larger than simple “memory loss.”
We observed at least three different failure modes:
- Complete knowledge loss: an agreed function disappears from later working documents.
- Semantic compression loss: the function name survives, but its rationale, conditions, timing, exceptions or safety boundaries disappear.
- Status loss: the technical content survives, but its decision status changes or becomes ambiguous — for example, AGREED versus OPTION versus OPEN versus DO NOT CHANGE.
The second and third failure modes are particularly dangerous because the resulting document can still look complete and internally plausible. There is no obvious indication to the user that information has been lost.
Scale of the problem
After forensic recovery, the PLC/HMI task register became dramatically larger than the earlier consolidated operator review.
I cannot responsibly claim that more than 50% of all current PLC changes had completely disappeared, because some decisions were genuinely made later.
However, our current comparison strongly indicates that for roughly half or more of the engineering-relevant task set, the earlier central working state did not preserve all information required to understand the function, its rationale, its status, or its implementation boundaries.
This is a very significant failure rate for technical project work.
Another serious issue: ChatGPT cannot reliably warn before conversation capacity becomes a problem
We had explicitly instructed ChatGPT to monitor conversation length and initiate a controlled handover before reaching the limit.
Nevertheless, a major project chat reached the maximum conversation length before a safe transition occurred.
ChatGPT subsequently explained that there is no reliable user/model-visible indicator of remaining conversation capacity because context consumption depends on conversation history, project context, files and tool output.
This makes proactive knowledge preservation extremely difficult.
From a user’s perspective, there is a fundamental mismatch:
ChatGPT can participate in very large, sophisticated projects, but neither the user nor apparently the assistant has a reliable “remaining project/chat context” indicator that allows a safe handover before context pressure becomes critical.
Why I believe this is an important product issue
For casual conversations, losing context may only be inconvenient.
For engineering, research, legal, scientific or other long-running knowledge work, silent semantic loss can be much more serious.
The dangerous situation is not when ChatGPT says “I don’t know.”
The dangerous situation is when a later consolidated state appears complete and authoritative even though information that existed earlier has silently disappeared.
We invested substantial effort creating safeguards around ChatGPT — master registers, source classifications, handover documents, recovery procedures, forensic searches, delta tracking and explicit status management — and still could not reliably prevent this.
This was a relatively small private engineering project. That raises concerns about much larger professional projects where hundreds or thousands of decisions accumulate over months.
Suggested product improvements
I would strongly suggest considering:
- a visible and reliable context/capacity indicator for long-running chats;
- configurable warnings before a conversation approaches a critical context/length boundary;
- automatic creation of lossless project checkpoints before context compaction or conversation limits;
- persistent structured decision registers independent of conversational context;
- automatic provenance links from summaries back to the source messages/documents;
- detection when a new summary or handover omits previously classified AGREED/COMMISSIONED/DO-NOT-CHANGE information;
- semantic diffing between successive project checkpoints;
- warnings when an item changes status without an explicit superseding decision;
- separation between a concise working summary and a lossless underlying project knowledge layer;
- and a project-level integrity check that can answer: “Which previously established facts, decisions, requirements or unresolved gates are no longer represented in the current working state?”
In other words, compression should improve navigation, but it should never become the only storage layer for established project knowledge.
I still find ChatGPT extremely useful for this project. The reason I am reporting this is precisely because the system is capable enough that users may trust it with increasingly complex and long-running work.
But for that use case, knowledge continuity needs to become much more observable and verifiable. At present, users can spend enormous effort creating safeguards and still discover later that important information disappeared silently.
I hope this case is useful to the engineering/product teams because the failure is unusually well documented: we have the older consolidated states, the original conversation export, the forensic recovery results, and the later reconstructed task register, so the knowledge loss can actually be demonstrated rather than merely reported from memory.