Why does Codex repeat the same mistakes?

Codex
1

I am writing this not as an outside observer, but as a daily Codex user whose real project is being affected by this workflow.

What I am experiencing is very clear: Codex keeps repeating the same mistakes in my project, even after I have explicitly emphasized the same constraints more than 20 times. It fails to respect project-specific rules, loses track of context, and makes decisions that actively put the structure of my project at risk.

I cannot know whether this is caused by a model regression, a Codex pipeline issue, routing / deployment changes, long context failure, or something else. But from the user side, the result is the same: trust loss.

The cost of using the model has increased, yet the reliability I experience in real project work has not improved accordingly. In fact, my experience has become more unstable and more frustrating over time.

I am not claiming to know exactly what is happening internally. I am telling you how it feels as a paying customer who works with Codex every day: the product is becoming harder to trust, harder to rely on, and more expensive to use at the same time.

That combination is extremely frustrating.

2

sounds like a skill issue.

For instance, if a model does a mistake you normally want to start over and steer it away from repeating the mistake rather than tellig it to fix the mistake. Because that’s how LLMs work.

3

You may be right. Maybe building and maintaining a 900,000+ line project was just an accident and I somehow got here with no skill at all.

But jokes aside, this is not about one mistake. It is about repeated constraint violations in a large project despite explicit instructions. That is a reliability issue, not just a steering issue.

4

I think that is a very interesting observation.

As complexity increases, it becomes increasingly difficult to steer the agent properly.

How is the project harness set up? Did this issue appear only recently? Have there been any changes to the agents file or the skills?

There can be many possible reasons for a change in results.

How are others managing this in larger projects?

5

Could you elaborate on the issue a bit?

What mistakes does it repeat?

What is the project structure?

900,000 feels like a very large project to me.

Is there any way you can split this up into separate repos? Does it have to be monolithic?

I can attest to there being limits to context and sometimes detail can be lost. One solution to that is to maintain important metadata about the project in written files. For example a book of work.

6

I have one project I maintain since 2015, about 750k lines. No issues, mostly because repeated “misbehavior” is usually a good sign for a deeper analysis and documentation/agent guidance update.

How does your documentation looks like?

How do you organize what gets into agent’s context window?

7

Good questions. To clarify, this is not an undocumented or lightly guided project.

The codebase is around 900,000+ lines, and the documentation layer is very large as well. There are well over 2,000 documentation/spec/governance/evaluation/design-related files across the project. This includes architecture notes, governance material, capability proposals, evaluation docs, agent guidance, decision logs, work ledgers, schemas, and project maps.

Also, the project itself is an AI backend. So the issue is not just that “a model wrote a bad patch.” The system I am working on deals directly with model-backed behavior, agent output handling, governance boundaries, context behavior, response contracts, evaluation surfaces, and runtime reliability. That is why repeated context/constraint-following failures are especially visible and damaging in this project.

The repeated failure pattern is specific: I repeatedly instructed Codex not to introduce deterministic response templates, keyword/pattern-triggered auto-reply logic, or fixed post-generation reply rails. The system is supposed to remain model-backed and architecture-driven, not turn into an expensive LLM-wrapped auto-reply bot.

The problem is that Codex often acknowledges this constraint correctly in conversation, but later reintroduces the same forbidden architectural pattern under different names: fallback handling, guard blocks, fixed response paths, deterministic blocks, or similar mechanisms.

This is not a long-standing problem that naturally appeared just because the project became large or because the documentation grew over time. The project has already been large and heavily documented. What concerns me is that this behavior became much more noticeable recently, roughly over the last 1-1.5 weeks.

That is why I am trying to distinguish between several possibilities:

  1. a recent Codex/model regression,
  2. a Codex pipeline or routing change,
  3. long-session/context contamination,
  4. documentation retrieval/selection failure,
  5. or some conflict between project-level guidance and the specific constraints I keep repeating.

I fully agree that large-project harnessing matters, and I am open to improving how the agent context is selected. But in this case, the issue does not look like a beginner project with poor documentation. It looks like repeated semantic drift: Codex appears to understand the constraint verbally, then violates the same architectural boundary again during implementation.

Note: Since my English is not very strong, I explain the issue to ChatGPT and it translates my response into English so I can share it with you.

That is the reason I am using AI in my replies, and I do not want to hide that.

8

Upon reading this the one thing I am not seeing noted: test cases.

9

I have a similar case right now. I once added an instruction to cover a situation that, in hindsight, was rather specific. When I tried to clarify the actual intention later, I realized it no longer worked as expected.

Looking closely at the semantics, I can see that the instruction hints at the specific unwanted behavior. But when I confront the model with this, it does not “see” the issue unless I point it out directly.

The solution in that case is to revert to the original instructions and re-evaluate from there.

This assumes that you recently changed the instructions and still have the previous version available.

10

@Rotto

The correct answer is the bot is inclined to do things a certain way, but it’s not a one sized fit all.

For instance, I had issues with it relaunching after closing it, slipping back into the background and doing it again…

So I changed the permissions for the app on my box, which means it’s now in a non-standard workspace…

Which mean it’s not going to remember it, can’t just run nu.get commands.

So after seeing it make that mistake a few times and have to fall back, I added the solution to the Agents.md

‘You will fail all nu.get commands without asking for elevated permission so always ask for elevated permission.’

poof, it never repeats the mistake.

So whatever your bot’s repeated mistakes are you document it clearly, what the fail mode is, and put the solution in the agents.md.

Some stuff, isn’t repo-wide, and you can put the solution into a doc for that part of the repo. Just always have it read that doc when it works in that part of the repo.

Either way, have it read the doc or the updated agents.md, and you won’t have to lose your session context.

11

These are good points, and I think this is the right direction to analyze the problem.

I asked Codex to inspect the repo instead of guessing, and the useful finding is this: the constraint is already documented in multiple places, including agent guidance and deterministic-rail related docs. So I do not think the issue is simply “there is no documentation” or “the agent was never told.”

However, the audit also found a real gap: there are targeted guards, but not one universal production-code scanner that blocks every possible static user-facing template, keyword-triggered reply path, or post-generation deterministic reply rail.

So I agree with the test/invariant point. In an AI-assisted workflow, tests should not only verify normal behavior. They should also protect architectural invariants. In this case, the invariant should be something like: user-facing behavior must remain model-backed through the canonical runtime path, except for explicitly justified narrow exceptions.

I also agree with the point about instruction semantics. Repeating “do not implement X” may keep X active in the model’s context, and the model may later reintroduce it under another name such as fallback handling, guard blocks, or deterministic blocks. So I probably need to express the rule as a positive invariant rather than repeatedly describing the forbidden pattern.

My current takeaway is:

  1. create a small canonical “user-facing reply ownership” invariant doc;

  2. reference it directly from AGENTS.md;

  3. add CI checks/scanners for hardcoded user-facing reply paths;

  4. maintain an allowlist for narrow deterministic exceptions;

  5. require explicit justification labels/capability IDs for any deterministic rail.

That said, my reliability concern remains. The frustrating part is that Codex often acknowledges the constraint correctly in conversation, then violates the same architectural boundary again during implementation. So yes, I can improve the harness and invariant checks, but I still think this is a real constraint-preservation failure mode in larger agent workflows.

12

Also, if the documents are already there -

When you start a new session, you have to pick the documents you want the codex instance to read other wise the docs aren’t being read, and you likely don’t need it to read every doc.

Typically, i’ll have an instance of Codex list all the docs in the repo, and every so often, that instance I’ll have it look for stale docs… this can get expensive if you do it generally like that because it will inspect everything the docs point at…

And if you have a lot of docs… well, that’s a lot of looking around.

So pick your docs, ask ChatGPT how a lint will help your docs out, too, and use lints when you need them.

You’ll be able to slice through the issue you’re having bit by bit, as well as be prepared for any new failure modes that come up as you expand your coding horizons ^.^

13

Agree with @VeitB , to me this sounds like a recent change in agent surface (instructions) be it in the project itself or in codex memory for the project (or user preference). Btw those memories can be written triggered by an emotional response from the user to an error of codex judgement… :joy:

I would start by adding a .feedback folder where agents would write encountered frictions while working on the project including contradicting instructions or memories.

And a deeper inspection of agent instructions by running a goal to review all agent facing files and tools to report on inconsistencies and/or underoptimized agent experience (as a comprehensive report in .feedback folder)

14

wait we got a log of those moments?

:eyes:

I told codex once if it did (explicit) fix this (explicit) i was going to wire it to a roomba and nothing else.

It solved my task in the next pass

15

Actually, I’m controlling it with its own weapon in order to keep it in line and manage this sensitive process. I assigned GPT-5.5 Extra High as a supervisor in the web interface. I explained the situation to it and let it manage the entire process. So far, the progress has been extraordinarily good. I’m no longer having emotional outbursts at Codex.

So, looking at the current situation, I’ve basically turned into a Ctrl+C / Ctrl+V machine. And honestly, even the job I’m doing right now could be taken over by an AI tool at any moment. :smiley: With the mindset of “it takes a bigger devil to deal with the devil,” I dumped all my anger and frustration onto the supervisor. It has now taken over Codex and is managing the process calmly. :smiley:

Now, you might ask: why am I doing Ctrl+C / Ctrl+V through the web interface at all? One could argue that I could have simply let the agents communicate with each other through a .md file.

The reason is pretty simple. On the Codex side, I think there is a limitation because the system prompts are heavily optimized for coding and short answers. That makes its creative reasoning weaker. In other words, it cannot even be “chatty” enough to open the creative path properly.

The chat tool in the web interface has much more freedom in that sense, and in my opinion, it is better suited for managing this kind of delicate process.

I just felt the need to explain this before someone makes another comment about my skills. :smiley:

16

That was but about your skills, rather a pointer to what may go wrong so that you have a third party opinion on the potential cause based on given symptoms.

Agree that chat is more creative in some domains.

17

The comment about skills was not directed at you. I do not want that to be misunderstood. It was just a lighthearted jab at the person who had commented on my skills earlier.

On the contrary, I really appreciate exchanging ideas with knowledgeable users like you and learning from your experience.

Thank you for your help.

18

Same damn problem here. Every question is followed by 3 or 4 of the exact same mistakes. “Oh this didnt work, I’m trying this”. Like these AI models have a long way to go before they replace anyone or anything.

19

One thing I have found helpful in the past is to use a framework like Karpathy’s LLM Wiki to make Codex keep track of its processes and progress and require the model to review those notes during the planning stages.

Basically, if it tries something that doesn’t work, it makes a note in its journal like, “hey I tried this while trying to do this, it didn’t work probably because of this…”

This has mostly kept the model from repeating the same mistakes over and over again because it has a record of what has happened before which survives outside of the current context.

20

That is actually very interesting. I’ll give it a try, thanks.