GPT-5.6 Sol vs Terra: what are you seeing in real development during these first days?

Codex Codex CLI
1

Hi everyone,

I have just completed my first serious real-world experiment with GPT-5.6 Sol and GPT-5.6 Terra, and I would be very interested to compare results with other developers.

Today I spent around eight hours working continuously in Codex on a real application. Most of the implementation was done with GPT-5.6 Terra using Medium reasoning.

This was not a spontaneous “build me an app” prompt.

Before starting, I had prepared:

  • detailed visual references;
  • exact screenshots of the expected interfaces;
  • seven Markdown files with specifications and implementation rules;
  • a structured existing codebase;
  • explicit acceptance criteria;
  • detailed instructions intended to reduce ambiguity, unnecessary exploration and repeated corrections.

Some of these materials took several days — and in certain areas several weeks — to prepare.

Despite that, my entire weekly Codex allowance was consumed in approximately six hours of active development.

That was unexpected, especially because Terra appears to be positioned as the more practical model for sustained everyday development.

The experience itself was mixed.

There were moments when the model handled complex implementation work impressively. At the same time, I encountered the kinds of problems that become extremely expensive during a long agentic session:

  • instructions being followed initially and then gradually forgotten;
  • existing visual references being interpreted too freely;
  • unnecessary changes outside the requested scope;
  • completed fixes being reported as successful even when the visible problem remained;
  • repeated repository analysis without a clearly visible reason;
  • large amounts of usage being consumed without detailed telemetry;
  • difficulty understanding whether the main cost came from reasoning, repository context, images, tool calls, repeated file reads or generated output.

I am not posting this simply to complain about the limit.

I am genuinely trying to understand how GPT-5.6 behaves in real production workflows, beyond short benchmarks and isolated coding tasks.

I would like to hear from developers who have already spent several days working with Sol or Terra:

  1. Which model are you currently using for serious repository work?
  2. Which reasoning level gives you the best balance between quality and usage?
  3. How quickly is your five-hour or weekly limit being consumed?
  4. Are you seeing context drift during long sessions?
  5. Does the model sometimes claim that a bug is fixed when the actual interface or behaviour has not changed?
  6. How well does it follow screenshots, design references and detailed specification files?
  7. Have you noticed unnecessary refactoring or changes outside the requested scope?
  8. Are Sol and Terra behaving significantly differently in real projects?
  9. What are the most common bugs or failure patterns you have encountered during these first days?
  10. What workflow changes have helped you reduce usage without sacrificing implementation quality?

I would also love to see more transparent task-level usage information from OpenAI.

For professional development, it would be extremely useful to know how much each task consumed in terms of:

  • input and cached tokens;
  • output and reasoning tokens;
  • repository context;
  • image processing;
  • tool calls;
  • repeated file reads;
  • subagent activity;
  • percentage of the five-hour and weekly allowances.

Right now, it is difficult to distinguish between an inherently expensive task, an inefficient workflow, unexpected model behaviour and a possible metering issue.

This was only my first full-day experiment, so I plan to continue testing both models and documenting what happens in real development rather than relying only on benchmark results.

I would be interested in turning this thread into a collection of practical observations from developers: what works, what breaks, how quickly limits are consumed, and which configurations currently provide the best real-world results.

OpenAI team: more detailed usage telemetry and visibility into agent behaviour would make these models much easier to evaluate, trust and use in production.

What have your first days with GPT-5.6 Sol and Terra looked like?

2

There is a lot of excitement right now around the Ralph Loop and similar autonomous coding workflows:

Define the goal.
Give the agent clear instructions.
Let it implement, test, inspect, fix, and repeat until the task is complete.

It sounds impressive. But today I ran into a very practical problem that makes me question how reliable this approach actually is in real product development.

Before starting, I prepared everything:

— detailed Markdown documentation;
— strict implementation constraints;
— visual references for every important screen;
— explicit instructions not to deviate from those references;
— clearly defined expected behavior;
— the latest GPT-5.6 model in High reasoning mode.

This was not a vague “build me an app” prompt. The context, requirements, references, and boundaries had already been prepared in advance.

And yet, during several hours of continuous work, I repeatedly encountered:

— incorrect interpretations of explicit requirements;
— deviations from provided visual references;
— changes to components that were not supposed to be touched;
— fixes that introduced new regressions;
— claims that an issue had been resolved when it was still clearly present;
— repeated rework of the same task despite detailed documentation.

This raises an important question:

What is the value of an autonomous loop if the model cannot reliably execute and verify a single constrained implementation step?

A loop does not automatically create reliability. It can also multiply incorrect assumptions, regressions, and wasted tokens.

The bottleneck is not always the absence of iteration. Sometimes the bottleneck is that the model does not accurately preserve constraints, compare the implementation against the reference, or verify whether its own fix actually worked.

I am not saying that agent loops are useless. I am saying that there seems to be a major gap between the current hype and the reality of using them on a complex, reference-driven production interface.

I would be very interested to hear from other developers testing GPT-5.6 and autonomous coding workflows:

Have you experienced the same thing?

What kinds of tasks are currently reliable enough for a Ralph-style loop?

How do you prevent the model from repeatedly “fixing” one issue while breaking another?

And most importantly: how do you make the agent verify the actual rendered result instead of simply reporting that the task is complete?

3

Not sure. I am pretty happy. Going for over a week now on Pro. Are you on Pro or the other one, whatever it was called?

To answer your questions:

1. Which model are you currently using for serious repository work?

5.6 Sol

Which reasoning level gives you the best balance between quality and usage?

Extra High, because Ultra just spawns a bazillion subagents and parallelizes work, most of my work requires active participation and occasional steering and getting it out of problems it cannot see for itself which happen occasionally and instead of wasting a lot of time and looping and deceiving itself because of its lack of access I am there to actively participate and resolve the issue and explain what is actually going on.

How quickly is your five-hour or weekly limit being consumed?

Five hour limit somehow disappeared but I never reached it.

Weekly limit has not been reached either in over a week (I think? When did it release again?). Coding for about 6-8 hours a day after work.

Are you seeing context drift during long sessions?

No, not to a noticable degree. I sometimes suspected a drift and let us clarify things, most of the time it came down to a simple misunderstanding than an actual drift. Not infrequently even a misunderstanding on my side.

Does the model sometimes claim that a bug is fixed when the actual interface or behaviour has not changed?

Rarely, in almost all cases it did so because it lacked the tools to see the actual bug or a new bug resulted which it lacked the tools for to see at that point after the supposed fix was applied. Most often it has to do with UX. For example a flickering effect, is near impossible to catch with the tools it has available on my setup.

How well does it follow screenshots, design references and detailed specification files?

It follows them, but it depends on what you mean by detailed and how much it is. Too detailed = no.

Detailed as in detailed way bringing across the vision, what you would like to see = works perfectly.

Giving AI, even if its not a diffusion based model, Specs that are ultra detailed and feel like a government issued norm to adhere to, is counterproductive to say the least.

From my experience at least.

Have you noticed unnecessary refactoring or changes outside the requested scope?

No, I have noticed necessary refactoring and changes outside of the requested scopes, including bugfixes. Which was very cool not gonna lie. I used Claude Code for a month before and it barely if ever did that, with Opus 4.8.

Are Sol and Terra behaving significantly differently in real projects?

What is a “real” project? I don’t have fake projects So I can’t answer this.

What are the most common bugs or failure patterns you have encountered during these first days?

AI getting stuck without output to see. About 2 times in that time. And also missing UX issues. The latter of which I expected.

What workflow changes have helped you reduce usage without sacrificing implementation quality?

Honestly none, but that’s because I can or rather I am willing to afford Pro for now and so far it’s been great. Haven’t hit a limit yet.

Generally, if I were having trouble I would play around with models, in fact, I would let Codex suggest me which model to use for the task I am asking to be completed. That I did in the past too, like 2 months ago where I had tried the 20 buck subscription only. Which allowed me to get quite a bit of work done with 20 bucks. I am not sure if the judgement was perfect but imma be honest, I was too lazy to figure it out myself.

4

Thanks, Tobias — this is exactly the kind of real-world comparison I was hoping to get.

I am currently on Plus, not Pro, so that may explain a significant part of the difference. Still, the contrast is quite dramatic: in my first full-day test, the entire weekly allowance disappeared after roughly six hours of active work, mostly with Terra on Medium reasoning.

The project is an existing production PWA with real users, authentication, payments, AI features, multilingual support, notifications, a database and a fairly large mobile interface. By “real project,” I simply meant sustained work inside an existing codebase with constraints, dependencies and users — as opposed to an isolated benchmark or a small test repository.

Your point about overly detailed specifications is interesting. In my case, the detailed documentation was not created because I enjoy writing bureaucratic specifications. It was created after repeated failures in previous development sessions.

For example, I had already encountered cases where an agent:

  • used reference screenshots as static images instead of rebuilding the interface;
  • changed unrelated parts of the application;
  • claimed a mobile UX issue was fixed when it was still clearly visible;
  • fixed one state while breaking another;
  • ignored explicit constraints that had already been documented;
  • reproduced the general visual idea but missed important interaction logic.

Because of that, I prepared seven Markdown files covering architecture, visual references, prohibited shortcuts, expected behaviour and acceptance criteria. The goal was actually to reduce ambiguity and token waste.

But I agree that there is probably a point where more documentation stops improving the result and starts increasing the amount of context the model has to process. I am now trying to understand where that threshold is.

Regarding screenshots and UX, my experience was noticeably less positive than yours. The model often understood the general direction, but small visual and behavioural details required repeated corrections. Mobile layout issues were especially difficult: menu jumping, overlapping content, incorrect fixed navigation behaviour and state indicators not updating after saving data.

I also had several situations where the model said that the problem had been fixed and verified, but after opening the application the original problem was still present. In some cases this was probably a tooling limitation, as you mentioned. In other cases, however, the implementation simply did not match the requested behaviour.

What makes your experience especially useful is that you are using Sol with Extra High and still not hitting the Pro limits after 6–8 hours per day. I expected Terra on Medium to be substantially more economical, but my first result was the opposite in practical terms.

A few follow-up questions, if you do not mind:

  • Approximately how large is the repository you are working with?
  • Do you regularly provide screenshots or visual references?
  • Does Sol have access to a running browser or any visual testing tools in your setup?
  • Do you usually continue in one long session, or start fresh tasks with reduced context?
  • How often does Extra High spawn subagents compared with Ultra?

I may test Sol with a higher reasoning level next, but on Plus I need to be careful: one badly looping task can consume a meaningful part of the weekly allowance before I have enough information to stop it.

Your reply also reinforces why clearer usage telemetry would be valuable. Right now I cannot tell whether the main cost came from the repository size, seven specification files, screenshots, repeated file reads, reasoning, failed implementation loops or something else.

Thanks again for sharing actual numbers rather than just saying that the model is “good” or “bad.” The difference between Plus and Pro usage may itself be one of the most important factors in this comparison.

5

First of all, unfortunately none of the providers I tried so far offered an experience on their 20 bucks type subscription, which they all more or less have, that can be used for real world development (as in large applications ofcourse, not saying small apps are not real projects).

You can one shot a marketing website or a small app that has limited complexity within those limits and even make it so that you actually like it.

For something like I have which goes beyond 100k lines of code by now it is completely unusable. You will run into limits even with the model swapping.

The one that took the longest in that was Antigravity, because it has I think 3 separate quotas and the Flash one is near impossible to exhausat at all, but Antigravity is also extremely stupid in comparison to both Claude and Codex, at the very least on the Flash model. From my experience.

Limited complexity projects with it you may be able to get done, but if it gets remotely complex you are going to create a mess with Antigravity + Flash.

As for Documentation and Specs, I would love to tell you where exactly that threshold is, I have absolutely no clue.

I can only tell you that neither at work nor with my private yet serious projects I have ever given AI any context beyond the repositories* itself and the task ahead. (*had to apply a little correction, because microservices exist, so I do give context beyond the current repository, sometimes dev db access and multiple other repos)

Other people have done so and I got rid of their instructions for a good part because it was for me, context waste, because the model in almost all cases deduced the correct patterns and things to adhere to by itself or by it’s harness or control structures around it (whatever it may be in the end). I know what I want to do and I inspect what has been done at work, when I don’t like it I tell it what I don’t like and that never made problems. *Well, little edit: Never may be a word too strong. Surely it has been a problem at some point in some context. But I don’t remember it so it probably wasn’t very upsetting.

For private projects, I am literally just having my linear board where I track issues and sometimes even brain dump stuff and let my agent organize it while maybe another agent is already working on something or just before I start going, or I type what I want straight into the chat, trying to bring across what’s up as well as possible.

One thing is for sure, the more massive your codebase, the more massive the application you want to build even if there is no massive codebase yet, the more likely you are going to experience drift, ignoring instructions, etc pp.

Because the context window is obviously still limited. When something is not in context, it’s not there.

So I don’t know about your exact case, but say for example you had a project with n lines of code, and a task completion (with accuracy aka not a broken result) requires the model to hold n tokens in its context. But the models context window is n+1 tokens, and you have specs or instructions one must adhere to exactly which would push it beyond n+1 tokens will have things that get ignored.

Either that, or the task execution breaks down. Therefore, I think ( because this “phenomenon” of ignoring specs and instructions happens with Claude and other providers too ) providers generally prioritize even absolute MUST HAVE instructions lower than what is required to have a usable end result.

Anyone feel free to correct me on that. But that’s how I would do it. Why would I risk completely butchering a task instead of ignoring a couple constraints instead?

About UX:
I think we misunderstood each other, my experience about UX is exactly what you described, but it’s been better with Sol than with 5.5.
I had this with any provider though and it is expected. None of the providers has models that have image understanding that naturally makes them catch User Experience problems like for example a mobile design being too small to actually be used comfortably. Or in general a UX that just has a “bad feeling” rather than outright bugs.
It also frequently fails at seeing things overlapping each other and whatnot, and declares a UX part done unless explicitly pointed out in a screenshot or at least hinted at by asking what is wrong in this example and posting the screenshot.

Your follow up questions:

Approximately how large is the repository you are working with?
Around 150k lines of code so far.

Do you regularly provide screenshots or visual references?
For UX work? Absolutely. Obviously especially for fixing UX problems which I don’t wanna solve myself.

Does Sol have access to a running browser or any visual testing tools in your setup?
Playwright + Computer Use + Chrome Plugin. All of them, chooses depending on use case sometimes needs reminder to use something for a certain task instead of working around a problem with another tool it chose which is just a waste of time when there is one available that can do it right away.

Do you usually continue in one long session, or start fresh tasks with reduced context?
Both, depends on what I am doing. I keep going until there is a clean cutoff, where it is just completely illogical and unnecessary to keep going in the same session.
With Codex I barely noticed degradation so far with 5.6. In Claude on the other hand it is EXTREMELY obvious, the degradation.
If you ever use claude, always always always keep sessions small. If you ever reach auto compact at roughly 1 Million token context, more than once, don’t go much further in that session is my experience with that. I think the degradation is also proven in studies.

How often does Extra High spawn subagents compared with Ultra?
I mean obviously I have no exact numbers but it does spawn quite a number, mostly for research tasks or tasks that can be safely parallelized. Ultra does spawn them always.
5.6 Sol Ultra plays the manager it seems like.

I think 5.6 Sol Ultra is probably a good thing to try out when I get back to my Linear board. Currently I am developing a library or rather a cross-platform plugin of sorts, which I will also use in the main project. For this I am still using one and the same chat eversince I started, 20k lines into it.

6

Thanks, Tobias — your explanation makes a lot of sense, especially the distinction between the $20 tier being suitable for smaller or limited-complexity projects and Pro becoming necessary once the application grows into something much larger.

My situation is slightly different because I have been deliberately comparing several systems in parallel rather than using only one provider.

Over the last few weeks, before GPT-5.6 appeared, I was working almost exclusively with the highest GPT-5.5 configuration available to me. At the same time, I also tested Fable 5 through OpenRouter, Antigravity, Google’s Pro models, and the latest Opus model.

In many cases, I gave them exactly the same task, based on the same repository, screenshots, requirements, and expected result. Even relatively small tasks turned out to be useful comparisons because they showed very different failure patterns.

One model might produce technically clean code but misunderstand the product intent. Another might understand the visual direction but damage existing behaviour. Another could complete the task quickly, but the result would require several rounds of corrections before it became usable.

So I am not comparing them only by benchmarks, response speed, or how impressive the first answer looks. I compare how much usable work I actually receive after the entire implementation and correction cycle.

Fortunately or unfortunately, GPT is still the system I keep returning to.

Compared with Gemini Pro, the latest Opus, Antigravity, and the other systems I tested, GPT-5.5 generally gave me the best combination of contextual understanding, implementation quality, and ability to connect an isolated task with the broader product.

I completely understand that my current subscription is not really designed for the kind of sustained development I am attempting. I am based in Ukraine, and at the moment I simply cannot pay for the highest subscription tier from every provider simultaneously. That is why I test them selectively and try to understand which system genuinely gives me the highest return on the money and time invested.

But what I managed to build with GPT-5.5 genuinely changed my understanding of what one developer can create with these tools.

I built a production PWA application around a concept for which I still have not found a direct equivalent in Google Play or the App Store. I was originally inspired by an existing application that was reportedly acquired for around $10 million and is said to be generating more than $5 million per month.

However, I did not simply reproduce that application.

I reconsidered the concept from the end user’s perspective, rebuilt the interaction model, created what I consider a significantly stronger interface, and connected functionality that the original product does not provide. The result is not just a clone with a different design. It has developed into a fundamentally different product.

And most of that was built with GPT-5.5.

That experience was honestly remarkable. GPT did not merely help me work somewhat faster. It allowed me to move the product several levels beyond what I initially believed one developer could realistically design and implement.

Now, using GPT-5.6 Sol and Terra, I am working on another version that is not simply a larger update. The entire concept is evolving in a direction that, as far as I can currently see, has not yet occurred to anyone else in this product category.

So despite the quota problem I described, and despite the fact that I clearly need to rethink how much documentation I preload into the context, GPT-5.6 currently remains the strongest result in my own practical comparisons.

Your point about context waste is especially useful here. My documentation was created as accumulated failure prevention, but I can see how seven specification files, screenshots, repository context, architectural constraints, and the immediate task may compete for the same limited context window.

For my next experiment, I will probably try something closer to your workflow: let the repository communicate more of its own conventions, provide only the task-specific context that is genuinely necessary, use a cleaner session boundary, and rely more heavily on browser-based verification.

I am also beginning to agree that Pro may not simply be “more usage.” For serious development, it may change the nature of the product completely — from something you can experiment with for a few hours into something you can actually use as a daily engineering environment.

At the moment, though, I have to test that conclusion from the Plus side.

Thanks again for sharing the details of your workflow. The fact that you are successfully working with approximately 150k lines of code, Sol Extra High, visual tooling, and long sessions gives me a much more useful reference point than any benchmark.

7

Yeah and keep in mind, specs and constraints and whatnot also mean relationships, plus moving parts, which need to be mapped.

So it’s not merely your whatever couple thousand tokens or more that you may have in specs that that needs.

I think what working with AI feels like, and how it seems to work best, is kinda this Spongebob meme, except less extreme (I don’t mean squidward throwing it away though, although I guess sometimes that happens too :rofl:):

8

For long sessions you really need to look at compacting and summarization every so often to reduce your burn rate. The extra context is usually noise anyway.

Also, instead of expecting a lot from a one-shot, go for iteration and focus on one problem/feature at a time, and incremental progress, all the time compactifying or starting a new prompt on each issue.

The large info up front may be too much as well, so solving little pieces, with specific info tends to provide better quality.

This works on any repo size, even repos with millions of lines of code, as long as the repo is easily structured for regex.

9

Tobias, that SpongeBob meme genuinely made me laugh. :grinning_face_with_smiling_eyes:

And yes, your point about specifications is important: they are not just a few thousand extra tokens. Every specification introduces relationships, dependencies, exceptions, priorities, and moving parts that the model also has to map against the repository and the current task.

That is probably where the real context pressure begins. The model does not simply need to remember the instruction itself. It needs to understand where it applies, what it affects, what must remain unchanged, and how it interacts with everything else already present in the application.

The funny thing is that while we are designing and building all of this, the process often feels completely logical. We think through the architecture, map the relationships, implement the features, click every button, check every calculation, and eventually reach that wonderful conclusion: everything works.

And then the testing phase begins.

Real users arrive, apparently from a completely different planet, press something nobody ever imagined could be pressed, interpret an interface in a way that seemed physically impossible, and suddenly reveal an entirely new dimension of the product.

That is usually where the most painful, but also the funniest, part of development begins.

Your meme immediately reminded me of an old video by Josh Darnit called “Exact Instructions Challenge.” He follows his children’s written instructions for making a peanut butter and jelly sandwich exactly as they wrote them.

It is basically developers, specifications, QA, and real users compressed into one video.

The forum does not seem to allow links from my new account yet, so here it is with spaces:

https : // www . youtube . com / watch?v=cDA3_5982h8

Just remove the spaces to open it.

I am sharing it purely because it made me laugh and because it feels painfully accurate, not as advertising. I hope it lifts your mood too.

This is not exactly the easiest time for Ukraine, so moments that genuinely make you laugh are especially valuable.

I am honestly very happy that I have found myself among people who immediately understand these situations without needing a long explanation. For a long time, I felt that almost nobody around me understood why these details, hidden assumptions, and strange user behaviours mattered so much.

Here, it feels like we are on the same wavelength, and that is incredibly refreshing.

In Ukrainian, we say “дякую” — it means “thank you.”

So, дякую, Tobias. I am really glad we started this conversation, and I would be happy to keep exchanging experiences, ideas, failures, discoveries, and developer memes.

10

Thanks, Curt — this is very useful, especially the point that the accumulated context is often mostly noise rather than useful project memory.

I should clarify that I was not really expecting a perfect one-shot result. The work was iterative, and I was actively reviewing the implementation, testing it, pointing out problems, and asking for corrections throughout the session.

The large amount of documentation appeared because of previous failures. Each file was intended to prevent a specific recurring problem: changing unrelated parts of the application, using screenshots as static assets instead of rebuilding the interface, missing interaction logic, declaring UX bugs fixed when they were still visible, and so on.

But I now see the contradiction more clearly: documentation created to prevent repeated mistakes can itself become context pressure, especially when the model also needs to map all the dependencies and relationships inside the repository.

Your proposed workflow makes sense:

  • isolate one problem or feature;
  • provide only the information required for that specific task;
  • verify the result incrementally;
  • compact when the session accumulates too much history;
  • start a fresh session when there is a clean task boundary.

That is probably the main change I need to test next.

One thing I would be interested to understand better: how do you personally decide when to compact versus when to start a completely new prompt?

Do you compact after a certain amount of work, when you notice repetition or degradation, or only when the current issue has reached a stable checkpoint?

Also, when starting a new prompt, do you usually provide a short summary of the previous work and current state, or do you rely almost entirely on the repository and the new task description?

My repository is structured, but I may also need to improve how easily the agent can locate features, routes, components, and related logic without repeatedly exploring large parts of the project.

I will try the next experiment with a much narrower scope and less upfront material. It will be interesting to compare both the implementation quality and the quota consumption against my first long session.

Thanks for the practical advice.

11

Have any of you noticed the same thing with visual verification?

I have started tracking where a large part of my time and usage goes, and one of the biggest sources appears to be the model reviewing the interface in the browser.

At first, this seemed like a major advantage. The agent launches the application, navigates through the screens, clicks the controls, checks different states, and then reports that everything looks good.

But when I open the deployed version on an actual phone, I often notice problems almost immediately: overlapping elements, layout jumps, incorrect spacing, unstable navigation, uncomfortable controls, or details that technically work but simply do not feel right in real use.

So the model may spend a significant amount of time, tokens, and compute inspecting multiple screens, only to produce a subjective conclusion that still needs to be checked manually.

For myself, I have currently concluded that this workflow is not worth it, at least for detailed mobile UX verification.

It seems faster and cheaper for me to inspect the deployed application myself, identify the exact problem, and then provide the model with a screenshot, screen recording, or a very specific description of what needs to be corrected.

The model is still useful for objective checks:

  • whether a route opens;
  • whether an element exists;
  • whether the expected state changes;
  • whether there are console errors;
  • whether an automated test passes.

But asking it to independently review the entire interface and decide whether everything looks and feels correct has so far consumed more resources than it has saved for me.

Have you noticed this too?

Do you let the agent perform full visual reviews, or have you also found it more efficient to keep the final UX inspection human and use the model only for targeted corrections?

12

Compact when you switch to a new problem, but are working adjacent problems. Start a new prompt if the problem has little relation to your context. You are basically trying to maximize “context SNR”. The higher the SNR, the better the result.

Also you need to play with the thinking effort. If you are familiar how to solve the problem, and you think it requires multiple decisions to explore, then go for higher thinking. If you are just doing routine maintenance stuff, go with a lower thinking level. You know you need to bump up the thinking if the bot is starting to make dumb mistakes, no matter what you think, but some people here may argue if the thinking is already high, then go low or switch models to get the project to shift correctly.

13

Thank you, Curt — I think you are absolutely right, and your explanation confirms something I had already started noticing in practice.

When a conversation becomes very long, the accumulated context eventually has to be compressed and summarized. That creates additional overhead, but more importantly, some details inevitably lose priority or become mixed with information that is no longer relevant to the current problem.

Since I am working inside one ongoing project, it often seems more efficient and reliable to start a fresh conversation for a new issue rather than continue carrying the entire history forward. The repository and project structure are still there, but the model can focus on the current task without all the accumulated conversational noise.

So your idea of maximizing context SNR gives a clear explanation for what I had already begun to suspect: compact when moving between closely related problems, but start a new prompt when the next task does not genuinely benefit from the previous discussion.

It is genuinely valuable to have someone confirm these observations and explain the principle behind them. When you work alone, you can spend a long time testing ideas inside your own workflow without knowing whether your conclusions are reasonable or whether you are simply adapting to your own mistakes.

That is why I appreciate this community so much. It is not always easy to find people with whom you can discuss these practical details seriously. Thank you again for the advice and for confirming the direction I was already beginning to explore.

14

That video is peak. I think I’ll use that as a reference at some point, whenever I have to explain why following some set of exact instructions in some context didn’t work out as planned. :rofl:

Yeah, laughing is healthy, especially in hard times. I like when people laugh, makes me happy too.

And about the part when real users come in:

That’s precisely why I think in these current times, where agentic development is big, but models and their setups are not quite the full replacement for workforce yet, QA is by FAR the most important thing. Nothing comes even close to it.

Not least since making your agentic tool use a Frontend side of things is not even remotely as fast as a human would do it, plus the native visual, spacial and “experience” understanding not being on human level.

One is better off to use AI to generate plans to do proper QA than offloading most QA work to AI. That’s what I think.

15

You are absolutely right, Tobias. Your point about using AI to design a strong QA plan rather than delegating the final QA judgment to AI is probably the most practical takeaway from this discussion.

That funny video also illustrates a serious product problem: something can be completely logical to the developer and still be interpreted differently by real users. I learned this from my first testers, who sometimes missed advantages that seemed obvious to me.

So I have adopted a simple rule: if a user can reasonably understand a feature, button, or instruction in two different ways, the interface is not finished yet. The label, placement, visual hierarchy, feedback, and resulting action must all communicate the same intention.

AI can help us generate the implementation and enumerate test cases, but human QA still has to answer the most important question: “Can a real person misunderstand this?”

Thank you — that distinction between AI-assisted QA planning and human product judgment is genuinely useful. I am adding it to my own development process.

16

Sometimes, after many attempts and results that are still far from what you expected, it is easy to lose motivation and wonder whether it is worth continuing.

But a few minutes ago, ballistic missiles were flying over my city. There were launches, interceptions, explosions, and consequences. I could attach a video of what happened, but that is not really the point, and I do not think anyone here needs another war video.

The important thing is that the missile missed. I am alive, my home is still standing, and I can still work. For me, that is not a reason to become pessimistic. It is a reason to try again.

Try, fail, learn, adjust, and try again. Then try again until it finally works. Because while we are still here, we still have an opportunity to create something useful and make the world at least a little better.

That is one of the principles many Ukrainians live by, and it has also become my principle as a developer.

What matters most to me in this thread is the moral support I have found in this community. I usually work alone, so having experienced people share honest observations, practical recommendations, and different perspectives is genuinely valuable.

Your advice helps me question my own workflow, experiment more intelligently, and grow as a professional. I take every thoughtful response here as an opportunity to become better at what I do and eventually contribute something valuable in return.

Thank you all for the support, the knowledge, and the conversation. I will keep learning, building, experimenting, and trying to make this world a little better — as a Ukrainian and as a developer.

17

By the way. Now that we have talked about this and I personally have no solution to the absolute top end of professional work in terms of things to track (Massive Project from scratch, especially contract work in B2B and involving lots of laws and norms and standards etc. Not simply feature dev, that for me works in any project size and even with our dozens of microservices at work), YT recommended this video to me https://www.youtube.com/watch?v=eLDHrqKplVI

It’s german unfortunately, but maybe you can translate it or just let AI transcribe it for you.

The question is does this guy have his methodology documented properly or is he just trying to sell his methodology and what is his harness, because it sounds like a custom or customized harness.

But he seems to have found a way and if his claims are true this is indeed kind of mindblowing, especially since he enforced traditional architectural solutions rather than something that is from my perspective tailored to the reality and strengths of agentic development and the future of development.

Eversince I’ve been using agentic dev I had the feeling that things like microservices are gonna die out in the near future. Because it is an architecture that makes development and maintenance in certain contexts for HUMANS much more managable, whereas for agentic development I considered it counterproductive personally. Because of the many many interconnections that a massive amount of microservices and possibly even microfrontends depending on what you do have and need to be understood and tracked to deliver a working product.

One thing he mentioned is that in terms of adhering to code quality and architecture and such things is he uses a bunch of heuristic and static tools I guess? Which the AI can use to verify what it’s built is fitting.

But yeah, I thought I share this video, thought it could also be helpful.

18

That is especially interesting, Tobias, because this is almost exactly the direction I have been exploring today.

I have already seen several developers on YouTube using their own agentic systems to coordinate multiple agents, distribute tasks, track progress, and verify results. Today I even finished a master prompt and prepared the initial structure for building a similar system through Codex.

Before starting from scratch, though, I decided to check GitHub first. I found several promising projects that already cover much of this problem: agent orchestration, task management, monitoring, context handling, and automated verification. Some of them have strong community interest, active development, and frequent updates, so it would be irrational not to study them before reinventing everything myself.

That is why your mention of a custom harness immediately caught my attention. The real breakthrough may not be a single model or methodology, but the entire engineering layer around the model: how agents are coordinated, what context they receive, how their work is inspected, and which static or heuristic checks prevent architectural drift.

I will definitely study the video and try to determine what is actually documented, what can be reproduced, and what may be hidden inside his custom tooling.

And after our conversation yesterday, I also made the decision to move to Pro as soon as possible. I increasingly see that for serious agentic development, Pro is not simply a larger quota — it gives enough room to properly experiment with the workflow, tooling, and orchestration around the models.

The timing of your video is almost suspiciously perfect. Yesterday we discussed context and scale, today I started designing an orchestration system, and now you have sent an example of someone who may already have solved part of that problem. Thank you for sharing it.

19

I have been thinking about how developers choose reasoning effort in Codex.

Low, Medium, High, and Max are often treated as intelligence levels:

More effort = smarter model = better result.

But that is not necessarily what happens.

A higher reasoning effort mainly gives the model more room to plan, explore alternatives, use tools, reconsider decisions, and check its work. That can be extremely useful for complex debugging, architecture, unfamiliar repositories, and long-running autonomous tasks.

But for a well-defined implementation task, more reasoning can also mean:

  • more tokens;

  • more execution time;

  • unnecessary exploration;

  • overengineered solutions;

  • changes outside the requested scope;

  • a result that is only marginally better—or sometimes worse.

So I am starting to think about reasoning effort as a budget, not a quality setting.

Here is a simple way to find the right level for your own workflow:

  1. Choose one representative task you regularly perform.

  2. Prepare one complete prompt with identical context, tools, constraints, and acceptance criteria.

  3. Run it separately on Low, Medium, and High.

  4. Compare the results using the same criteria:

    • Did it complete the task correctly?

    • Did it preserve existing functionality?

    • Did it follow the requested scope?

    • Did it run the necessary tests?

    • Did it introduce unnecessary complexity?

    • How much time and token budget did it consume?

  5. Use the lowest effort level that produces a reliable result.

  6. Escalate only when you can identify a specific failure that additional reasoning could solve.

For UI work, you can compare visual accuracy and unintended changes.

For debugging, compare whether the model found the root cause or only patched the symptom.

For architecture, compare the quality of trade-offs, migration risks, and failure scenarios.

For autonomous tasks, compare how well it validates each milestone before continuing.

The important part is to test effort levels on the same task. Otherwise, we are comparing impressions instead of evidence.

A precise specification on Medium may be more effective than a vague request on Max. And in many cases, better tools, repository instructions, tests, screenshots, and acceptance criteria may improve the outcome more than simply increasing the reasoning budget.

Which reasoning mode do you normally use in Codex: Low, Medium, High, or Max?

Have you actually compared the same task across several effort levels—and did the higher mode produce a meaningfully better result?

20

Oooo, fun discussions!

Let’s start with this question before I poke at your bullet points:

I think the term that’s been thrown around now is known as a harness. Since it’s been a minute since I’ve checked in with other devs, I can only speak for myself. But I have personally leveraged my programming expertise to build custom harnesses that address these specific problems, which include many of the phenomena you’ve touched on. In truth, now that I’ve been at this a few years, it feels more like an ever-evolving process that grows along with you and the models you use.

I still use Openai’s newest sets they release, along with Gemini’s models as well as a few local models (ironically enough I run Nvidia’s newest Nemotron model (120B A12B) on AMD hardware. AMD’s strix halo chips are incredible if you want to run decently sized models for something actually affordable.) And I’m working on trying out even some of Anthropic’s models now too. I want to use Gemma 4, but I think there may still need to be some tweaks upstream iirc before I can run those locally because I use vllm, and getting it to run well on strix halo chips can be…time consuming. And also just forcing one to wait for patches.

The question isn’t which option is best now, but instead how do you effectively communicate and write in a way where models and agents can effectively delegate long-horizon tasks.

I don’t really hit my limits often, but that’s because i subsidize my tasks with local models. Might be a genuine investment depending on how big your usage bills are. A lot of tasks nowadays can be handled perfectly by local models. This was a fever dream not that long ago too.

If I use a plain jane IDE like codex or antigravity, yes to both. However, this is why I invest most of my development time in harness engineering (I guess we’re calling it that now?) and iterating/developing my long term memory and RAG system. I have found early on, even when I was more active on here, that these problems aren’t going to go away unless you have a good system that manages context and memory. Manipulating text, and manipulating relevant text in a way that prevents context collapse is a fun challenge. Everyone has a different system for managing it, for both OSS and bigger companies. If I were to give folks advice today, I’d have them study the different fundamental approaches that have existed and been implemented. This should also allow you to manage tokens better, and thus your usage limits too btw.

I seldom ask models and agents for visual analysis still. Probably an old habit at this point, but I’ve never found any multimodal model fast or reliable enough for me to use that kind of modality very often.
Then again, my domain in AI continues to center around and leverage my knowledge and skills in language and linguistics. So no, my models typically refactor within scope, likely because of how I prompt them. And if they change something interesting or solve a problem in a way I wouldn’t, at this point I take a second to analyze and interpret what it did, because it’s about a 70% chance it was capable enough to think of something I didn’t think of (and I go back to learn from), and a 30% change it was being dumb because of an excessively long conversation thread.

I have not tried them yet! curious to know any capability improvements myself!

The last question ties into my earlier statements with harnesses and memory/rag system developments. It’s the system orchestration that maintains quality without sacrificing capability or my wallet lol.

I hope this helps and gives you all insight to what us veterans are doing nowadays! :wink: