Codex strengths and weaknesses - some tips for GPT 5.6

Codex Codex CLI
1

I got some decades background in programming and I used GPT models for programming since early 2023 and I worked with GPT 5.6 and codex for 11 days straight.

sleep, work, repeat - no breaks - just pure concentrated babysitting the machine with a plan and a mission.

I kind of knew exactly what I wanted to build and here is what I found and I think some of it might be really helpful.

Let me start with a little intro about programming - for the guys in here who use codex without a background in software development. I think this is neccessary, because it will help you to understand why codex behaves like it does and why this may seem odd to you.

Imagine you are building a website. It has 6 pages of content and a page with 10 products that can be bought there.

You want to try to find new customers by rolling out to another countrie’s market. So you tell codex to translate the page to a second language.

It does that - easily. No problems. It is fast and it does it fairly well.

And powered by this quick solution you decide “why not translate it to 250 languages” and sell the product world wide.

Now you may think the website should have a size of 6 pages * 250 languages. But it isn’t like that because you need to implement versioning of the content too. You need to check for legal stuff - since some products may not be sold to all countries.

But you still ask codex to implement that too. Which adds up - with functionality for fallback, currency, different pictures because e.g. cultural differences…

Each of thoose changes is adding to the complexity of the website and it is adding up to the things that codex has to keep “in mind” or let’s say in context. Which means for each change it needs to gather more information, read more files, do more research.

This is no difference to when humans work on a software project. We call it the onion antipattern. Because each layer of compleexity is added around the existing software core.

There are certain techniques that help us developers to reduce this complexity. For example modularization. We create a core that loads modules when they are needed and when you have to make a change you just have to know the core and the module you are working on and not the whole software at once.

But there multiple ways of designing such a core and not every core works on everything.
And then there is also the economical part in it. Because changing the whole software - implementing such a core takes time.

If you don’t do it then with the raising amount of stuff that needs to be added this so called technical debt compounds. It takes more time to implement new features and the rate of error also raises at some point because it is just not possible to load enough information in a way that it still makes any sense - and there is no difference wether it is done by a human or a bot - except that the bot can do it faster.

The problem that I see is that the model can’t decide when to do a refactoring or when to add a new layer of onion is the economical or some other reason based logical decission. It could even be that you just have like 40% of tokens left in your weekly codex budget but a refactoring would cost 80% or sometimes - when the code is huge it may even be way more.

I don’t have a solution for that for you people without technical background - but I want you to know that you are not having a machine that magically makes you a senior architect that is able to decide when to do what.

The moment you start using codex you become a junior developer. And junior developer usually - at least when they were smart - tried to find a mentor… a senior developer who teached them.

It makes no sense to come to this forum and yell at the model because it did the wrong decission. This decission is not possible for a model to make without giving it the right context.

If you read this text I hope you understand what is happening when you tell the model

“my customer will kill me if I don’t finish this in 3 hours”

or if you say

“I don’t like how slow this all works”

Because they will cause different behaviour. And the one who decides if you want a growing compounding blob of technical debt in exchange for fast money or if you want a scalable system that takes twenty times as long but then scales in features without adding more time after each feature - that decission is yours.

Just for the comparision: It took me more than 30 years to find this architecture. Something that works well with code (programmers/codex) even when it gets very complex in business logic.

Codex - at least sol is now very very capable of doing a refactoring. But - that does not mean it does not make mistakes. It is capable for me. Because I know what the outcomes must look like and I can tell it where it goes wrong and what the consequences are.

But if you don’t have that experience then do not use /goal! You have to learn! Like every junior developer had to do in the past. Or you must wait until someone like me gives their architecture to you and explains how to use it. And I can assure you that is not going to happen any time soon.

2

Let me first start by explaining what I wanted to build:

I am - for quiet a while - building a system that allows to create SaaS Platforms in a standardized way.

Basically it is a core with some modules and the modules need to use a SDK so it is possible to add more modules later.

The core should not be changed for that .. which means: immutable core + lously coupled modules.

This has the advantage that the model doesn’t need to see the whole application by design. It just needs to know the core application → which is represented by the SDK (because it is not really neccessary to show the model the full code of the core either. It just needs to know what a module can do e.g. using the cores RBAC or i18n logic) and maybe one or two example modules and/or the core ui library with UI elements (based on atomic design principles).

And that context can be reduced more .. by alot. Of course you can add AST graph and context from a vector DB and build communities inside the AST and from that decide what the context needs - and replacing functions with up to date doc blocks or interface definitions is also possible.

I have tried to build this as small as possible and I also tried to make it as complete as possible. Which came out eventually to be a combination of a marketplace + package manager…

But I optimized more over the years. I thought instead of giving the model 100 lines of code of a function I could build an optimized version of that and add a command to a programming language - I selected PHP for that and it’s PECL / Community Extension mechanism.
Which I had to update later because PIE became the new standard.

So I extended PHP with C + Rust + C++ and got rid of Rust then and Prof. Alexander Kolpakov replaced it with Assembler - which made the CI/CD pipeline alot faster.

was the result of that.

When the programming language has a build-in webserver and object store and many other things - then we don’t really need devops and all it’s helmcharts, kubernetes, loadbalancers, terraform/pulumi/… stuff. Hell, I’ve even added a llama2.cpp inspired inference command with a graph grounding and build in memory :slight_smile: - it is not perfect yet .. but runs a local model - maybe saying “like ollama” is a bit much - but it is way more than nothing.

Using a programming language that has alot of stuff build in reduces the amount of token obviously. That’s why Python works so well with AI coding. I kind of hoped it would end up in the training data for GPT 5.5 .. doesn’t look like that :frowning: - it is not even in 5.6.
And building an open source lib is quiet alot of efford just to get faster responses and less token consumption from a GPT model.

… Well, I still continued and the main idea was to - instead of letting the llm build code to build a graph schema of entities and AST and then manipulate that with analyzers / GPT and then from that compile the entire application.
Which surprisingly works.

1783025939076
17830259390761246Ă—991 106 KB

anyways… GPT 5.6 came out - and I thought before I continue with that I will try to get myself updated on the capabilities of OpenAI models - because my whole Idea was basically based on the idea that I wanted a less probabilistic code generation and smaller workflows because GPT 3.5 only had like 2k context window if I remember that correctly.

So - I tested GPT 5.6 and I made a new core in Python and a frontend system in vue.
I gave it a slight idea about the architecture. Basically: use frontend and backend service + a websocket service and a worker to run workflow steps + a few more microservices.. not a full microservice architecture but some things logically split - mostly for me.

What I figured is crazy. GPT 5.6 sol + codex can already reason over the whole architecture (I think there is a vectordb or a postgres locally??? I don’t know did not check it - but it can basically remember architectural decissions that I did in the past .. but it may as well be that I just learned them because AI kind of pressures you to use them… or OpenAI used my stuff as trainingdata .. who knows.. I will try to figure that out later).

… to be continued

3

and also this one for a list of usable tips