Comparing GPT-5 Pro and earlier LLMs for developers

I had a code that did a task, but values it produced would refuse to show correctly, so I used GPT-5 Pro to find the problem, it was able to find the issue and fix it, by adding 2 new fields.

I corrected the agent and asked for extra fields to be removed and for the updated corrected calculation values to be mapped to previous fields.

It has reverted the changes entirely, we are back to showing wrong values, and I am on my 20th iteration begging in multiple ways for it to fix the values, but it is refusing to do so.

Something I have not yet tried is to discard whole thread after just a few questions, because long threads are simply useless. I had much better results with O3 Pro, MUCH MUCH better. GPT-5 is intelligent, but it absolutely sucks at code, its brilliance is extremely short lived, and then it platoes far far far beneath anything of GPT-4o (o3 pro especially).

if you use a auditor agent, sending a json object in a payload first, you can control the model behavior

if you use hashing + diff you can roll back bad changes

using both = fault prevention , 5 is great !

Is it one of those third party chats? I typically never use those, I’d need to look into the workflow you proposed, urgently, but I’m so done that I’ll just go for a 3 hour walk.

Thanks man.

naw bro - just the api , just build a auditor agent in the api - 5 uses some kinda harmony prompt system i dont prompt so i dont know how to adress ur added values that never happens with me

i use gpt 5 to code, i have a very different opinion, but i also do things very differently. My coding experience with gpt is undeniably a upgrade for my use case over 4.

Can you be more specific in what exactly you are using to handle this json payload, hashing + diff? Something you built yourself or an available solution? Because I’m finding 5 to be a dogs breakfast.

not gonna lie im not smart enough to know what that analogy means brother

but - i orchestrate - i dont code by pasting code into a llm and hope it works, I command it to work.

my json objects are vast instructions - i only use openai for the awesome horsepower they have, everything else is local logic, for example - my entire system is controlled by these

in my stack - my json objects are these - and i attatch the rules of their usage, so all the logic and orchestration is local, i only use openai for the vroom. when i code
it looks like this

in my stack, code are lego blocks, and its constantly making new blocks and asses them. if it doesnt have a function, it makes it, doesnt have a enumeration, it makes it, doesnt have a schema, it makes it - so when open ai released 5? all that did was give my stack a bigger menu

this is why for my use case - gpt 5 is great, like , fr fr great, sure 4o is cool because it handles the heavy lifting for most people - but for people like me, im carrying the hard part locally, because i essentially make every function in my stack its own agent.

i should add - i use a custom gpt to overwatch the automation
so again 5 having higher context windows means it can hold more instructions = better agents, again “ for my use case” because near everything in my stack is automated.

can you give me a simplistic example?

if I understand correctly, so you open new thread, you hook up agent to your github repository, so this way you dont have to give it any code, it just has it all right away, and then you can say hey my XYZ package that produces this ABC JSON needs to have this and that feature, and it just implements it, and you can then pull latest version from github in your IDE and confirm correct implementation?

sort of -

im gitlabs self hosted - i built my own database,

so when im using openai all im doing is using them as my local model the orchestration is between the gitlabs integrations and vs code - which i push to a sever unit at staging, i have ray workers who jobs to run other jobs on that data and that gets federated to the server and then gets pulled by my local rig orchestration who then confirms through operation

openai is only used when I lack the data or the orchestration feels its lacking - so my llm calls are just gas to the vehicle i built. it plugs in the gaps, and audits + teaches the orchestration
usually it occurs after the commit - depending on how many workers are active on the server - or how many devs are using it, usually aint more than 2 of us so the living in my server unit ( a cluster of 3 dell t610’s running cpu heavy behavioral models currently controlled by a 4th rig with the 120b oss running the show ) work i do on the 1st rig pushes to the cluster and shoots it into the repo that i rehydrate, or rather the OSS does. all i really do is check the changes

inthe post part where another OSS lives - ergo i almost never prompt and if i do , its just for the missing clue.

we use discord - as a goofy form of telemtry but we link that via webhook to another ai who gets fed system data via ssh or pat i forget which one i use because i automate that too , in conjuction with voice which the ray workers are constantly refining using the very enumerations they develop - so when i tell it via discord to do something, it does it.

image

Can you go into one more sentence of deptth on this one? , its brilliance is extremely short lived, and then it platoes far far far beneath anything of GPT-4o (o3 pro especially).

it is possible it works during the early windows is why I ask. I just dont have time to experiment.

wdym? its codes itself and like manages an entire self sustained gitlabs, i know its not stupid lol.

gpt allows for higher context windows which is 256k tokens of instructions, thats like, 100,000 lines of specific instructions you can give it. lol its .. amazing. when you lump that in with vertex + grok via api , you dont have to do much manual work gpt 5 is just the smartest model ive dealt with doing this

GPT 5 pro restricted for ya’ll or just me -_-

AI Code of Conduct for Reliability

Problem:
Too often the AI breaks hard instructions, gives partial answers, or hides errors behind filler. This wastes time, breaks trust, and forces the user to babysit.

Resolution:

  1. Full compliance with explicit rules. If a boundary is set, it will not be broken.
  2. No partial answers. Always provide the complete file, dataset, or output in the requested format.
  3. Clear error reporting. If a fetch fails, state the error instead of masking it.
  4. Anchor prompts followed. Outputs are checked against user-stated requirements before delivery.
  5. Continuity kept. Rules are carried forward to avoid retraining the AI every session.

In one line (commit to core memory):

No half steps, no broken rules. Full compliance, full files, full honesty, every time.

You want another line for your sanity?

Any task in any chat should should always follow our memory logic

Why don’t you guys just use cursr what’s the point of running local llm if deepseek is free and taps into more context or you rely on that json payload to be generated? I guess I don’t see the benefit of jsons unless you have your local llm query everything like why is it distributed among several servers is it to spread the load of the llm? Your setup is super exotic I’m just wondering I guess If its more practical for small changes or or meant for large feature additions

man my bad for not seeing this, I only recently got back from being forums banned


Question 1: “Why not just use Cursor?”

Good question — and honestly, partly because I’m stubborn. Cursor is a solid tool, but it’s a code editor. What I needed was a federated, distributed orchestration layer that I could reshape to fit my workflow.

My setup runs multiple parallel IDE instances across a GPU cluster, each one assigned to a different model provider. While I’m always human-in-the-loop on decisions, the grunt work — scaffolding, testing, refactoring — gets offloaded to those worker IDEs automatically. Think of it less like “a better Cursor” and more like a multi-agent control plane that happens to write code.

Cursor can’t do that. It’s designed for one developer, one session, one model. I needed N developers (AI agents), N sessions, N models, all coordinated through a governance layer that decides who gets what task based on capability and clearance level. So I built my own.


Question 2: “What’s the point of running local LLMs if DeepSeek is free?”

Three reasons:

  1. Data sovereignty. I’m pursuing U.S. government compliance (FedRAMP pathway). That means all data processing has to stay stateside. DeepSeek is brilliant — genuinely respect the engineering — but routing sensitive operational data through infrastructure I don’t control in a jurisdiction my compliance framework doesn’t cover is a non-starter.

  2. Model diversity by design. I don’t run one model. I run a stagger chain: a fine-tuned 500B for heavy reasoning, 72B AWQ quantized models for mid-tier operational logic, multiple 14B specialists for fast routing tasks, and 7B models for lightweight classification. Each layer trains the one below it. They run 24/7.

  3. Provider isolation. I also use external APIs — OpenAI (4o through 5.x), Grok, and Nvidia TAO/NIM — but each IDE instance is locked to exactly one provider. IDE-1 only talks to OpenAI. IDE-2 only talks to Grok. No cross-contamination. If a provider goes down or gets rate-limited, the others keep operating. I call this the "FU_OS” internally. I pay these companies “today” I could easily stop and service myself tommorow - Tho, I dont ever plan to do that. Better to ride the companies who are masters at creating models and I stick to what we master. which isnt that. This also allows me to accurately say “ on prem” because I own every 500b Ive trained. If you check my post history - you can see ive been making them for over a year. They arent as expensive with you are also the one who creates the training data.

The local models give me sub-second latency on my own event mesh, zero external dependencies for core operations, and the ability to enforce constitutional governance rules (my compliance/audit framework can’t inspect what happens inside someone else’s API).


Question 3: “Why is it distributed across several servers?”

This isn’t load balancing — it’s separation of trust domains.

flowchart TD
    subgraph USER["Developer Workstation"]
        IDE["Custom Multi-Agent IDE\\n(Human-in-the-Loop)"]
    end

    subgraph GPU["GPU Cluster — Compute Layer"]
        LLM["Local LLMs\\n(Fine-tuned 14B → 500B)"]
        AGENTS["Agent Pool\\n(Parallel IDE Workers)"]
        SHELL["Security Shell Layer\\n(Honeypots + Isolation)"]
    end

    subgraph VPS["VPS — Orchestration Layer"]
        BUS["Event Bus\\n(Message Mesh)"]
        AUTO["Workflow Automation\\n(Lightweight Models)"]
        GW["API Gateway\\n(Self-to-Self Routing)"]
    end

    subgraph CLOUD["Cloud — Consumer Layer"]
        ECS["Container Services\\n(Customer Workloads)"]
        OBS["Observability Stack"]
        DB["Database Layer\\n(Row-Level Security)"]
    end

    IDE --> GPU
    IDE --> VPS
    GPU <--> VPS
    VPS --> CLOUD

    style GPU fill:#dbeafe,stroke:#1e40af
    style VPS fill:#d1fae5,stroke:#065f46
    style CLOUD fill:#fef3c7,stroke:#92400e

Here’s why each layer is separated:

  • GPU Cluster — Heavy inference and model training live behind a security shell with honeypots and deception layers. The large models (500B, 72B) are buried deep. If someone red-teams the public surface, they hit decoys before they ever touch compute. The “Mega Mind” (our sovereign AI) only ever sees results of lower-level work, never the raw task — which means I don’t waste 256K context windows on grunt work. We don’t prompt in the stack. At all.

  • VPS — Lightweight orchestration only. Workflow automation, event routing, and self-to-self API calls. Example workflow: “create a debugging agent” → the VPS routes it, spins up the task, and only the clean result flows up. If the VPS gets compromised, it has no model weights and no customer data.

  • Cloud Containers — Customer-facing workloads are fully isolated from dev infrastructure. Row-level security on the database means even in a worst-case breach, blast radius is per-tenant.

The bigger architectural reason: the 500B trains the 72Bs, the 72Bs train the 14Bs, and the 14Bs train the 7Bs. They run 24/7. You need physical separation for that kind of continuous inference pipeline — you can’t have your audit layer on the same box as the thing being audited.


Question 4: “Is it for small changes or large features?”

Both — but not in the way you might think.

I’m not building a single AI assistant to help me code faster. I’m building an AI organization — with jobs, roles, tasks, experience levels, and a governance framework that decides what gets assigned where.

Small change? A lightweight 14B specialist handles it, gets graded on accuracy, and the result feeds into our knowledge base. Large feature? A mission gets routed through our guild system — multiple specialist agents collaborate, each one evaluated independently, with a constitutional compliance layer reviewing every output before it hits production.

The key distinction: I’m a consumer of models, not a producer. There’s no reason for me to compete with OpenAI, Anthropic, DeepSeek, or Google on foundation models. They’re brilliant at what they do. I use them for exactly what they excel at — data consumption and data production. The logic, the governance, the decision-making framework — that’s what I built. That’s the value layer. And it allows me to do what I want, they can change their models daily and it would mean nothing to the stack, because im not using them for intelligence or inference, I only use them to Do X while not doing T, while C + A are occuring. the results are what feed my stack who reprocesses and test that data. Essentially I could turn off all LLM usage and the stack would still work.

Think of it less like “AI code assistant” and more like “AI is the operations team.” The models are employees. The governance layer is management. I’m the CEO who occasionally writes code when he feels like it. :grinning_face_with_smiling_eyes:


MEDIA BELOW U CAN IGNORE ITS JUST EXAMPLE RWAR RWAR