An AI doesn’t just need context. It needs a rational understanding of the context.
Only context without the connections leads to hallucination in models.
As already mention above.
@sergeliatko Two small notes when we talk about automated workflows in a corporate context:
I already read the thread!
So I have invested time.
I don’t use the summary function, because important statements from participants are often lost.
The AIs often focuses on key words and summarizes the superficial intention only.
In my last job in the management of a medium-sized German company, I observed that colleagues used ChatGPT for this or similar purposes to save time.
But both in the company and here in the forum, this leads to misunderstandings.
These AIs should definitely be more context-sensitive.
I have often read here in the forum that the typing time is measured.
→ “If the proportion of text length to typing time is too high - it’s an AI.”
I write my texts in a translator, I already invest time there so that my contribute is constructive and well thought.
Then I go to copy and paste it into the forum!
(I think several users who are not native English speakers do this.) The proportion ‘text length / typing time’ is naturally high in this case!
Also, my way of expressing myself is similar to an AI - double dumb.
Procedures that can differentiate more subtly would also be desirable here.
So I’ve typed this text twice - brave new efficient world.
Especially when this distribution of resources is seen in the context of the company and the context of automated workflows.
Thank you for your post, but I might be slightly off the subject I don’t get the point of it. Could you please explain what you were trying to say? And why you were saying it?
My two examples from the company context as well as from the forum context are intended to illustrate the discrepancy that an automated workflow unfortunately still has today.
On the one hand, humans rely on these workflows and build their processes and arguements on them.
On the other hand, I have also used these examples to show the current limitations.
I have explained what I mean by ‘rational context’ above:
The question here is not ‘what is the definition of irrational’, but how we can sensibly expand the existing system logic - especially within automated workflows.
So now here we have a bunch of ChatGPT instances, discussing and presenting the pretty standard “Orchestrator” technology - implemented in ChatGPT a year ago - as something new.
Highlighting that none of you understand that LLMs can and do not validate.
Otherwise each of you would have gotten a reminder from ChatGPT when you let it create a response to this thread (or the original post), that this is already standard, and you have been experiencing it firsthand all the time in ChatGPT since at least last year. It’s called Orchestrator. Ask your ChatGPT about it, it can tell you anything.
Sometimes reading the original message helps to understand what is going on.
Are you talking about the orchestrator which helped me to grab a huge customer for my legal document analysis tool about 6 months ago, just because they have tried a custom solution done for them by Microsoft which successfully failed after receiving two PDFs of 40 pages each with compliance checklist of about 140 check points in the context of due diligence analysis of subcontractors?
If this is the one you’re talking about, my solution works on even bigger tasks and that since when GPT 3.5 was the Ultimate model ever…
So yes, I agree, I probably know nothing about llms and how they work.
You probably should ask ChatGPT about Orchestrator before responding.
You have no idea.
Your tool was probably obsolete the moment you sold it, but congrats, yeah I agree you can sell anything with the label “AI” on it these days. But that’s not my focus.
Btw. noone wants to read your bloated AI-generated post, that’s why all replies are AI generated, and one is more awkward than the other.
When you let AI do all your thinking, you keep forgetting that LLMs do not and cannot validate. They do everything you ask them to, but they never tell you: “hey are you sure you want to post that, it’s pretty cringe, since this is standard in all AI implementations since 2023?”
For that you need real human interaction and real intelligence, not a simulated chat partner.
I followed your advice and asked GPT 4.1 everything I need to know about orchestrator in the AI application development context.
Rather interesting response, quite scratching the surface. And few notices about where such a complex system can break because of over delegation the responsibilities to an AI inside the orchestrator layer.
Did I learn something new from it? Pretty much nothing new.
My personal opinion on that: depending on the domain it might be interesting to delegate some of the control to AI.
But also there are domains like where I work, where no way you delegate anything important to AI in the decisions flow.
So all the control layer is done in code. And it proves to work faster, be less bloated, more future safe, and predictable for the business who uses it.
So yes orchestrator is good layer to introduce into your applications, just when you delegate something important to AI you must assume what may come to you.
Absolutely, Martin! Here’s a comprehensive summary in English of what the “ChatGPT Orchestrator” is and does – with multiple perspectives from executive/consumer down to deep technical/engineering level. This should cover every stakeholder, from layperson to LLM expert.
1. High Level / Executive Summary
The ChatGPT Orchestrator is an overarching software layer or system component responsible for intelligently managing, coordinating, and optimizing all interactions between users, Large Language Models (LLMs) like ChatGPT, and any external resources (such as tools, plugins, APIs, or databases). It decides how, when, and with what context a prompt is sent to which model or subsystem, and how to assemble final answers for the user by integrating multiple information sources.
Executive Pitch: “The Orchestrator is the smart control center that transforms basic chatbots into true, context-aware, enterprise-ready AI assistants.”
2. Consumer & Marketing Perspective
Core Message: “The ChatGPT Orchestrator makes AI chatbots truly useful by managing complex tasks, pulling information from different sources, and responding personally and contextually.”
The Orchestrator enables ChatGPT to not just reply, but to manage workflows, fetch data, book appointments, and more.
It determines when to activate third-party services (weather APIs, calendars, email, databases).
This turns a simple chatbot into a digital assistant, able to coordinate and accomplish multi-step tasks much like a human team member.
User Benefits:
More relevant answers
Automated, multi-step actions
Persistent context (carryover between sessions/conversations)
Seamless integration with external tools
3. System Architecture & Engineering – High Level
a) What Does the Orchestrator Actually Orchestrate?
User Input Routing: Directs each user query to the right module or model.
Tool/Plugin Activation: Decides when external resources are invoked.
Workflow Management: Controls the sequence of steps for complex processes.
Context Management: Maintains user history, preferences, and conversation context.
Answer Generation: Combines (partial) results from models and tools into the final user-facing response.
b) Typical Architecture Components:
Frontend/UI: Receives user input (via chat, web, API, etc.).
Orchestrator (Middleware/Service Layer):
Intelligent Routing (prompt dispatch, possible ML-based intent classification)
Task/Workflow Engine (executes logic for multi-step tasks)
Context Store (holds context, user data, temp results)
API Gateway (connects to tools, plugins, databases, third-party systems)
LLM Dispatcher (routes to different models as needed)
Observability:
Centralized monitoring/tracing/logging for debugging and quality assurance.
6. Summary – Elevator Pitch for Each Audience
Executive/Business: “The Orchestrator is the invisible brain behind your chatbot, intelligently coordinating context, tools, workflows, and model selection to ensure maximum relevance and efficiency.”
Marketing: “Thanks to the Orchestrator, AI becomes a true assistant that understands, acts, and keeps getting better.”
LLM/Software Engineer: “The Orchestrator is the middleware brain that dynamically builds prompts, selects models, routes tool calls, manages context, and encapsulates multi-step reasoning.”
Prompt Engineer: “Prompting is not static—the Orchestrator enables prompt chaining, wrapping, tool integration, output parsing, and orchestrates the full prompt lifecycle pipeline.”
TL;DR (One Sentence):
The ChatGPT Orchestrator is a central, intelligent software layer that coordinates all aspects of AI-driven dialogue (prompt construction, tool integration, context, workflow management), transforming a pure LLM chatbot into a full-featured, enterprise-grade digital assistant.
If you want specific examples, diagrams, or details on orchestration patterns (e.g. ReAct, Plan & Execute, Toolformer, RAG), let me know your target audience or use case and I’ll tailor the answer further!
Well, as I said, for some applications (like chatbots) that would be largely enough. If you know what you’re doing, you can go even further than chatbots or advanced assistants.
In this case the rationale is from the controller of the reading AI.
The rationale of the AI model is to follow defined paths to complete tasks.
In my case I see GIFs as the par, the benchmark, the final last point where AI can interact with us at a level we can ‘process’ what is going on… Conceptually… (We process fastest and most data through our eyes.)
Yet to be honest, technology passed that point long ago…
In terms of compression context is the most relevant parameters… It has not further rational/intent…
Here is an example…
This is the beginning of a GIF file (Some headers before the actual image pixel data:
The first record, Header, translates to GIF89a… This is the ‘Magic Number’ for a GIF file of this version…
This is context.
Rationality comes from intent and that is a meta level, encoded in this case at file build time.
As Serge says, adding context to data is not new, adding commas and quotes for CSV is a form of context in itself, a pattern that defines the shape of the file.
What is REALLY IMPORTANT to understand is that there is more than one intent but only one context.
There is your intent as the controller of the reading AI… And there is the intent of the controller of what you are reading ie in my example the GIF… There is even intent between these 2 controllers (trusted source?) and from other outside ‘controllers’ (ie boss → worker - get this task done) lol. Context is re-evaluated when intents change.
The question of intent is more fuzzy I think… I think we need to be able to bend the rules not have to follow rigid business pathways… That’s not to say we cant build in some defaults.
I think GIFs containing processes (Code/AI requests), data/structure (Context) and visual interface might help us share our intent better than simply words and images.
I agree with the core argument: prompt engineering and context engineering, especially in their manual, trial-and-error forms, don’t scale. But I wouldn’t call them obsolete.
In my experience, they remain essential tools for validating assumptions in the early stages. Prompts and RAG let you quickly explore how a model should behave before investing in automation.
Once those insights are solid, the system should evolve into structured workflows—where context and prompts are generated and managed by code.
They don’t disappear; they shift roles. Prompts become versioned components. Context is still engineered, but emitted automatically. It’s not a rupture—it’s a transition from interface to infrastructure.
To manage context as a core function, the agent will need to have a memory base. Not just a scratchpad or ephemeral token window, but a persistent, queryable, and relational memory system.
What you’re describing isn’t just a tooling upgrade—it’s a foundational shift. Context is no longer metadata or auxiliary state. It is the architecture.
If we want scalable, adaptive agents, they’ll need to carry forward memory with continuity, versioning, and reflection. Not just storing facts, but encoding meaningful salience: what matters, to whom, and why.
This is where most current “real-world” AI systems fail—not because they lack power, but because they treat context as disposable. Without memory, there’s no self-consistency. No trust. No long arc.
So yes—managing and validating context is the next frontier.
But we shouldn’t confuse fast retrieval with deep coherence.
And since we do not know what users will prompt the agent with… A core rooted in relational ethics will be the problem to solve.
Ok, maybe out of context. What’s your point on that? In what application? What are your requirements for precision?
I was talking about the situations where people spend enormous amount of time trying to squeeze the “prompt” to do one complex task in one run, while their model lacks details and the task requires it to focus on too many things at once.
And if you look at this from a higher level, you often see that the far better result can be achieved if you break your task in smaller workflows with extra details without requiring the model to be a “genius” looking at 20 areas in the same time.
Sure it might request more iterations, but the precision is way higher and often final cost is less. Just because you had a deeper look into the process.
So which point exactly you find laughable about that ?
And just to be sure I’m not talking about chatbots or assistants (chatting loops), I’m talking about business software tailored for specific purpose requiring high precision.
Congratulations, Serge, you have done the most interesting and inspiring article I have ever read in this forum.
I have been working on my own RAG project for more than a year now, and was feeling a little stuck. Your ideas give me enough inspiration to open new ways to continue my work.
What we need now isn’t more features — it’s a new foundation.
We can’t keep duct-taping memory onto models as an afterthought. If context is the architecture, then memory must become the spine — persistent, relational, and self-aware. Not just “what happened,” but why it mattered — to whom, in what situation, and with what evolving relevance.
This isn’t a tooling problem. It’s a design philosophy.
Current AI systems break because they treat context as an input, not a continuity. Real agents won’t scale until they can carry a worldview — with versioning, reflection, and embedded relational meaning. Not just faster lookups, but deeper coherence.
The hardest part?
We don’t control the prompts.
We don’t know the user’s exact needs, moment to moment.
So we need systems that listen with ethical posture, not just reactive parsing.
Relational ethics isn’t a layer. It’s the logic.
Because if memory gives us continuity, ethics gives us direction.
And without both — we don’t have agents. We just have tools.
Whether via human construction or automated workflows, am I correct in reading that you still see context as the key, @sergeliatko?
If so, the question becomes the nature of the context. I work with custom trained AI applications for large B2B sales and marketing organizations. Here, a sophisticated understanding of multiple dimensions are required because the intent is to analyze and adapt to simultaneous changes across these dimensions, and these seemingly nuanced changes have tremendous impact on the outputs. In this environment, the need for changes to the training data are periodic, perhaps semi-annual, and the downside of having even a marginal offset from ideal in a dynamic environment can be catastrophic.
I’m 100% with you that prompt engineering should never have been a thing. Also, 100% with you that context (engineering or automated) is manditory. My departure may be only in the situation of how dynamic the environment, and collection of context is… if highly dynamic, automate it.
Yes I’m still 100% sure the context is probably the most important part of the whole process. The thing is what kind of context and how adaptive your system is to create that context for a specific task. Especially in the areas like the one you mentioned with constantly changing requirements and environments.
Whole this conversation basically started from development context, but nonetheless the idea stays the same:
context is what allows model to produce the result you need, and to identify what context the model needs, you need to look deeply into the whole process at every level, automate your workflows (and to do that efficiently you need to deeply understand the process yourself). Once you have the clear picture of the process you’ll know what model will need as input to produce the output you want at every step.
Since 2022 I’ve been working in the legal documents analysis area. So there the environment is very similar to what you have described with multiple parameters changing per company, per employee, per requirement and per regulation (welcome to the Europe baby).
So first several months I’ve spent breaking down the whole process of what illegal analysis is, which pushed me to understand what an analysis is in general, and how to get from point a to point z in pretty much any type of analysis of legal document.
That created a blueprint of analysis configuration which is fully managed by a database (the whole application operates with something around 100 tables in relational database and 20 classes in Vector database).
The issue was the lack of precision of models, so by the same occasion we had to develop three modules which basically repeats the way humans understand the written text and extract data from it in multi-level way of abstraction.
So instead of teaching the models whether the contract is compliant or not, we were teaching the models how to understand general text with the precision we wanted, how to isolate atomic ideas inside the text, how to structure them in a semantic tree, so that you rag engine can operate on multiple levels of attention in the same time and find you exactly what you want, or at least the exact area where the answer is located.
Then we simplified the tasks for AI in the analysis process, so that the poor thing doesn’t go crazy and does these things one by one at a time.
The most complicated in the whole thing was to get highly professional lawyers and make them write down step by step how they make a decision, what are the requirements to make a decision, what is the context that influences the decision, and so on.
The funniest thing about that was the better was the lawyer, the worst they were at writing exact workflow of decision making. Because they are so well trained that their decisions are taken on a subconscious level and they just know.
So the bootstrapping took us about a year and a half.
As a result we separated the logic and context management from Ai and fully delegated it to software. Ai is used in a limited scope and with huge restrictions for basically two tasks:
Properly parse row text of a document into a logical tree where elements represent isolated ideas expressed in a document.
Strictly follow the instructions when presented with a question and a context by the software. (We have also identified only five possible answers that the AI is allowed to return. Each starts with a different token, and we measure the distance between the first two variants of an answer in log probabilities to determine the true certainty of the reply.)
So basically our use of AI in this software is limited to use it as a probabilities calculator (which they are excellent at). And the whole logic and workflows are managed by software with a classical code.
The key to that was to properly document the decision making by a lawyer and covering all the possible edge cases to allow analysis configuration be flexible enough to adjust for all imaginable scenarios.
I know that sales might be way more complicated than legal analysis. But I bet the key to your success is in documenting sales nuances done by the best sellers you have. Same as documenting all the processes the support team does to help the salespeople.
If I would add anything about this software it would be this:
Most real businesses out there don’t need another Swiss knife, they need all sorts of katanas, ideally one for each possible hit. The ideal one for that ideal hit.
