I’ve been playing around with tool calling and agents recently, and I keep coming back to the same question:
At what point does a workflow actually need to become an agent?
For example, say I have an application where the model can access a knowledge base, customer data, and a few APIs.
I could define the flow myself:
Get customer → get order → check policy → generate response
Pretty straightforward and predictable.
Or I could let the model decide what it needs:
“I need the customer’s order first… now I need to check the policy… okay, I have enough information.”
That feels much more like an agent.
But I’m not sure where people are drawing the line in real applications.
Do you let the model decide the entire sequence of tool calls, or do you keep the important parts of the workflow deterministic and only use the model where reasoning is actually needed?
And then there’s MCP. If tools are becoming easier to expose to models, does that change how you think about this architecture, or is it mostly just a better way of connecting tools?
Curious how people here are approaching this, especially if you’ve actually deployed something with the Responses API or MCP.
Workflow: pre-determined process with fixed states. Might include branches travelled under certain conditions. Might include AI automated decisions. Often includes human gates.
AI Agent: determines it’s own process, states, process order and whether or not it employs available tools.
Where do you draw the line between a workflow and an AI agent?
Before answering, could you define these terms based on your thinking more precisely?
Workflow
AI agent
Agent
AI
It would also help to know the context in which you are comparing them. For example, are you asking about coding, a conversational chatbot, batch processing, theorem proving, or something else? The distinction may depend heavily on the intended use.
For your customer-support example, I’d keep that fixed flow as the baseline. Having three tools available isn’t enough reason to hand over their ordering.
I’d run the same requests through both versions and compare correct resolutions, cost, latency, and failures, not just whether the model picked a plausible sequence. If adaptive sequencing helps only a subset enough to justify the extra cost and failure modes, that’s where I’d introduce it and leave the rest deterministic.
I’d treat MCP as the tool connection layer in both versions, not something that changes which parts of this support flow need the comparison.
Using your working definitions, I would say: when the task benefits from AI deciding which actions to take toward the goal.
I would start with the task’s requirements before choosing how much control to give the model.
I tend to prioritize accuracy and correctness. I assess speed, architectural patterns, programming languages, and tools against that priority.
As I develop and refine the process, I ask questions such as:
Is AI needed for this step, and what kind?
Which decisions can be expressed reliably as explicit rules?
Which parts need deterministic behavior?
Are there existing applications or tools that would help?
Are custom validators needed to check outputs and enforce constraints?
How will I verify that a change improves the results?
Although your focus is on LLM applications, my use of AI also includes older symbolic approaches, such as expert systems and logic-based reasoning implemented in Prolog. Using Prolog does not, by itself, make a program AI.
Determinism and correctness are also separate properties: a process can consistently produce the wrong result.
I expect MCP to remain useful for some time, although that is a prediction. I use it, but its availability alone does not determine my choice of interface. I still consider whether MCP, a direct API call, a library, or another interface best fits the task.
I decided that LLMs are best suited to creating or performing transformations on artefacts, rather than trying to manage their own workflow. So I tried to separate the deterministic logic of the workflow while preserving the flexibility of what LLMs can generate.
I made a whole framework / workflow architecture based on that premise! I’d love to hear what you think of it (there’s a link in my profile page). You’re clearly in a similar headspace to what lead me there
@rjj1 Yeah, that makes sense. I like the idea of keeping the deterministic flow as the baseline and only introducing agentic behavior where it actually improves the outcome enough to justify the trade-offs.
Yeah, this makes a lot of sense. I especially like the idea of starting with the actual task instead of deciding upfront that it needs to be an agent.
The point about determinism vs correctness is interesting too. A workflow can be completely deterministic and still consistently give you the wrong result.
For me, the real question is: when does giving the model control over the process actually improve the outcome enough to justify the extra complexity?
I had a quick look at the framework. This is actually very close to the direction I was thinking about.
I really like the separation between the model proposing/generating something and the framework deciding whether that output can actually be executed or accepted.
I have been programming for over 40 years, so I see no reason to suddenly hand every problem over to AI. When I can solve a problem reliably with deterministic code and validate the results, that is usually my preference. It also avoids unnecessary token usage.
For problems better expressed as rules, constraints, or a search for solutions, I would consider Prolog first. If those approaches are not a good fit, I might consider generative AI. One of my main concerns then is validation: how do I check the result, and can I obtain it in a format that supports those checks? Having the right format does not, by itself, make the result correct.
Using a model for part of a task is also a separate decision from giving it control over the process. I would consider the latter when choosing the next step dynamically provides a measurable benefit that justifies the additional cost, complexity, and validation effort.
There is no single approach that fits every problem, but that is how I tend to approach the choice.
I 100% agree with @EricGT that AI shouldn’t be used to solve every problem, though I can only boast 30 years of programming myself
One of the earliest failings I found with agents was simply keeping track of where they were in a long list of tasks. This was a while ago, using older models, but the simple fact that I took away was this: We don’t need to reinvent the wheel. We already have programs that can track progress through a list of tasks very reliably, handing this over to an LLM had no advantage and only introduced risk.
Once you establish this fact, it becomes obvious that a hybrid workflow is needed, where deterministic code and the LLM stick to what they are good at. This was the design philosophy that underpins my framework.
@suryacreatx , I have been looking for someone to be a successor / inheritor for my framework for about a year now. Feel free to take a deeper dive or experiment with it! I’d be so glad if someone did find it useful
That makes a lot of sense. I’m still pretty new to thinking about this from an architecture perspective, so I hadn’t really considered the validation side as a separate decision from using the model itself.
The point about giving the model control only when it provides a measurable benefit is especially interesting. I think that’s the part I’m trying to understand better, not where AI can be used, but where it actually makes sense to give it control.
