LLM for Development, Scripts for Production: A Lesson from a Failed AI Workflow

Codex
1

A useful lesson from today.

I had a business idea that required collecting, filtering, enriching, and analyzing a large number of small-business records. I initially prepared a very detailed prompt and ran the entire task through an OpenAI reasoning model at maximum settings.

The result: all available tokens were consumed in about 14 minutes, while the task itself was still far from complete.

Then I followed advice that another developer from this community had given me earlier: repetitive and deterministic work should not necessarily be performed by an AI model. Use AI to write the scripts, then connect those scripts into a pipeline.

That is what I built today. The pipeline now downloads the data, applies filters, enriches the records, checks matches, assigns statuses, and generates review files automatically.

After the scripts were written, I tested the full pipeline end to end. It produced the final result with zero additional AI tokens.

The main takeaway: do not use an LLM repeatedly for work that can be expressed as deterministic logic. Use AI once to help build the automation, then let the pipeline do the repetitive work.

Thanks to the developer who suggested this approach. I finally applied it, and the difference was enormous.

2

That sounds like me.

However, much of the repetitive processing was performed locally rather than through model reasoning.

I think it is excellent that you asked the question, accurately paraphrased the idea:

You moved deterministic and repetitive work into ordinary Python code, while /goal remained responsible for orchestration, interpreting failures, distinguishing real defects from false positives, and changing the implementation when necessary.

You then implemented that approach and successfully completed the task. Well done.

3

Before you could write code so efficiently with AI this was already the case.

But it is also worth noting LLMs have very useful properties for processing data in specific ways, eg classification, image recognition, which you can’t easily do with deterministic code. So it has a place in data processing workflows too where appropriate.

And yes it’s great for coding and on the fly natural language interpretation, take a look for example:

4

Yes, that was you. Thank you again.

Your explanation is more precise than the way I originally described it. The key was not simply replacing the model with scripts, but separating responsibilities correctly: deterministic and repetitive processing moved into local Python code, while the model remained responsible for orchestration, diagnosing failures, identifying false positives, and adapting the implementation.

Today I finally applied the approach you suggested, and the result confirmed how effective it is. The full pipeline completed the task reliably without repeatedly spending tokens on work that code could handle much better.

5

In the other topic, you asked:

how did you design the interface between the Python pipeline and /goal?

I gave a very simple description of the actual implementation in my answer.

I suspect many people will have the same question. My simple answer might initially appear to be an evasive answer— that avoids addressing the substance of the question—but it was not.

Could you share more about what you learned in relation to that question? What did you originally envision, what ultimately worked in your successful implementation, and what did you learn along the way?

6

Yes, and I am especially grateful that you did not give me one rigid implementation.

At first, your answer seemed broader than the literal question I had asked. But that turned out to be valuable. If you had given me one concrete architecture, I probably would have attached myself to it and tried to reproduce it exactly. Instead, I had to think through several possible ways for Python and /goal to cooperate.

What I originally imagined was a more direct interface: /goal would pass work to Python, Python would process it, and /goal would receive the result and decide what to do next.

In practice, I did not begin by designing that complete interface. I first needed to prove that deterministic scripts could actually replace the expensive repetitive processing that had consumed my limit without completing the task.

So I worked slice by slice:

  1. I divided the larger task into separate processing stages.

  2. I created a Python script for each stage.

  3. I ran each script independently and inspected its output.

  4. Once several stages were reliable, I connected them into one pipeline.

The practical interface became much simpler than I had initially imagined. Each stage receives defined inputs and configuration, performs deterministic processing locally, and produces structured outputs such as CSV files, status information, and logs.

The model does not need to reason over every record. Its role is mainly to help design or modify a stage, run the workflow, inspect unexpected results, distinguish data problems from implementation defects, and adapt the code when the assumptions change.

After the first three stages worked, I started building a dashboard so that I could configure and run the pipeline more conveniently. That exposed an architectural weakness: my original scripts were too narrowly coupled to one specific use case. I then had to separate the reusable processing logic from the adapters, configuration, and interface layer.

My current process is therefore:

  • define and test one narrow slice;

  • automate it with a deterministic script;

  • validate the output;

  • generalize its interface where necessary;

  • add it as the next stage of the pipeline and expose it through the dashboard.

The most important thing I learned is that the interface between Python and /goal is not primarily about a sophisticated integration mechanism. It is about having clear contracts between stages: explicit inputs, deterministic processing, structured outputs, and visible failures.

The model is most valuable outside the repetitive processing loop—where interpretation, debugging, architectural changes, and handling exceptions are required.

So your answer was not evasive at all. By leaving the implementation open, you gave me room to discover an architecture that matched the actual problem instead of forcing the problem into a predefined solution. Thank you again for that.

7

Your reply and the details you shared about your interface are invaluable. I hope others will follow a similar path and discover that, in journeys like this, what one learns along the way is often more valuable than reaching the final destination.

8

Day 4 of trying to make an AI system less eager to be helpful.

After four days of changing the workflow, tightening the prompts, adding validation steps, and repeatedly breaking things that looked promising, I finally got this:

“Result of this pass: 0 ideas I’m willing to call viable. That’s better than producing three attractive ideas and then having to destroy them myself.”

The run took 54 minutes.

And strangely, this may be the most useful output I’ve gotten so far.

The interesting part isn’t that the model found nothing.

It’s that I’m finally getting it to prefer “nothing passed the evidence threshold” over “here are three plausible answers because you asked for three.”

For me, that feels like a much more important step than improving generation quality: teaching the system when not to generate an answer.

Curious how other developers handle this:

How do you design prompts or agent workflows where “no valid result” is treated as a successful outcome rather than a failure to complete the task?

9

Because context is critical, it is difficult to give a specific answer without knowing the problem being solved. Here, “context” refers not to the LLM’s context window, but to the nature and constraints of the problem itself.

Consider the N-Queens example. For (N = 2), the row variables can be constrained as follows:

A in 1..2,
B in 1..2,
A #\= B,
abs(A-B) #\= 1.

After applying the domain constraints, only four assignments remain:

1,1  invalid because A #\= B
1,2  invalid because abs(A-B) #\= 1
2,1  invalid because abs(A-B) #\= 1
2,2  invalid because A #\= B

Because every possible assignment within the defined search space has been eliminated, “no solution” is the correct and successfully established result.

Although I mentioned using Python, I also prefer Prolog, constraint-solving systems, and similar formal tools for this type of problem. They are often better suited to distinguishing among:

  • a correctly proven absence of solutions;
  • an incomplete or incorrectly defined search space;
  • a flaw in the constraints or logic; and
  • a search that stopped before exhausting the valid possibilities.

For an agent workflow, I would apply the same principle: define the search space, constraints, stopping conditions, and required evidence in advance. The agent should then report “no valid result” as success only when it can demonstrate that the relevant search space was covered and that every candidate was rejected for a stated reason.