How should we evaluate AI-assisted engineering decisions?

ChatGPT Use cases and examples
, , , ,
1

I’ve been thinking about a problem that has become increasingly common for me while building AI applications:

When several AI models give different answers to an engineering question, what should actually determine which answer we trust?

For example, suppose I’m building a RAG system and ask:

“Should I use pgvector or Pinecone for this project?”

I can ask ChatGPT, another model, search documentation, read blog posts, and look at examples from other developers.

But I don’t think the difficult part is getting more answers.

The difficult part is deciding:

  • Which evidence is actually relevant to my constraints?

  • How should conflicting recommendations be handled?

  • How much confidence should I have in the final recommendation?

  • What information would change the recommendation?

  • And, most importantly, what should I actually do next?

I’ve been experimenting with a small system around this idea.

Instead of treating an LLM response as the final answer, the system tries to structure the process as:

Context → Evidence → Deliberation → Decision → Action → Reflection

One thing I found particularly interesting is that model agreement isn’t necessarily evidence of correctness. Multiple models can agree for the same wrong reason.

This made me wonder whether a useful AI engineering assistant should expose things like:

evidence quality, disagreement, uncertainty, trade-offs, and counterfactuals

rather than simply producing a confident answer.

I’ve also been using ChatGPT and Codex extensively while developing the experiment. ChatGPT has mainly been useful as a reasoning/review partner, while Codex has helped with implementation, debugging, refactoring, and verification.

That raised another question for me:

If AI is increasingly involved in making and implementing engineering decisions, how should we make those decisions easier for humans to inspect and challenge?

I’d really appreciate perspectives from developers here:

  1. Do you think this is actually a problem worth solving?

  2. Would exposing model disagreement make your engineering workflow better, or just slower?

  3. What signals would you want to see before trusting an AI-generated engineering recommendation?

  4. Are there existing tools or approaches that already solve this well?

I’ve been exploring these questions through a project called GenPHD. If anyone is interested in seeing the implementation or discussing the approach, I’m happy to share it.

I’m much more interested in hearing where this idea is wrong than in promoting the project.

Thanks!

4

Career developer here, programming since 1983, professionally since 1992.

Answers to your questions in order first, then elaboration if needed will follow.

  1. No, at least not exactly, not like you’ve approached it here anyway. You have slightly over thought this. Here is a very simple metaphor to save a long winded attempt to make it complicated where it need not be.

How do you know your Texas Instruments Calculator is correct? Because the answer is correct. It’s really exactly that black and white, in any engineering endeavor. NOW, is there sometimes more than ONE correct answer? Certainly. Some problems can have multiple possible correct solutions. I can provide examples, but I am sure most people capable of reading this can think of some anyway. So you want the “best” answer? Ok, define best. Cost? Performance? Complexity/simplicity? etc etc etc. What makes the definitive best? With those metrics, the answer is mathematic (or logic anyway) and your TI calculator cannot do that wrong, unless its faulty. If its faulty, that is caught by simple deduction, the things it produces are wrong.

  1. Exposing model disagreement? You mean between two disparate models opinions/responses/results? Or simply removing the sycophancy baked into them via the way RHLF and other supervised training inherently cause so that the model will argue with you? I am go the latter, the former is just model bias and training data, find the right model for the right-er answers, its not the comparison to find the best right one, its comparison to eliminate the most wrong one. And then not using it anymore because its unreliable.

  2. I would take the responses and assess and test them in my specific framework or use case against my requirements and specified/expected outcomes, if I cannot assess or test the validity of the response, no matter how right the LLM is, I should probably not be the one signing off on its work in the first place. Yes, LLMs can enable you do to things you might not know how to do on your own, but you, the user, are ultimately responsible for garbage out, which comes from garbage in. In the case of models, if the “garbage” is the models training prior to your use as a user, well, you cannot fix that, but you can test for it, which is literally a whole industry right now. Comparing models efficacy. So you begin then, choose the right model for your job, then you work and test and assess its outputs till they are acceptable to you, and your use case, and your acceptance standards, for which there are no one-size-fits-all solutions.

  3. Yes, A/B testing for your own use case, for your own standards, acceptance criteria and needs. A/B testing models, variants, and prompts and response actions. That is, unfortunately, your job as user, and always has been since that first line of code ever written by a user on a punch card in a mainframe room in the 60s. Can automation run some basic tests, checks and analysis on an idea/output/codebase/whitepaper/experiment report/dataset yeah, sure, as long as the rules for YOUR use case, in YOUR application, and YOUR specific acceptance criteria are… We have all kinds of code linting, unit testing, lookup validation tools and they work here in theory and can be easily created for any situation with the same agent, as this is not so much “decision making” as it is just mapping output against acceptance rules, and that’s just rote data parsing for a machine fundamentally. Verifiable by case specific rule sets.

The signals I would look for from a calculator to know its correct is that it says 2+2=4, not 5. Its provable by math. All things are provable by something that existed before computers did, be it math, biology, astronomy, chemistry, engineering physics, astro physics… the rules are written, all you need is a validation loop on your output. Its called testing. And its been fundamental to software… and well … ANY “solution” ever proposed since the discovery of fire and choosing the best stick in the cave to use as a club against other cavemen.

Bottom line. Your instinct to validate outputs is correct. Your concept of a general purpose tool for it? You already made it, it’s a process, not a discrete machine. Follow your validation steps process you typed above manually, customized to your very specific task and expected outcomes. Accept the most useful answer, and get your task done, if it doesn’t work, improve iteratively. Incremental improvement is how the world has come to be and will continue to evolve. And there will be mistakes. Cost of doing business. Cost of doing life.

5

Q1. If you were redesigning this from scratch, what would you keep as the core product: domain-specific validation, automated acceptance criteria, or something else?

Q2. Would you consider model comparison useful as a secondary signal, provided it is never treated as proof of correctness and the final decision is based on external validation?