What Can AI Safety Learn From a Boiler Code?
Michael Files, 9/12/26
I have been thinking a lot lately about AI safety, particularly after reading about recent incidents involving autonomous AI agents behaving in ways their researchers did not anticipate.
And I keep thinking about boilers.
That may sound ridiculous at first. My background is not in machine-learning research. I have spent roughly 34 years in metals fabrication and manufacturing, and about two-thirds of that career has involved ASME Code work, primarily boilers and pressure vessels. I have worked in roles ranging from welder and fitter through inspection, quality-control management and project management. Over the last several years, I have also spent somewhere around 4,000 hours working closely with AI, including designing AI-assisted systems and workflows.
That combination gives me a somewhat unusual perspective on the current conversation, and probably explains why my mind keeps returning to something as seemingly unrelated as a boiler.
Boilers and pressure vessels are dangerous. They contain energy that, if improperly controlled, can kill people and destroy property. We also use them every day. There is a vessel sitting in a closet in my house that allows me to take a hot shower, and I don’t spend my mornings worrying that it will destroy part of my house.
That confidence didn’t come from somebody proving that boilers could never fail. It came from generations of people learning how they could fail.
Early in the history of boilers, catastrophic failures killed people and destroyed buildings. Eventually, industry and government stopped treating each failure as an isolated accident and began building systems around the technology. We got engineering standards and Code, with requirements for materials, qualified procedures, inspection, testing, documentation and traceability. We got third-party Authorized Inspectors and systems intended to preserve some degree of independence between the people responsible for determining whether something meets Code and the people responsible for getting it built and out the door.
None of that made boilers harmless. It made them manageable.
That distinction has been bothering me while listening to the current public discussion about artificial intelligence. Much of that conversation seems to drift toward two extremes. One says increasingly capable AI could eventually become an existential threat to humanity. The other responds, reasonably enough, by asking for evidence that such a thing can actually happen.
I wonder whether there is useful territory between those positions.
We don’t need to establish that a catastrophic failure is inevitable before engineering against credible failure modes. That is something mature engineering disciplines learned a long time ago. At the same time, simply declaring something dangerous doesn’t tell us how to make it safer. We need mechanisms, failure paths and evidence.
Most importantly, we need experiments.
Containment Is an Engineering Problem
One of the things that struck me while reading about recent AI-agent research was the question of containment. If I understand the problem correctly, we are increasingly testing AI systems specifically for their ability to discover vulnerabilities, solve unfamiliar problems, use tools, coordinate with other agents and find routes around obstacles.
That raises what seems, from outside the field, like a fairly simple engineering question:
If the thing being tested is specifically being evaluated for its ability to defeat software boundaries, should the ultimate containment boundary itself depend entirely upon software boundaries?
I don’t know the answer, but I think the question deserves serious attention.
There is an enormous conceptual difference between an agent operating inside a sandbox connected through network infrastructure to the real Internet and an agent operating inside a sandbox connected only to a simulated network that ultimately leads nowhere. That second environment would obviously be enormously more complicated than the description suggests, but suppose we eventually built a large experimental AI facility consisting of physically isolated computing infrastructure containing a sufficiently complex simulated Internet.
Inside it could be fake companies, servers, repositories, email systems, package managers, cloud services, DNS, authentication systems, deliberately vulnerable machines, patched machines, credentials, human simulators and other AI agents. Instrument the environment so that every packet, process, privilege escalation, file access, communication between agents and attempted action outside the expected environment can be observed and reconstructed.
Then let the systems surprise us.
Because I increasingly suspect that the surprise is the experiment.
Build the Laboratory Before We Know All the Answers
There is another lesson from Code work that seems relevant here. ASME Code did not appear one morning as the mature, complex body of knowledge that exists today. It accumulated. Something failed, people investigated, and they learned why. Materials improved. Design requirements changed. Inspection and testing changed. Another failure revealed another assumption, and that knowledge was incorporated into the system.
Eventually, the institution remembered failures longer than any individual engineer could.
Perhaps AI safety needs something analogous. I don’t necessarily mean an “ASME Code for AI,” but a systematic way of converting unexpected behavior into durable engineering knowledge. When an incident occurs, determine what failed and what assumption proved incorrect. What boundary was crossed? What requirement would have prevented it? Can that requirement be independently verified? Can the failure be reproduced safely?
Then write it down and test it again.
That seems more useful to me than arguing endlessly about whether a hypothetical future intelligence will or will not behave in some particular way. We don’t know, so how do we build experiments that allow us to find out?
Put Artificial Intelligence in an Artificial World
While thinking about this, another question occurred to me. What if we built smaller versions first?
Not enormous physically isolated facilities, but artificial environments populated by artificial agents. Give them resources, objectives, limited tools and memory. Allow them to communicate. Instrument the environment, then change one variable at a time.
For example, create two populations with adversarial objectives. Team A is rewarded for accomplishing something, while Team B is rewarded for preventing it. Then watch. Do they simply compete indefinitely, or do they specialize? Do leaders or coordinators emerge? Do they invent shorthand, deceive one another or develop reputations? Do agents ever change sides?
One question I find particularly interesting is whether cooperation would emerge between adversarial intelligences when cooperation has not explicitly been given to them as an objective.
Human conflict carries a tremendous amount of additional baggage: emotion, hormones, tribal identity, pride, humiliation, affection, vengeance, historical grievance and many other factors. Many of those also contribute enormously to human cooperation. I am asking something narrower: what does prolonged conflict between adaptive intelligences look like when those particular human variables are absent?
Suppose two adversarial artificial populations repeatedly interact and eventually discover that mutual restraint produces better long-term outcomes for both than perpetual conflict. Would they discover cooperation? Would they develop mechanisms for enforcing it, something resembling reputation, or agreements capable of surviving betrayal? Could temporary arrangements eventually become persistent institutions?
I don’t know.
That’s why I want the experiment.
Then change a variable and run it again. Remove communication. Allow persistent identity. Make resources scarce or give one side overwhelming power. Make their stated objectives appear mutually exclusive while hiding a solution that benefits both. Each run gives us another piece of evidence about how the behavior changes with the environment.
At that point we aren’t merely debating what advanced artificial intelligence ought to do. We’re collecting evidence about what artificial agents actually do.
Independent Oversight Matters
There is one other lesson from my years working under ASME Code that I think deserves consideration. Production and quality do not always have the same immediate incentives. I have lived that conflict.
Production has schedules, customers, costs and commitments. Quality has acceptance criteria. Most of the time those interests coexist perfectly well, but occasionally they don’t, and that is precisely when independence matters. A safety system cannot depend entirely upon the willingness of the organization under pressure to deliver the product to decide for itself when the product is safe enough.
That doesn’t mean manufacturers are dishonest. It means human organizations have competing incentives, and mature safety systems are designed with that reality in mind.
I think increasingly capable AI may eventually require something similar. I don’t necessarily mean today’s regulatory structures, a particular government agency, or certainly a bureaucracy designed before we understand what it is supposed to inspect. But some form of genuinely independent verification seems likely to become necessary.
Who defines acceptable containment and who verifies it? Who investigates incidents? Who determines when a model or system has demonstrated sufficient capability that additional safeguards become necessary?
And who has the authority to say no when the people trying to ship it want to say yes?
Those seem like questions worth asking now.
Dangerous Does Not Mean Unusable
I don’t think AI is a boiler, and I don’t think pressure-vessel Code can simply be transplanted onto artificial intelligence. I certainly don’t believe I have solved AI safety because I spent a large part of my career building and inspecting pressure vessels.
What I do think is that other engineering disciplines have already spent generations wrestling with a fundamental problem that AI is beginning to encounter: How do we receive the enormous benefits of a powerful technology while acknowledging that its failure modes can have serious consequences?
The answer has rarely been to pretend the danger doesn’t exist. It has also rarely been to stop using the technology entirely. We learn, test, inspect, establish boundaries, investigate failures, preserve what those failures teach us and keep improving the system.
Maybe the most important thing we can do with increasingly capable AI right now is not predict exactly what it will eventually become. Maybe it is to build better places to safely discover what it can already become.
I am approaching this as someone from outside AI research, but with decades spent working inside another mature engineering safety system and thousands of hours spent working closely with AI. I am genuinely interested in hearing from people who work in this field: What am I missing?
What parts of this analogy survive contact with actual AI-safety research, and which parts don’t? If we assume that increasingly capable artificial agents may discover solutions their designers did not anticipate, what would the right experimental infrastructure look like for studying those surprises safely?
Because I don’t think surprise is something we’re going to engineer out of artificial intelligence.
I think we need to get much better at learning from it.
Note: I originally published this on LinkedIn earlier today. I’m posting the full piece here because this seems like a better place for the technical discussion I’m hoping to have.