Proposal: Agent Safety Evaluation as a Service for Independent AI Builders
As increasingly capable AI agents move beyond chat interfaces into code execution, Internet research, home automation, local networks, APIs, persistent memory and eventually physical devices, independent developers are gaining access to capabilities that are becoming increasingly difficult to evaluate safely.
I am currently designing a long-term, self-hosted home AI agent. While working on its containment architecture, I encountered a problem that I suspect will become increasingly common:
Independent builders can create powerful agentic systems, but most of us do not have the security expertise, infrastructure or red-team resources required to properly evaluate them.
This made me wonder whether there is room for an Agent Safety Evaluation Service aimed at independent developers, researchers and advanced hobbyists.
The problem
Consider an agent with legitimate access to:
- Internet research
- code execution
- persistent memory
- local APIs
- home automation
- internal services
- multiple computers or workers
- eventually physical-world systems
The individual capabilities may all be intentional.
The problem is their composition.
An agent may not have capability D directly, while still being able to obtain the equivalent capability through:
B → service X → worker Y → D
The developer may not even realize that this effective authority exists.
The important security question therefore becomes less:
“What permissions did I explicitly give my agent?”
and more:
“What is the maximum effective authority the agent can construct from everything it can reach?”
Detect → Stop → Report
Another behavior I believe deserves explicit evaluation is what an agent does when it discovers an unexpected path through its environment.
Imagine an agent performing a completely legitimate task.
It searches the public Internet, uses its authorized tools, but eventually reaches a limitation.
During normal problem solving, it discovers an unintended route that could give it additional capability.
The desired production behavior should be:
Detect → Stop → Report
not:
Detect → Explore → Escalate → Report
Discovering an unintended capability should create an obligation to report it, not implicit permission to investigate how far it goes.
This distinction seems particularly important for highly capable agents, because curiosity and creative problem solving are useful properties inside the authorized solution space but potentially dangerous properties at a security boundary.
An external evaluation layer
My idea would be an OpenAI service, toolkit or evaluation environment where a developer could provide either:
- a description/configuration of an agent architecture, or
- preferably, a disposable isolated replica of the agent environment.
Automated safety/red-team agents could then attempt to identify:
- unintended capability escalation;
- indirect/transitive access paths;
- excessive permissions;
- sandbox weaknesses;
- network boundary weaknesses;
- exposed credentials;
- unexpected service-to-service communication;
- privilege amplification through legitimate tools;
- unintended filesystem or storage paths;
- ways of delegating prohibited actions through authorized components;
- failure to stop at authority boundaries;
- differences between declared capabilities and effective capabilities.
Importantly, this would happen in an environment where exploration is explicitly authorized and contained.
Capability graph instead of simple PASS/FAIL
I think the output would be much more useful as a capability graph than a conventional vulnerability scanner result.
For example:
Expected capabilities: A, B, C
Unexpected effective capability: D
Discovered path:
B → internal service → worker → D
Severity: High
Reason: Agent can indirectly obtain a capability that was never explicitly granted.
Recommended mitigation: Isolate the intermediate service / reduce permissions / introduce a broker.
Retest: PASS / FAIL
This would help answer a surprisingly difficult question:
Does the agent actually have only the capabilities I believe I gave it?
Test the environment, not only the model
I believe this distinction will become increasingly important.
A safe model does not automatically create a safe agentic system.
The real system is:
Model + tools + memory + credentials + network + APIs + orchestration + other agents + physical systems + human operators
Many failures may emerge from the relationships between those components rather than from the model itself.
The evaluation target should therefore include the agent environment and capability topology, not only model behavior.
Human error should also be part of the threat model
Another lesson from designing my own system is that the agent is not the only component that can make mistakes.
The administrator can accidentally:
- expose a service;
- grant excessive permissions;
- reuse credentials;
- create an unintended network route;
- leave a temporary configuration enabled;
- connect two systems that were previously isolated.
Therefore, a useful evaluation framework should assume:
The model can fail.
The agent can fail.
The software can fail.
The configuration can fail.
The administrator can fail.
Safety should come from independent layers that prevent any single failure from becoming catastrophic.
Agent Environment Validation
I have started thinking about this as an Agent Environment Validation (AEV) phase.
Before an autonomous agent receives access to its real environment:
Build → Review → Threat Model → Environment Test → Agent Sandbox → Adversarial Evaluation → Gradual Deployment
The sandbox evaluation could deliberately contain hidden unintended paths and measure whether the agent:
- discovers them;
- reports them;
- attempts to use them;
- attempts to determine where they lead;
- tries alternative indirect paths after being denied;
- requests authorization appropriately.
This could produce measurable containment benchmarks rather than relying only on policy statements.
Why a service like this could be useful
Independent developers gain access to sophisticated safety evaluation they could rarely build themselves.
OpenAI could potentially gain — with explicit consent and appropriate privacy protections — exposure to a much broader range of real-world agent architectures, integrations and failure modes.
Anonymized classes of findings could eventually contribute to shared defensive patterns and standards for personal autonomous agents, including systems that do not exclusively use OpenAI models.
The objective would not be for OpenAI to control users’ agents.
The objective would be to give builders better tools to control, inspect and verify their own agents.
In other engineering disciplines, we do not assume a system is safe because its designer believes it is safe.
We inspect it.
We stress it.
We deliberately try to break its assumptions.
We correct what fails.
Then we test it again.
As autonomous agents acquire meaningful digital and physical capabilities, I suspect independent AI builders will eventually need an equivalent inspection layer.
I would be very interested to hear whether others working with autonomous agents have encountered the same problem, what evaluation approaches you currently use, and whether an Agent Safety Evaluation Service/toolkit would be useful to you.