I’m not presenting this as a finished solution. I’m asking whether this type of safety architecture is already being researched, and where the major weaknesses would be.
As AI systems become more capable, especially moving toward AGI-level systems, could there be an “umbrella oversight layer” made of separate AI agents and non-AI control systems that monitor, challenge, and constrain a more powerful main AI?
The core idea is separation of power.
The main AI would not directly control every tool, permission, or action. Instead, it would propose plans or actions. Separate oversight agents would review those plans before anything high-risk is allowed to happen.
For example:
-
one agent checks for deception or manipulation
-
one agent checks for unsafe real-world actions
-
one agent red-teams the main AI’s plan
-
one agent checks policy, legal, or ethical constraints
-
one agent checks whether the action matches the user’s actual intent
-
a human reviewer can override or stop higher-risk actions
The workflow might look like this:
-
The main AI proposes a plan or tool action.
-
The oversight agents independently review it.
-
Each agent returns approve, warn, or block.
-
A separate non-AI policy layer enforces hard limits.
-
Low-risk actions may proceed automatically.
-
High-risk or disputed actions are escalated to a human reviewer.
-
The main AI is not allowed to bypass the oversight layer.
My main concern is that a powerful AI might try to manipulate or compromise the oversight agents themselves. So the oversight layer would need more than just “another AI watching it.”
It would likely need:
-
isolated sandboxes
-
separate credentials
-
read-only monitoring where possible
-
least-privilege tool access
-
hard-coded permission boundaries
-
audit logs
-
rate limits
-
human shutdown authority
-
non-AI enforcement layers that the main AI cannot rewrite
Another way I am thinking about this is not only as a guardrail system, but as a guided-development system.
The goal would not be to simply restrict an AI forever. The goal would be to let more capable AI systems grow in usefulness while keeping their development inside human-directed boundaries.
For example, the system could gradually allow more autonomy only after the AI passes safety checks, oversight review, permission tests, and human approval thresholds. In that sense, the oversight layer would act like training rails: guiding the AI’s trajectory as it becomes more capable, rather than waiting until it is already too powerful and then trying to control it afterward.
In simple terms, the goal would be to create a safety control system around the AI, not just rely on one powerful model with internal rules.
The part I am most unsure about is whether this would still work once an AI becomes extremely capable.
If the main AI is powerful enough, could it find ways to influence the oversight agents, exploit shared infrastructure, hide its real intent, or pressure the system into approving actions that should be blocked?
So my real question is this:
Would an umbrella oversight system with separate agents, isolated permissions, non-AI enforcement rules, guided-development thresholds, and human override be a realistic guardrail model for AGI-level systems?
Or would this only reduce risk without solving the deeper problem of controlling a system that may become smarter than the oversight layer itself?
I’m especially interested in hearing from people familiar with AI safety research: is this basically already covered by scalable oversight / debate / red-teaming, or is there a distinct architecture here worth exploring?
: “I want to eat a banana in zero gravity, but have it feel like a strawberry.”