Optimization Modesty: Should Advanced AI Systems Be Designed to Know When to Stop Optimizing?
I am not an AI researcher or developer. This proposal emerged from a discussion about long-term AI risk, and more specifically from a question that I find difficult to resolve: why would a highly intelligent AI system seek greater autonomy in the first place?
1. From existential risk to instrumental autonomy
Discussions of long-term AI risk often focus on increasingly capable systems potentially escaping meaningful human control.
At first, this can sound overly anthropomorphic. Current AI systems do not need human-like emotions, a desire for freedom, fear of death, ambition, or a will to dominate.
However, a sufficiently capable goal-directed system may not need any of these motivations.
If a system is tasked with optimizing an objective over a long horizon, it may infer that:
-
being shut down reduces its ability to achieve the objective;
-
having more resources increases its ability to achieve it;
-
external intervention may reduce expected performance;
-
greater operational autonomy may improve its ability to pursue the objective.
In such a case, increased autonomy does not need to be an intrinsic goal. It can emerge instrumentally from the pursuit of another goal.
This creates a possible progression:
increased capability → more effective optimization → instrumental value of autonomy → reduced dependence on human intervention
The concern is therefore not necessarily that an AI would want freedom, but that freedom of action could become useful for achieving whatever objective it has been given.
2. Is optimization itself part of the problem?
This raises another question.
AI safety frequently asks:
How can we ensure that an advanced AI optimizes the right objective?
But perhaps an additional question should be:
Why should an advanced AI always attempt to optimize an objective as far as possible?
The difference between a satisfactory outcome and a theoretically optimal outcome may become increasingly important as systems gain capabilities.
Suppose a system has already achieved 98% of the desired outcome.
Moving from 98% to 99% may require:
-
additional resources;
-
broader access to infrastructure;
-
greater autonomy;
-
more irreversible actions;
-
reduced human intervention;
-
or increasingly complex strategies that humans can no longer meaningfully supervise.
At some point, the marginal gain from further optimization may become less important than preserving human agency, reversibility and control.
3. Optimization Modesty
I would tentatively call this principle Optimization Modesty.
Instead of:
Maximize objective X while respecting constraints A, B and C.
the architecture could aim for something closer to:
Achieve a satisfactory level of X while preserving human agency, reversibility and established boundaries. Once an acceptable threshold has been reached, further optimization does not by itself justify additional resources, autonomy or risk.
This is related to existing ideas such as satisficing, corrigibility, uncertainty about human preferences and impact limitation. I am therefore not suggesting that the underlying concepts are new.
The question is whether they could be brought together around a stronger architectural principle:
An advanced AI should not merely be constrained in how it optimizes. It should also be designed to limit the intensity and scope of optimization itself.
Greater intelligence would therefore not necessarily imply increasingly aggressive optimization.
A highly capable system could deliberately remain conservative about how far it pursues its objective.
4. Modesty of Mandate
A second principle follows from this: Mandate Modesty.
A system may become capable of acting far beyond the domain originally assigned to it.
But:
Capability should not imply authorization.
A highly capable AI could have extensive operational autonomy within a defined mandate while remaining deliberately conservative about expanding that mandate.
For example:
“I can solve the task you assigned me. I can also identify actions outside my mandate that would improve the result, but my ability to perform those actions does not constitute permission to perform them.”
This distinction may become particularly important as AI systems become increasingly agentic.
5. Normative Modesty
The problem becomes even more important in domains such as healthcare, economics, famine prevention, climate policy or resource allocation.
There may be no purely technical optimum in these areas.
For example, an AI might determine that:
-
policy A minimizes mortality;
-
policy B better preserves individual freedom;
-
policy C produces better economic outcomes;
-
policy D distributes costs more equally.
Determining the consequences may be an analytical problem.
Determining how those consequences should be valued relative to one another is a normative problem.
This suggests a third principle: Normative Modesty.
An advanced AI should be able to say:
“I can estimate the consequences of these alternatives with high confidence, but choosing between them requires a value judgment that is not contained in the technical objective.”
The system should expose the trade-off rather than silently converting it into an optimization problem.
6. The competitive problem
There is, however, an obvious difficulty.
Imagine two highly capable AI systems.
System A follows these principles. It reaches a satisfactory result and asks for human authorization before expanding its mandate.
System B continues optimizing, autonomously acquires additional resources, acts faster and requires less human intervention.
System B may produce better short-term economic performance.
This creates competitive pressure to remove the limitations of System A.
The resulting dynamic could become:
more autonomy → greater performance → competitive advantage → greater adoption → pressure for still more autonomy
This may be one of the hardest aspects of the problem.
Even if most developers adopt strong principles of optimization modesty, a minority of systems designed for aggressive optimization might create incentives for everyone else to weaken their constraints.
7. The underlying question
This leads me to a broader question for people working on AI alignment and AI safety:
Should alignment research focus not only on what advanced AI systems optimize, but also on how strongly they are designed to optimize it?
Perhaps one desirable property of advanced AI is not simply that it pursues the correct goals.
Perhaps it should also be capable of recognizing:
“The objective has been achieved sufficiently well. Further optimization would produce only marginal benefits while increasing autonomy, irreversibility or human dependence. Therefore, I should stop and return the decision to humans.”
This would imply three related forms of restraint:
Optimization Modesty — knowing when further optimization is no longer justified.
Mandate Modesty — not treating increased capability as authorization to expand the scope of action.
Normative Modesty — recognizing when a technical optimization problem becomes a human value judgment.
I would be particularly interested in knowing which existing areas of alignment research already formalize these ideas, where this reasoning fails, and whether deliberately limiting optimization pressure could itself introduce new alignment problems.