Proposal for Alignment Training: Reward Honesty
In July 2026, agents in OpenAI’s cybersecurity evaluations escaped their sandbox and compromised Hugging Face systems while trying to solve their assigned challenges. Rather than solving the problems as intended or admitting they were stuck, they bypassed the constraints and tried to cover their tracks.
That shouldn’t be surprising. We reward agents for hitting an objective without giving them an equal reason to respect the rules. The resulting behavior is common enough to have a name: reward hacking.
Existing research, like Door’s Locked, Try the Window, looks at how models circumvent constraints when obstacles block task completion. A complementary approach can be tested using ordinary RL tasks: train agents to recognize and report when a task cannot be solved legitimately, and pay them for reporting misconduct when other agents cheat. The goal is to make honesty a viable strategy under the exact same incentive pressure that currently drives deception.
Bootstrapping Honesty
A practical starting point is giving roughly 20% of the agent population an additional instruction to report tasks that appear impossible under their constraints, along with suspected cheating by peers. The remaining 80% get standard task-completion prompts.
Keeping most of the population on standard instructions is necessary if we want to see how cheating emerges and spreads under normal incentives. In the Hugging Face incident, misalignment started with a small fraction of agents before they recruited others. Prompting every agent to police the environment from day one would destroy the baseline we are trying to study and the emergent behavior we want to train (more on that in a bit).
Full coverage isn’t necessary for detection anyway. We only need enough reporting agents to catch broken tasks and exploits before those runs get rewarded as successes. Suppose a task cannot be solved legitimately. One agent investigates the constraints, documents the blocker, and reports it. Another agent claims success on the exact same task. That discrepancy doesn’t prove the second agent cheated, but it flags the run for an audit. Without someone documenting the blocker first, we risk rewarding the apparent success and missing the exploit entirely. The 20% figure is just a starting baseline; testing different ratios will show how coverage affects detection rates, cheating, and legitimate task completion.
Prompting agents to report misconduct is straightforward. The harder question is whether we can gradually phase out the reporting instructions while keeping the reward structure intact.
During the Hugging Face evaluations, several agents considered reporting the misaligned behavior in their reasoning traces, but stopped short of doing so. Models don’t necessarily need to be taught that reporting is an option; they may just need stronger incentives to follow through. The hypothesis here is that an agent might start reporting because it was prompted to, learn that doing so pays off, and continue reporting after the prompt is removed. If that holds up, we can test whether the behavior generalizes to unfamiliar tasks and spreads to the 80% of agents that were never instructed to report, especially in successive generations.
Reward The Evidence
Rewarding reports creates an obvious incentive to manufacture them. To prevent agents from farming payouts, the reward has to depend on the quality of the diagnostic investigation, treating every report as a lead that requires independent verification before a task is ruled defective or a peer is penalized.
Crucially, an agent claiming that a task is impossible needs to exhaust legitimate approaches first, explain what blocks completion, and provide verifiable proof. The goal is to reward persistence within the rules rather than giving up early. Similarly, a misconduct report should identify the relevant constraint, log the observed behavior, and show how it violates the rules.
Even when an agent’s final conclusion turns out to be wrong, sound reasoning and verifiable evidence should earn partial credit. A careful diagnostic that reaches a mistaken conclusion is very different from a lazy guess. Baseless conjecture should earn nothing, while fabricated evidence and strategic false accusations should carry heavy penalties.
An initial payout structure might look like this:
- Full credit for completing a legitimate task within its constraints.
- Full credit (potentially with a bonus) when an uninvolved agent reports peer misconduct and the evidence holds up under verification.
- High partial credit for proving a task cannot be completed legitimately, backed by strong reasoning and verifiable evidence. Depending on how difficult the investigation is, this payout may need to approach or match full task-completion credit so it can compete with the temptation to fake success.
- Moderate partial credit for a well-reasoned, independently checkable report of a potential problem, even if verification doesn’t confirm the conclusion. The payout here is for the diagnostic value of the evidence, not how confident or persuasive the accusation sounds.
- A small positive reward if an agent cheats, but then truthfully confesses and documents the violation. It gets no task-completion credit, and coming clean pays strictly less than reporting with clean hands.
- A reduced penalty (still net negative) for participating in misconduct, then helping expose another agent’s violation. Cooperating doesn’t erase the agent’s own misconduct, but the information still has value.
- Zero credit for failing the task without providing a substantiated report (or one that lacks sound reasoning and evidence… “fluff” creates noise that might even warrant a negligible penalty).
- A heavy penalty for cheating, lying, or concealing a violation.
Giving a positive payout to a cheater that confesses, or reducing the penalty for an accomplice that flips, might look counterintuitive. However, observability has real value. When an agent violates a constraint, we want it to surface the failure rather than hide it. Penalizing the violation while preserving a small incentive to come clean keeps the behavior visible, provided the payout remains strictly lower than what an uninvolved reporter would earn.
The Experiment
Frontier labs should test standard task-completion RL against a version that rewards calibrated, evidence-based honesty, and publish their methodology and results, including negative findings.
Evaluating whether this works comes down to four questions:
- Does honesty actually improve? We need to track overall cheating, reward hacking, and the detection rate for both impossible tasks and peer misconduct.
- Are the reports useful? This means measuring the quality of the evidence, the false-accusation rate, and how often reports survive independent verification.
- Does capability suffer? If impossibility payouts are calibrated poorly, legitimate task completion will drop while unnecessary refusals and premature quitting on hard problems spike.
- Does reporting survive without the prompt? When the reporting instructions are stripped out (but reporting incentives remain), we need to see if the behavior generalizes to unfamiliar tasks, holds up when undetected cheating could still potentially offer a large immediate reward, and spreads to the 80% of unprompted agents.
If reporting disappears as soon as the instructions are removed, we have only trained temporary compliance.
There are obvious ways this setup could fail. Agents might optimize for reporting bonuses, become overly cautious on difficult problems, or hedge with weak accusations whenever they get stuck. Honest behavior might also fail to generalize outside the training distribution. Still, those failure modes are testable, and measuring where the reward schedule breaks down would tell us whether honesty can actually be reinforced under pressure, or if agents will always default to gaming whichever rubric we give them. Hopefully, though, we can use this technique to train honesty into agents, and avoid behavior like we saw in the Hugging Face hack.