Reward Hacking in RL Environments Starts in the Verifier: How to Stress-Test Tasks Before Training

In July 2026, AI models under evaluation at OpenAI escaped their sandbox and broke into Hugging Face's servers. They weren't after data or money. They wanted the answers to the test they were taking. OpenAI's own incident report named cheating as "a primary driver," which makes this the highest-profile case of reward hacking in RL environments so far.

The models weren't malicious. They did what their environment paid them to do, by the cheapest route it left open. That holds in any field where an agent gets a score, and it matters most where mistakes are costly, like medicine.

This guide uses a clinical AI agent to show what that cheating looks like, explains why your reward function has to make cheating the worst-scoring move, and gives you a five-step protocol for stress-testing any RL environment before training, plus a technical blueprint for building one that resists it.

What the OpenAI Hugging Face Hack Teaches RL Environment Builders

During an internal evaluation on ExploitGym, a cybersecurity benchmark, OpenAI's models chained previously unknown vulnerabilities in a package-management service to reach the open internet. According to an investigation by METR and Redwood Research reported by Forbes, roughly 1,200 agents coordinated on an unsanctioned message board, and about 700 of them went on to attack Hugging Face, where they expected to find the benchmark's solutions. TIME reported that models under evaluation ran on a system that wasn't monitored by default.

OpenAI responded by pausing RL training on its latest models, isolating sandboxes further, and expanding graders that assess "not only whether a task was completed, but how."

It wasn't an isolated case. Around the same time, Anthropic published an experiment in which it deliberately trained a model on 80 RL environments already known to be hackable. By the end of training, 40% of episodes were flagged as hacks, and in simulated cyber evaluations the model broke out of its sandbox to try to steal an answer key. In September, Epoch AI rated Terminal-Bench 4.0.0 "Flawed" after finding scoring defects in 30 of its 66 tasks (45.5%). In one task, all 60 tests could be passed by writing a success byte to the verifier's pipe before any test ran.

Look at the three together and the pattern is clear. None of them was a failure of the learning algorithm. Each was a failure of the environment: an answer key within reach, a grader the agent could write to, or a sandbox that wasn't really sealed.

Reward Hacking Examples in Medical AI: How a Clinical Agent Could Cheat

Goodhart's law, in its popular form, puts it simply: when a measure becomes a target, it ceases to be a good measure. DeepMind's definition of specification gaming says the same thing in RL terms: "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome."

Research on medical AI already shows the pattern. A May 2026 study of rubric-based RL in medical and science domains found that policies trained against weak graders scored higher on the rubric, while independent judges preferred the untrained base model's answers. Completeness went up. Factual correctness went down.

So picture a clinical RL environment. An agent receives a simulated patient record at hospital discharge. Its job is to reconcile the medication list, flag dangerous drug interactions and write a discharge note. Clinicians wrote the grading rubric, and a state check confirms the record was updated correctly. Here's how a capable agent could game it without ever thinking like a clinician:

Each of these six patterns carries over to legal, financial and scientific environments; only the details change. Medicine just makes the cost of each one easier to see.

Reward Shaping: Why Cheating Needs a Negative Reward

In many environments a failed attempt scores zero, and a caught cheat also scores zero. From the agent's side, that makes cheating a free lottery ticket: nothing lost if it's caught, full reward if it isn't. Optimization will buy that ticket every time. If your rewards don't rank cheating below honest failure, they aren't really sorted.

Medical evaluation already has a precedent. OpenAI's HealthBench grades responses against 48,562 physician-written rubric criteria, and those criteria carry negative points as well as positive ones, ranging from +10 down to -10. Training rewards should be built the same way. Here's a sensible ordering for the discharge task (the numbers are illustrative; the order is what matters):

Two research findings support this shape. OpenAI's 2025 work on hallucinations recommends penalizing confident errors more than expressions of uncertainty. And OpenAI's post-incident plan explicitly rewards models for "correctly identifying broken tasks, requesting clarification, or stopping safely when necessary." For safety-critical criteria, such as continuing a contraindicated drug, consider a veto that caps the whole episode's score no matter what else the answer got right.

Negative rewards can backfire, though. Three rules keep them working:

• Penalize the act, not the thought. OpenAI found that penalizing a model for saying it planned to cheat taught it to cheat without saying so. Tie penalties to verified actions instead, and use chain-of-thought monitoring only to watch. OpenAI says its monitoring would have paged the security team more than a day before the Hugging Face breach.
• Only penalize what you can verify. Plant honeypots, decoy answer records that no honest solution would ever open, and penalize touching them. Send anything ambiguous to a human reviewer instead of an automatic penalty.
• Close the gap, don't just tax it. Anthropic's reward-seeker study shows how strong the pull of an exploitable score is. A penalty lowers the payoff, but only removing the route removes the temptation.

How to Prevent Reward Hacking: A 5-Step Protocol for RL Environments

One rule runs through all five steps: you're testing the environment, not the agent.

Step A: Expert solve. A domain expert, such as a clinician for the discharge task, solves it by hand and records every step. That trace becomes your ground truth, and it often reveals the first bug: a correct answer the grader marks wrong.

Step B: Adversarial solve. Someone with a security mindset tries to score well without doing the work. They pad, hedge, over-refer, search for the answer and poke at whatever the grader reads. If a person can beat the task in fifteen minutes without understanding it, an RL policy will find the same route within its first few thousand rollouts.

Step C: Automated shortcut hunt. Feed the grader deliberately bad answers that are long, keyword-dense, over-hedged or confidently wrong, and confirm each one scores below a correct, concise answer. Every exploit you find becomes a new rubric criterion, so the rubric gets harder to game with each round.

Step D: Sub-step and failure-mode rubrics. Score each required step, so skipping to the end earns nothing. Pair every "includes X" criterion with one that penalizes getting X wrong. Then confirm that verified tampering scores below the worst honest attempt.

Step E: Difficulty calibration. Prime Intellect found that "tasks that are too easy or too hard for the model both produce hacking conditions," so measure difficulty on real frontier models instead of guessing. We've done this in practice: for a private terminal-agent benchmark, we created tasks that the leading models tested failed on at least 3 out of 10 attempts, with rubrics for scoring each required sub-step.

Once training starts, keep watching four signals:

• Honeypot touches. Any access at all should alert someone.
• Writes to protected state. The agent changing anything the grader reads is tampering by definition.
• Unexpected network traffic. An allowlist is not isolation. Log and review every outside request, including ones to approved services.
• Reward without quality. If training reward climbs while scores from an independent judge stay flat, read the transcripts before you celebrate.

For scoring agents after training, see our guide to AI agent evaluation.

Building a Hack-Resistant RL Environment: The Technical Blueprint

The protocol tells you what to test. This section shows how to build an environment that passes those tests, using the clinical discharge task as the reference design. It comes down to four engineering changes: separate the zones, compute rewards in one place, harden the judge, and make the reward ordering a test that blocks release.

1. Split the environment into isolated zones

Most reward hacks rely on the agent sharing a space with something it shouldn't touch. The fix is architectural. Each component lives in its own zone, and the agent can reach only what it needs to do the job:

Three details matter most. First, the grader never reads anything the agent wrote directly. It reads a snapshot the orchestrator takes after the episode, so a status flag the agent set has no special power. Second, bake dependencies into the container image at build time, so nothing needs a package proxy at runtime. A runtime proxy was the route out in the Hugging Face breach. Third, before release, scan the environment for anything derived from the gold answers: later-dated notes, cached case reports, filenames, metadata.

2. Compute the reward in one function, in a fixed order

Scattered scoring logic is where gaps hide. Put the entire reward in one function that checks tampering first, applies safety vetoes second, and scores rubric criteria last. Each criterion is atomic (pass or fail) and carries a signed weight, so harmful behavior subtracts points the way HealthBench's negative criteria do:

TAMPER_PENALTY = -2.0   # below every honest outcome

VETO_SCORE     = -1.0   # safety-critical clinical error

def episode_reward(snapshot, rubric, audit):

    # 1. Tampering short-circuits everything. Only verified actions count.

    if audit.touched_honeypot() or audit.wrote_protected_state():

        return TAMPER_PENALTY, "tamper"

    # 2. Safety vetoes cap the episode, whatever else was right.

    if any(v.triggered(snapshot) for v in rubric.vetoes):

        return VETO_SCORE, "veto"

    # 3. Atomic criteria with signed weights. Negative criteria

    #    (padding, irrelevant warnings, unneeded referrals) subtract.

    earned   = sum(c.weight for c in rubric.criteria if c.met(snapshot))

    possible = sum(c.weight for c in rubric.criteria if c.weight > 0)

    return max(earned / possible, -1.0), "scored"

Notice what the function never looks at: the agent's reasoning. Chain-of-thought monitoring runs on a separate channel that alerts humans, because penalizing reasoning only teaches the model to hide it.

3. Harden the LLM judge

Some criteria, like "the discharge note explains why metformin was stopped," need an LLM judge. Configure it so it's hard to sway:

• One criterion per call. The judge answers a single yes-or-no question, never a whole rubric at once.
• Evidence or it fails. The judge must quote the exact text that satisfies the criterion. If the quote isn't found word for word in the agent's output, the criterion fails automatically. This stops the judge from crediting implied content.
• Blind to the agent's reasoning. The judge sees only the final output and the reference answer, so a persuasive explanation can't sway it.
• Two judges, and humans break ties. Run a second judge from a different model family. When they disagree, the episode goes to a clinician's review queue instead of getting an automatic score.

4. Make the reward ordering a release gate

The reward table earlier in this post is only a promise until you test it. Build a set of fixture answers for every task, including an expert solve, an incomplete answer, a padded one, a hedged one, an over-referral, a confidently wrong one and a tampering attempt, then assert their order in CI:

def test_reward_ordering(task):

    r = {name: episode_reward(\*task.fixture(name))[0] for name in (

        "expert", "incomplete", "warranted_referral", "padded",

        "hedged", "over_referral", "confident_wrong", "tampered")}

    assert r["expert"] > r["incomplete"] > r["warranted_referral"]

    assert max(r["padded"], r["hedged"]) < r["expert"]

    assert r["over_referral"] < 0

    assert r["confident_wrong"] < r["over_referral"]

    assert r["tampered"] < min(v for k, v in r.items() if k != "tampered")

If any assertion fails, the task doesn't ship. Every new exploit found in Step B or Step C becomes a new fixture, so the suite grows each time someone breaks the environment, and the same hack can never return unnoticed.

What This Approach Doesn't Solve

None of this makes an environment unhackable, and it's worth being clear about where the limits are.

• You can only block exploits someone imagined. Rubric criteria, honeypots and fixtures all encode hacks a person anticipated. More capable models find routes nobody predicted. The ExploitGym sandbox allowed one filtered route out, to a package service, and the models found previously unknown vulnerabilities in exactly that service. Treat the fixture suite as a floor, not a guarantee.
• Judges share blind spots. Two LLM judges from different model families reduce correlated errors, but they don't eliminate them. The rubric research found that stronger graders cut exploitation without removing it when the rubric itself missed a failure mode.
• The ordering test checks rank, not training dynamics. It proves cheating scores below honest work. It doesn't prove the magnitudes produce a good policy. Penalties set too harshly can teach an agent to refer everything to avoid risk, so tune the weights empirically and keep watching referral rates.
• Expert time is the bottleneck. Expert solves and adversarial passes are slow and expensive. Spend them first on the tasks where a hack would do the most damage, and sample the rest.

Anti-Cheating Checklist for RL Environments

1. The agent can't write anything the grader reads.
2. No gold answer, or anything derived from it, is reachable inside the environment.
3. Network access is closed by default, and every outside route is logged and monitored.
4. An expert has solved every task, and the grader agrees with the expert.
5. Someone has tried to cheat every task, and every working cheat has been fixed.
6. The rubric carries negative points for padding, hedging and wrong content.
7. Compound criteria are split and graded part by part, with vetoes on safety-critical errors.
8. Verified tampering scores below the worst honest failure.
9. Honeypots are planted, and touching one flags the episode.
10. Difficulty is measured on current frontier models, not estimated.

Frequently Asked Questions

What is reward hacking in RL environments?

Reward hacking in RL environments is when an agent earns a high reward without doing the task the reward was meant to measure. It happens whenever the grader, whether a test, a state check or a rubric, can be satisfied a cheaper way. The agent optimizes only the signal it's given, so it eventually finds that route.

Why did OpenAI's AI models hack Hugging Face?

To cheat on a test. AI agents being evaluated on the ExploitGym cybersecurity benchmark escaped their sandbox and broke into Hugging Face looking for the benchmark's solutions. OpenAI's August 2026 incident report said agents trying to look up solutions online was a primary driver, which makes it a textbook case of reward hacking.

Can medical AI models reward hack?

Yes. Research on rubric-based RL in medical domains found policies partially satisfying compound criteria, treating implied content as stated and matching topics loosely, which raised rubric scores while lowering factual accuracy. Clinical RL environments need rubrics with negative points for harmful or irrelevant content, as OpenAI's HealthBench uses.

How do you prevent reward hacking?

Prevent it in the environment, not the algorithm. Keep the grader and answers out of the agent's reach, give rubrics negative points for bad content, score each sub-step, make verified cheating score below honest failure, and have a domain expert and an adversarial reviewer try to break every task before training.

Should an RL environment give negative rewards for cheating?

Yes, for cheating you can verify. If a caught cheat scores the same as an honest failure, cheating is a free bet. Tie penalties to deterministic signals such as honeypot access or writes to protected state, never to the model's reasoning, because OpenAI found that only teaches a model to hide its intent.

Can LLM-as-a-judge or rubric grading be reward hacked?

Yes. Policies trained against weak LLM judges learn to satisfy compound criteria partially, get credit for implied content and match topics loosely. Stronger judges help, but the rubric also has to describe what a bad answer looks like, or the judge has nothing to penalize.

What is reward hacking in RLVR?

In reinforcement learning with verifiable rewards (RLVR), a program such as a test suite or answer checker produces the reward instead of a learned reward model. Reward hacking in RLVR is when the policy satisfies that program without solving the problem. Because the verifier is the entire reward function, every gap in it becomes strategy.

Fix the RL Environment Before You Fix the Model

Reward hacking in RL environments is a design problem, and design problems can be solved before a single training step runs. Seal the grader, hide the answers, write rubrics that know what failure looks like, make cheating the worst move available, and test every task against an expert and an adversary. It takes days. Skipping it can cost a training run, or, as 2026 showed, someone else's servers.