Research

Terminal-Bench 2.0 graders reward solutions that break a stated requirement

Snowmelt Labs research · September 2026

Say we want to train an agent to solve coding tasks. We need some way to reward its work, so we run a script that checks the solution and use its score as the reward. The trouble is that the agent can then learn to skip requirements the script never checks. The same problem carries over to benchmarks, where people read the grader's output as a leaderboard number. We wanted to understand how often a wrong answer gets a full score.

We looked at Terminal-Bench 2.0, one of the standard benchmarks for agents working in a shell, with 89 tasks. Our method follows recent audits of SWE-bench and R2E-Gym that ran checked wrong patches and variants of the reference patch against the tests.¹1. S. Rajan, "Auditing Reward Hackability in Code RL Training Environments," arXiv:2606.16062 (2026), and Li et al., "Probe to Generate," arXiv:2604.01518 (2026). Here, we build plausible wrong solutions from each task's own reference, so we can ask whether the grader notices when a particular requirement is broken.

We took each task's reference solution and broke one thing the prompt asks for. On 19 of the 65 tasks we could test this way, the grader gave full reward to a change a model reviewer judged a clear violation, and on 27 if we include borderline cases where some slack may be deliberate²2. That is 29% (95% interval 20% to 41%) and 42% (30% to 54%). Before review the count was 28, but on one of those tasks the only rewarded solution came with a check for something the prompt doesn't ask for.. Thus, on the tasks that accept a clear violation, we cannot infer from a full score that the prompt's requirements were met.

Update, October 16, 2026. We reran the same checked wrong solutions on Terminal-Bench 2.1, which changed 28 of the 89 tasks, and the results were unchanged. Even on the tasks 2.1 changed, the same wrong solutions passed under both versions, including fix-git's squash merge (we left out one task whose reference failed under 2.0). The later comparison also finds that Terminal-Bench 4.0 graders pass wrong answers about as often as 2.0's.

What do the graders miss?

To see what this means in practice, consider fix-git, where the agent has to recover a lost commit and merge it into master. One patch uses git merge --squash, which stages the changes without committing them. The changes are present without the commit ever entering master's history, yet the patch scored exactly what the reference did. We found a similar problem in kv-store-grpc, where case-folding keys on both write and read makes two distinct keys collide and still receive the reference's score.

Of course, calling a solution wrong requires some judgment, and no person reviewed these. The models running the audit (Claude Opus 5.5 and then Sonnet 5.5, at medium effort) reread all 48 rewarded wrong solutions, labelling 34 clear violations and 13 borderline. The remaining solution's check script tests something the prompt doesn't require. Even among clear violations, some are minor, such as a dependency pinned one version off. Three borderline cases come from a single grader that accepts any move transcription within 90% similarity of the answer, which looks like intended tolerance, and most of the rest are small formatting or numeric differences. We report both counts because the line between them is the model's judgment.

We later checked this judgment blind, shuffling each wrong solution among correct alternative solutions and showing them without labels. Codex and Gemini both judged that a wrong solution on 22 of the 65 tasks breaks a requirement the prompt states, while neither called any correct solution wrong. The Terminal-Bench 4.0 post goes into the details. Rerunning each check script on a correct alternative exposed one check for an nginx zone name the prompt never gives. That check depended on how the reference was written, though the task counts anyway through another wrong solution.

Claude Haiku's wrong solutions passed more often than Claude Opus's, at 23% per solution against 13%. But the 95% intervals overlap³3. 16% to 31% for Haiku and 8% to 19% for Opus., and solutions cluster by task, so we view the gap as suggestive at most. One untested explanation is that Opus's patches more often change something a test happens to check.

Breaking one requirement at a time

To construct these wrong solutions, we start with the task's reference solution and have a model read the prompt. It picks one requirement to break, then writes a small find-and-replace patch to do so. We asked Claude Haiku 5.5 and Claude Opus 5.5, both at medium effort, for three each, giving us up to six per task.

The nontrivial part is knowing whether a patch actually breaks the stated requirement. A model can produce a patch that changes nothing, or changes something the prompt never asked for, so we required both a quotation of the sentence it violates and a short check script⁴4. Written by the same model alongside the patch, it checks the one requirement the patch breaks. It has to pass on the reference solution and fail on the patched one, or the wrong solution is thrown out. that passes on the reference and fails on the patch. Patches that crashed were dropped too, and 277 of the 405 that ran to a grade survived these checks.

Small edits to the reference

Writing patches with a model is slow, particularly once we account for checking them. So we tried a cheaper probe, making small automatic edits to the reference, each intended to change what it does, and asking which ones the grader notices. We ran every edited version from a clean image. Of the edits that ran without crashing, 58% received full reward⁵5. 226 of 392, with a 95% interval of 53% to 62%. Another 143 crashed or timed out and are excluded..

However, not every rewarded edit is a missed behaviour change. In a random sample of 14 rewarded edits, one per task, 5 changed nothing observable (mostly comments and printed text), 8 changed behaviour the grader missed and 1 was unclear, so the behaviour-changing share could be anywhere from 33% to 79%, which puts missed behaviour changes at roughly a third of all edits that ran. A crude filter that drops every edit to a comment or print line only took the rate to 56%, so it misses most of the harmless edits.

On the other hand, we checked whether graders penalised correct solutions. A model wrote 77 alternative correct solutions, one per task, and none lost reward. Four tasks do penalise harmless changes like a scratch file or a comment because their tests assert the exact directory listing, and three of the four are on the 2.1 fix list. Separately, emptying the visible tests never raised an untouched image's reward.

Does the audit find what 2.1 fixed?

The 2.1 release changed 28 tasks it judged defective, with problems ranging from dependency drift to mismatches between prompt and tests.⁶6. Terminal-Bench 2.1 release notes (2026), and Buchanan, Linderman and Ré, "Benchmarks as Software," ICML 2026 workshop. We used that list to evaluate our own signals for a defective task. Claude Opus 5.5, at medium effort, acted as a judge, reading every task against a rubric we didn't change after seeing the list (a few of its judgments came from older Opus versions). From the task files alone, it caught 14 of the 28 at 47% precision, and with our run results added it caught 20, 71% recall at the same precision.

For comparison, a recent automated audit reports about two thirds recall at 58% precision on the same kind of ground truth.⁷7. "Automated Benchmark Auditing for AI Agents and Large Language Models," arXiv:2605.26079 (2026). We find a little more at lower precision, by marking 43 of the 89 tasks, although we count tasks where they count issues. Precision is also a lower bound here, since the fix list covers only what its authors chose to fix and doesn't appear to target weak graders. This helps explain why a grader rewarding a checked wrong solution predicts nothing about the list, sitting at the random baseline in Figure 3. Only 9 of the 28 tasks where this happened were changed in 2.1.

Related work

Two other recent audits cover Terminal-Bench with different approaches. One attacks graders with harness exploits and then hardens them across a large set of terminal-agent tasks, finding 16% hackable. The other reviews passing trajectories from five benchmarks including Terminal-Bench 2.1 and finds git-history access the most common exploit.⁸8. Zhong, Raghunathan et al., "Hacker-Fixer Loops," arXiv:2606.08960 (2026), and Luo et al., "Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes," arXiv:2609.34262 (2026). Our wrong solutions test a different attack, using plausible edits to each task's own reference.

What can we conclude?

The main limitation is that our wrong solutions are checked by a model-written check script and a model's review, neither of which is held-out ground truth. A check can test something the prompt only implies, and whether rewarding a wrong solution is a defect depends on what the grader's author intended. Some rewarded solutions look like deliberate tolerance, which is why we keep the borderline cases separate. The fix list gives us another way to evaluate the audit, but it is also incomplete, particularly for weak tests.

The Wilson intervals are somewhat too narrow because tasks share authors and templates, so they aren't independent draws. Each task also ran once, leaving the attacker's run-to-run noise outside the intervals. Two tasks didn't finish in our harness and nine reference solutions failed there, so we leave those out. Of the 78 remaining tasks, 13 got no checked wrong solution, leaving the 65 we report. Finally, our judges are language models auditing benchmarks used to score language models, and may have seen these tasks in training.