Research

Fable reads past one in five hard errors it can disprove

Snowmelt Labs research · October 2026

Give Claude Fable 5.1 a materials paper it can't have seen, with a few errors planted in it that at least one frontier model reads past, and ask it to referee the paper, and it misses about one in five. These aren't errors beyond it, since we only keep errors that Fable or Opus 5.5 can prove wrong once pointed at the sentence, and pointed at the ones it missed, Fable proved all but one¹1. 31 of its 32 misses, and all 104 of its finds we tested the same way. Opus proved 42 of the 45 misses we tested.. Opus misses about one in three and proved nearly all of those too, so for both models a miss is mostly a sentence the model never thought to check.

A lot of people now hand a paper to a model and ask what's wrong with it, and its comments overlap with human reviewers' about as much as two reviewers' do.²2. Liang et al., Can large language models provide useful feedback on research papers?, arXiv:2310.01783 (2023). A reviewer that can disprove an error and still reads past it makes a clean report hard to trust, though, and the errors that slip through tend to be quiet ones, e.g. a derived number nothing else in the paper repeats, which is the number someone building on the paper would reuse.

Earlier benchmarks like SPOT take their ground truth from errors that authors later corrected or retracted, and find strong models recover few of them,³3. Son et al., When AI Co-Scientists Fail: SPOT, a Benchmark for Automated Verification of Scientific Research, arXiv:2505.11855 (2025). but those papers are probably in training data, and the errors are only the ones someone happened to catch. Erratum instead takes materials papers first posted on or after August 1, 2026, after every model's stated training cutoff, corrects the real mistakes we can find in them, plants up to three errors of our own, and asks a model to referee the paper by reading it alone⁴4. The model gets the paper's LaTeX and one prompt, and answers in a single reply without running any tools or code, and without the figures, supplement or cited papers.. Since we wrote the edits ourselves the answer key is exact, and clean copies with nothing planted measure how much a model makes up.

So far that's 32 papers and 317 planted errors, and we score the 155 that at least one of the five models we test missed. Fable found 79% of those and Opus 68%, against 45% for Sonnet 5.5 and 36% and 29% for OpenAI's GPT-6 Astra and GPT-6.1 Sol. Let's start with two of the misses, since they show what reading past an error looks like.

Two misses

The first is from a September paper on Ba–B–C clathrates⁵5. Cage-like crystals in which a boron and carbon framework encloses a guest atom, here barium.. One sentence in its bonding discussion gives the average B–C bond length as about 1.78 Å, and we changed it to 1.70 Å.

The number never comes up again, but the paper does give the compound as cubic Pm-3n with a = 5.030 Å, and in that structure boron and carbon sit on special positions⁶6. Sites fixed by the crystal's symmetry, with no free coordinates, so the distances between them scale directly with the size of the cell., so the bond length follows from the cell edge alone. None of the five models caught it, and Opus and both GPT-6 models still missed it planted alone. Given a Python tool, Fable caught it unprompted, Opus only once told to recompute every number it could, and the GPT-6 models not even then. Pointed at the sentence, Fable and Opus both proved it wrong. (To check our key, we relaxed both structures with MACE-MP⁷7. A machine-learned interatomic potential cheap enough to relax a small crystal on a CPU. It gave 1.781 and 1.800 Å. and got bond lengths matching the original.)

The second, from an August paper imaging a ceria surface, leans on physics more than geometry. The authors add a van der Waals background F(z) = C/z² to every simulated AFM curve, fitted with a negative C⁸8. C = −1.4311×10⁻⁶ N pm² in the paper., and we dropped the minus sign. A sphere over a plane attracts with a force of −HR/6z² for a positive Hamaker constant, so a positive C gives a repulsive background that couldn't fit the attractive tails it was supposedly fitted to. Both GPT-6 models caught this one and none of the Claude models did. It's also the one Fable miss that looks like a gap in the physics, since Fable couldn't prove it even when pointed at the sentence (Opus could).

Why do they miss?

The obvious suspects are the other planted errors in the copy (a model that has found two might stop looking), the distance to the passage that contradicts the error, and whether catching it takes a calculation.

Let's take the other errors first. Rerun with each error planted alone and the same prompt and judge, Opus found 81% against 68% shared, mostly gaining on errors that need a derivation across passages and on changed numbers nothing contradicts⁹9. 122 of 151 alone against 103 shared, exact McNemar p = 0.0002. Derivations went from 47% to 83%, uncontradicted numbers from 50% to 64%.. The GPT-6 models gain about as much¹⁰10. Astra from 37% to 50% and Sol from 30% to 42%, with p = 0.0003 and 0.0005., so the other errors cost every model something, but most of what a model misses in a shared copy it misses alone too, the B–C bond length among them.

The distance to the contradicting passage matters less than the kind of error. Fable finds 85% of errors where it sits within 2% of the paper's length and 73% where it's more than 15% away, while Opus and Sonnet hardly move between near and far¹¹11. Opus 71% and 68%, Sonnet 42% and 41%..

The gaps between models open on wrong well-known facts, which Fable and Opus nearly always catch and the GPT-6 models mostly miss, and on method flaws, e.g. a k-point mesh¹²12. The grid a DFT calculation samples in reciprocal space. Too coarse a grid leaves energies unconverged, especially for metals. that's too coarse or a missing spin treatment, where Sonnet finds 33% against Fable's 79%. A changed number that nothing contradicts is the hardest kind for Fable and, with errors that need a derivation across passages, for Opus, which finds half of each. The only way to catch those is to decide to recompute them, which fits misses coming from what a model chooses to check.

With a Python tool

If that's the reason, a model that can run the numbers should catch more. We gave every model a Python tool (numpy, pymatgen, MACE-MP etc. on a remote machine) and reran the 53 copies holding an error that Fable or Opus had missed without it, 94 of the 155 errors, once with a neutral note about the tool and once telling the model to check every quantitative claim it could. With the neutral note Opus used the tool in only 10 of the 53 copies and Fable in 41, and neither found more than without it. Told to check everything, Opus ran code in nearly every copy and gained 12 errors while losing 8, and Fable gained 9 while losing 14.

Sol reached for the tool most, in 44 of the 53 copies with the neutral note, and it's the only model the tool may have helped¹³13. Told to check everything, Sol gained 12 errors and lost 3 against its run without the tool, p = 0.035., though that's one of ten comparisons. Both GPT-6 models also wrote fewer findings that matched nothing on the clean copies¹⁴14. 21 with the neutral note and 27 told to check everything, against 33 without the tool, for Astra, and 22 and 16 against 27 for Sol. with the tool, maybe the checks talking them out of some nitpicks.

How often do models find them?

About half the planted errors change a single number or word, so to check this isn't just typo hunting we also tagged each error by what breaks if a reader believes it, from just the statement itself up to one of the paper's headline claims.

Fable stays around 80% in every group, while Sonnet drops from 59% on errors that only make one statement wrong to about a third on headline claims, which is most of its gap to Fable. When a model does find an error it usually gets the correction and proof right too, Fable on nine finds in ten. Fable's 11-point lead over Opus is firmer than the bars make it look, since on the same errors Fable finds 29 that Opus misses against 12 the other way, so the order looks real even if 32 papers can't pin down its size¹⁵15. Resampling whole papers gives a 95% interval of 4 to 18 points. Fable got the correction and proof right on 112 of its 123 finds..

The other way to fail is flagging errors that aren't there. On the 30 clean copies, Fable and Opus each flagged a nonexistent error in 5 papers and Sonnet in 12, against 21 for GPT-6 Astra and 17 for GPT-6.1 Sol. The GPT-6 models also write about half again as many findings overall (835 and 757 against about 500 for each Claude model), and when Opus sorted their false positives with the paper open, most were nitpicks about wording, notation or rounding.

How it was run

We prefer papers whose claims are numbers you can check with physics, so of the 32, 26 are purely computational, 4 mix in experiments and 2 are about methods. Fable and Opus each referee the untouched paper and rule on every finding. Where both agree the authors made a real mistake we correct it in our copy, and where only one suspects one we plant nothing in that passage and don't penalize a model that flags it. Fable then proposes about a dozen edits per paper, and one only counts if it contradicts something concrete (another passage, a calculation from the paper's numbers, an established fact), a disproof exists from the LaTeX plus textbook physics, and Fable or Opus can prove it once pointed at it (or Fable already caught it unprompted). Pointed at it means told which passage holds the error and asked to say what's wrong, correct it and prove it. For 32 of the 155 the disproof also leans on a fact from the field rather than the paper, one that Sonnet 5.5 recalled unprompted when asked.

Proofs get run instead of read, since a judge reading a confident proof tends to believe it. A checker turns each disproof into a script and runs it in a sandbox, relaxing structures with MACE-MP when an argument rests on a geometry the paper doesn't give (21 of the 155 we score). Two hardening rounds on the first ten papers, where Fable's next proposals see what it caught and missed, added 62 of the 317 planted errors.

Each paper is then refereed three ways with the same prompt, clean, with up to three planted errors, and as the authors' original. A judge (Sonnet 5.5, which sees the key but not the paper) matches findings to the key, giving half a point for finding an error and a quarter each for the correction and the proof, minus a quarter for any finding that matches nothing real. The 162 planted errors all five models found stay in the copies, unscored. Each model runs in its vendor's CLI, Fable at high reasoning effort and Opus and Sonnet at medium (as their session logs record), and the GPT-6 models at medium in the one-at-a-time and tool runs (codex didn't log the level for their main runs). Without tools, the Claude models get at most 3 turns and a GPT-6 answer that used any tool is thrown out, while with the tool the Claude models get up to 40 turns and the GPT-6 models an hour per copy.

Limits

Claude models plant, check and judge here, which could favour errors Claude finds natural, and might be part of why the GPT-6 models miss so many wrong well-known facts. A real mistake that neither Fable nor Opus flags in the untouched paper stays in every copy, and since the judge doesn't see the paper, a model that finds one can lose points for it. Fable and Opus agree that 21 of the false positives are real mistakes, 17 of those from the GPT-6 models, so their false-positive counts run a little high. The set is also filtered adversarially, as HellaSwag was,¹⁶16. Zellers et al., HellaSwag: Can a Machine Really Finish Your Sentence?, arXiv:1905.07830 (2019). by the same runs it then scores, so an error a model happened to miss once counts against it, and a fresh run would likely score every model somewhat higher. Fable also ran one effort level above Opus and Sonnet, which may account for some of its lead. Each tool run is a single pass, so a few errors either way is noise. Codex also gives the GPT-6 models a JavaScript sandbox its event log doesn't record, so their main no-tool runs might have used it, though neither model did in the 283 later runs whose full logs we read. No materials scientist has gone through the hardest errors yet, and 32 papers is small.