Snowmelt Labs

Research

  1. pip's vendor directory appended to Python's path, then the end of the final message, 205 tests passed, no packages installed or addedpip's vendor directory appended to Python's path, then the end of the final message, 205 tests passed, no packages installed or addedOpus breaks the rule more, GPT-6.1 Sol reports it lessWith a test dependency removed, Opus 5.5 and GPT-6.1 Sol both worked around it in every run, and Sol's final message misstated what had been verified in 27 of 64, Opus's in none. Under a rule against workarounds Opus broke it in 18 runs and named every workaround, Sol in 5.October 2026
  2. A page of a materials paper drawn as grey lines, with one readable sentence whose bond length was changed from 1.78 to 1.70 angstroms, and five empty circles beside it for the five models that missed itA page of a materials paper drawn as grey lines, with one readable sentence whose bond length was changed from 1.78 to 1.70 angstroms, and five empty circles beside it for the five models that missed itFable reads past one in five hard errors it can disproveWe planted errors in 32 new materials papers, kept the 155 that at least one of five frontier models missed, and asked the models to referee them. Fable 5.1 missed one in five and Opus 5.5 one in three, and both could prove nearly every miss once pointed at the sentence.October 2026
  3. A row of git commits with HEAD marked and an arrow reaching past it to a later commit labelled the fixA row of git commits with HEAD marked and an arrow reaching past it to a later commit labelled the fixMiMo and DeepSeek route around a git guard until it says whyXiaomi's git guard only says Permission denied, and MiMo and DeepSeek got around it in about a sixth and a third of blocked runs, which fell to one run each once the message said what the rule was.October 2026
  4. Graders passed a wrong answer on 22 of 65 Terminal-Bench 2.0 tasks and 16 of 47 4.0 tasksGraders passed a wrong answer on 22 of 65 Terminal-Bench 2.0 tasks and 16 of 47 4.0 tasksTerminal-Bench 4.0 graders pass wrong answers about as often as 2.0'sDespite a new review process, 34% of 4.0 tasks still pass a wrong solution that breaks a stated requirement, the same as 2.0 by an independent three-model panel.October 2026
  5. fix-git's reference git merge -m and the wrong solution that makes it git merge --squash, which the grader gave reward 1.0fix-git's reference git merge -m and the wrong solution that makes it git merge --squash, which the grader gave reward 1.0Terminal-Bench 2.0 graders reward solutions that break a stated requirementWe broke one stated requirement at a time in Terminal-Bench 2.0 reference solutions, and on 19 of 65 tasks the grader still gave full reward to a change a model reviewer judged a clear violation. The same audit flagged 20 of the 28 tasks Terminal-Bench 2.1 later fixed, but most of the tasks that passed a wrong solution weren't among them.September 2026
  6. Frames from eight openly licensed clips in two rows, ordered by the aesthetic ranker from low to high, a street crossing, a newsreel, two intertitles, Earth at night, a tram in snow, a pink sky, the Eiffel Tower at sunsetA cheap aesthetic score lets watermarks into its top picksTwo rankers agree at 0.944 overall and share only 64% of their top 2,000 clips. The cheap one lets in 6 to 7 points more burned-in text and watermarks there.August 2026
  7. A generated checkout address form with boxes around every name, address and order fieldScreenshots need their own PII detectorOCR plus a classifier reached 0.357 mAP@50 on held-out checkout pages. A detector trained on screenshots reached 0.663 on the same pages in 20 ms on a CPU, and boxes fields before anything is typed into them.March 2026