Manifesto

Labs talk about data quality as though it were printed on the file, and most find out otherwise after the compute is spent. A team training an open-source code model once kept only code from repositories with five or more GitHub stars, and the model got worse, because stars track what people like and that had little to do with what the model needed. Some labs have stopped asking and train on everything, on the theory that at enough scale the good data carries the bad, though the large studies keep finding that it doesn't. When DCLM filtered its Common Crawl pool with a model before training, the model it trained outscored the best open-data model before it on MMLU with 40% less compute. FineWeb-Edu kept under a tenth of a 15 trillion token crawl for its educational content and did much better on knowledge and reasoning benchmarks than the full crawl, so even with trillions of tokens to spend, which tokens a run spent them on still decided what the model could do.

What is a dataset worth?

A dataset is worth what it adds to a particular model. A corpus the model already knows adds little, and so does one it lacks the groundwork to use. Induction heads, the attention circuits that let a model continue a pattern from earlier in its context, are that kind of groundwork. They form early and much later learning builds on them, so the same data can be noise before they exist and make a capability click after.

Value also moves as models do, with some data losing it as the frontier absorbs what's in it and some gaining it as stronger models pull more out, which is roughly what epiplexity tries to measure. The form data arrives in and the compute behind a run shift it too.

Why doesn't it trade?

Data is sold by the token, hour or file because those are easy to count, and none of them track what a model gets out of it. If value depends on the buyer's model, there's no single right price, only a price per buyer, and today neither side can compute it, so buyers won't pay without looking at the data and sellers won't show it without being paid. Buyers who can't tell good data from bad bid as though it were all average, which leaves whoever holds the best data with the least reason to sell. Google paid $10 million for Spirit Airlines' email, chat logs and code at a bankruptcy auction without any way to know what it would add to a model.

What's left to learn from

The public internet is mostly finished work, and models have already learned most of what it can teach. What's left is the record of how the work got done, which is mostly private. In a CAD vault that's the revisions, each one a design bending to what the shop floor could make, and in an agent trace it's the mistake and the recovery. Cleaning pipelines throw those parts away because they look like noise, and synthetic data can't replace them, since a model can't produce a record of constraints it never saw.

Seeds for environments

RL environments have the same problem one step removed. An environment can be checked against its spec, but its value comes from the seed it was built from, such as a codebase and its history of changes. The obvious signals about a seed mislead, e.g. years in production only show that a codebase survived, sometimes because nobody dared touch it. SlopCodeBench found that agents' code grew more bloated and tangled with each round of changes while the tests kept passing, which is why labs want seeds from enterprise code that has lasted through years of other people's edits.

Many of the seeds that matter most aren't public. Banks and Social Security still run on COBOL, and with the people who know it retiring and models weak at it, keeping those systems running has become a national security question. Whether a seed is good only shows up downstream, in how a model trained on it does on systems held out from training.

What we do

We measure what a dataset does to the model that will use it, with short training runs at fixed compute and whole sources held out, plus checks a machine can rerun, like whether a parser accepts a file. Then we look for the form a dataset teaches best in, which is rarely the form it arrived in.

We're working toward a measurement a seller can show without giving the data away, so buyers can see what a corpus would do before paying, which data needs before it can be held, financed and traded like other assets. Each deal also records the price a corpus fetched and how much a model improved on it, and enough of those records make a dataset of their own, one we'd eventually train a model on to price data before anyone trains on it.