Usable information in a fixed dataset

Allen-Zhu and Li wrote a biography for each of 100,000 invented people, six facts apiece, and trained a 124-million-parameter model on them until it predicted the next word of every biography with over 99% accuracy. Then they fine-tuned it to answer questions about half the people and asked about the other half. It answered 9.7% correctly. They rewrote each biography in five wordings from fixed templates, with the sentences shuffled and no outside model involved, kept the facts and the number of training steps, and trained again. It answered 96.6%. Allen-Zhu and Li, 2023a

Khandelwal and colleagues took a 247-million-parameter model trained on the 103 million tokens of WikiText-103 and, at test time, let it look up the nearest matching contexts in its own training set and mix their next tokens into its prediction, with no further training. Its test perplexity fell from 18.65 to 16.12. Khandelwal et al.

Neither result adds data, and the second does not retrain the model. Both are impossible if what a model learns were set by the information in its training set, and both are routine once two quantities are kept apart. What a sample holds about its source is fixed. No rewriting, selecting, coarsening or rereading of the sample can add to it, because anything computed from the sample carries no more than the sample does. What a given learner can extract from the sample is a different quantity. Xu, Zhao and colleagues call it usable information: for a family of predictors, taken here to mean a model class together with the procedure that fits it, the drop in the best loss the family can reach once it is shown the input. Here the input is the training sample and the output is new text. Unlike the first quantity, usable information rises with the family and can rise when the input is processed. Xu et al., 2020

So the question “given a fixed corpus and every other degree of freedom, how do you get everything out of it?” is a question about the gap between those two quantities, and this digest maps the gap. Nearly every method that works at fixed data repairs one of four shortfalls of the training procedure: it stopped short of the optimum, it is one run where an average was needed, its regularization was set for the wrong number of passes, or it stored facts where it cannot read them. The rest enlarge what the model can represent: a larger model, a store over the sample, computation before answering. The remaining lever is a prior whose shape fits the data: what the learner assumes before seeing any text, including how many rare things it expects. For neural language models the evidence on it is empty. The question matters because public text grows about 3% a year while the compute spent on training grows about fourfold, so the fixed-corpus case is where training is heading. Villalobos et al., Kim et al., 2025

What a sample settles

The most any learner can get out of a sample has an exact answer once two things are stated: which sources could have produced it, and how much each was believed before any data arrived. The second is called the prior.

Take a coin of one of three kinds, equally likely before any flip: fair, heads-heavy (heads 90% of the time) or tails-heavy (heads 10% of the time). It lands heads four times in a row. The posterior, how much to believe each kind now, is each kind’s prior times its chance of that run, rescaled so the three sum to one. With equal priors, that is each kind’s chance divided by the sum of the three chances, 0.719.

The posterior holds everything the four flips say about which coin this is.

The best prediction for the next flip averages the three kinds’ chances of heads, weighted by the posterior. It comes to 0.865, a little under the heads-heavy coin’s 0.9, because the fair coin is not yet ruled out. This prediction is the posterior average.

“Best” has a precise meaning. Score a prediction by its loss: the negative log of the probability it gave to what happened, in nats. If the coin’s kind is drawn with the prior’s odds, no predictor that sees the same flips has lower expected loss than the posterior average. For one particular kind of coin it can be beaten: a predictor that always says 0.9 does slightly better on a heads-heavy coin, and far worse on the other two.

The posterior average also loses little by not knowing the coin’s kind. Measure this as its extra loss: its total loss over a run minus that of a predictor told the kind in advance. Its total loss is the negative log of the probability it gives the whole run, the product of its flip-by-flip predictions. That product equals the prior-weighted sum of the three kinds’ chances for the run, because each prediction is the weighted chance of the run so far plus the new flip, divided by the weighted chance of the run so far, and the product cancels. For the four heads it is a third of 0.719, or 0.24, against the heads-heavy coin’s 0.656, an extra loss of ln(0.656 / 0.24), or 1.0 nat.

That extra loss has a bound. The prior-weighted sum is never less than a third of the true kind’s chance, since that third is one of its terms, so the extra loss is never more than ln 3, about 1.1 nats, however long the run. In general the bound is the log of one over the true source’s prior weight. It holds run by run, so it holds whether or not the coin was drawn from the prior. Rathmanner and Hutter

The bound needs each possible source to carry prior weight of its own, as each of the three kinds does. A coin whose bias could be any number between 0 and 1 has a continuum of possible sources, and so does a neural network with real-valued weights. In a continuum no single source has weight of its own, so there is no fixed bound. For the coin with every bias equally likely, the extra loss keeps growing with the logarithm of the number of flips. It grows more slowly than the run, so the extra loss per flip still shrinks toward zero: about 0.003 nats per flip after 1,000 flips, and 0.0004 after 10,000. Grünwald

The prediction itself depends on the prior. Start with the fair coin at 98% and the other two kinds at 1% each, and after four heads the posterior average says 0.54 instead of 0.865. Whether 0.54 or 0.865 does better depends on how often each kind of coin actually turns up. Over many flips, any prior that gives the true kind some weight approaches the same prediction; after four, the prior still matters a great deal. Rathmanner and Hutter

Avoiding a prior still leaves a choice to make. The standard prior-free method judges a predictor by its worst run: on each possible run of a given length, compare its loss with that of the source that fits the run best, and take the largest gap. The predictor with the smallest worst gap, called normalized maximum likelihood, needs no prior. It still needs a set of sources to compare against. For a coin of any bias over 1,000 flips its worst gap is 3.7 nats, and a posterior average under one particular prior, Jeffreys’, has a worst gap about a third of a nat larger. Grünwald

So even by the worst-case standard, the best predictor is close to a posterior average. Call the posterior average’s expected loss the ceiling on what a sample can give: no predictor does better on average over sources drawn from the prior, and the ceiling moves when the prior does.

On real text no prior is exactly right, so the ceiling is only as good as the prior. Language shows how much the prior’s shape matters. A model of word triples has to decide how much probability to give words never seen after a given pair of words. A hierarchical Pitman-Yor prior expects a long tail of rare words, and its posterior average answers with discounts. Each word’s count after a pair is reduced by a discount that grows slowly with the count, and the probability removed goes to an estimate from the last word alone. A word seen once after a pair keeps about a fifth of its count there, and one seen fifty times about nine tenths (both read off Teh’s figure). A Dirichlet prior, which expects no long tail, scales every count down by the same share instead. Interpolated Kneser-Ney, a standard way of smoothing word counts, falls out of the Pitman-Yor posterior under one approximation, which subtracts the same discount at every count. Teh

Teh compared these on about 14 million words of news. The Pitman-Yor posterior average had a test perplexity of 114.3, against 116.7 for interpolated Kneser-Ney and 140.5 for the Dirichlet version, so the shape of the prior made by far the largest difference. But the posterior average was not the best. Modified Kneser-Ney, whose discounts are tuned on held-out data, scored 113.6, and the same Pitman-Yor model with its prior’s six settings, the discounts among them, tuned on held-out data instead of averaged over, scored 113.2. Teh’s conjecture is that the model is not a perfect model of language, so the settings its posterior favors are not the ones that predict best. Teh

Other tests point the same way, with small gains. On meeting transcripts of up to 211 million words, the same model’s perplexity was 4 to 6% lower than modified Kneser-Ney’s, though there the Kneser-Ney discounts were not tuned on held-out data. A test of averaging alone comes from recurrent networks on word-level benchmarks: averaging predictions over samples from an approximate posterior on the weights lowered perplexity by 0.8 to 2.4% against predicting from a single setting of the same weights. Huang and Renals, Gal and Ghahramani, Maddox et al.

The Pitman-Yor posterior still leaves a once-seen word a fifth of its count rather than none. Feldman showed, for classification, what it costs to give such examples up entirely. In his setting, examples come from many groups whose frequencies follow a long tail. With correct labels, the learner with the lowest expected error reproduces the label of every example it was shown, under any prior. Each once-seen example a learner gets wrong adds to its test error at least the frequency that a group seen once is expected to have. Under a long tail many groups have frequencies near one over the sample size, so that expected frequency is not small. In his example with Zipfian frequencies and 10 classes, guessing at random on the once-seen examples raises test error from about 15% to about 22%. Feldman

So getting everything out of a corpus takes a prior whose shape fits the data and its posterior average computed as well as possible, which keeps what was seen once rather than smoothing it away. A single trained network is one setting of its weights, fit under a prior chosen because it is easy to optimize, such as the Gaussian prior that weight decay amounts to. Whether it keeps what was seen once depends on how it was trained. The rest of this digest is about the distance between such a network and the posterior average.

Where a trained model falls short

The loss on new text has four parts: what the source itself leaves unpredictable; what the model family cannot represent; what the sample does not settle under the prior, which the best possible learner would also get wrong; and what the training procedure loses on top of that.

The split between the last two parts needs a rule, since a trained network has no prior written down. Count as prior whatever is fixed before the sample is seen and would still bind a perfect learner: the family’s shape, the invariances built into it, and the form of the regularizer. Count as procedure whatever decides which member of the family the sample produces: the optimization, the number of runs, the regularizer’s strength, and the form in which the sample is presented. Weight decay’s Gaussian shape is prior; its strength, which the sections below match to the number of passes, is procedure. A prior’s shape, as used below, means two things: its tail, which decides how much dropping something seen once costs, and what it treats as the same item, which decides how fast the curve falls.

The part the sample does not settle is where the data exponent lives: the power that says how fast loss falls as the sample grows. In the simplest model of a learning curve, a learner that remembers every item it has seen and nothing else, that exponent is set by how much probability sits on rarely seen items. A learner that needs several sightings before it keeps an item sits on a parallel curve. By my arithmetic in that model, with the item frequencies set so that the curve’s exponent is 0.23, needing 2, 4, 8 or 16 sightings costs 2.5, 5.4, 11 and 23 times the data at the same exponent. Only a change in what counts as a distinct item moves the exponent, and only for the part of the loss that shared structure can explain. On images that lever is real: the data exponent of kernel regression on CIFAR-10 ranged from 0.05 to 0.25 with the kernel. For neural language models it has never been measured. The only study with several recipes at several data sizes, by Kim and colleagues, fits four points per recipe, and those four points fit any exponent from 0.10 to 0.35 equally well. Hutter, Bahri et al., Kim et al., 2025

That leaves the procedure’s part and, through model size and answer-time computation, the family’s. Every data-efficiency multiplier below acts there. A multiplier of 2 says the baseline would need twice the data to match. At the exponent Kim and colleagues fit, 0.23, that is also a 15% cut in the part of the loss above the floor, the source’s own unpredictability, and a multiplier of 5 is a 31% cut, so a shallow learning curve turns modest loss gains into large-sounding factors.

Finishing the optimization

The cheapest shortfall to repair is that one pass over the data does not reach the optimum of the training objective. Muennighoff and colleagues measured the alternative with everything else held fixed. Models of 2.8 to 8.7 billion parameters trained on a quarter of the data four times, with the same batch, schedule and step count as one pass over all of it, ended 0.6 to 0.8% higher in loss. That gap is the information the repeated tokens lack, and it is small. At fixed data the same runs show what the passes buy: a 4.2-billion-parameter model on 12 billion unique tokens fell 0.27 nats going from one pass to four, 89 to 95% of what four times the fresh data would have given. Muennighoff et al.

Could the first pass have reached the same loss by taking smaller steps? Only partly. At fixed tokens, a smaller batch is worth at most a factor of 1 + B/B_crit in data, where B is the batch size and B_crit the size beyond which more examples per step stop helping. In single-pass runs of a 610-million-parameter model on 12 billion tokens, cutting the batch from 4.1 million to 0.5 million tokens gained 0.074 nats, and a further fourfold cut lost 0.016. Applied to the run above, by my arithmetic from those two fits, a smaller batch could have supplied about 0.05 of the 0.27. The lever matters most on small datasets, since the critical batch grows with data, and it costs no extra arithmetic but more sequential steps. Annealing the learning rate is worth a bounded amount, 0.07 to 0.24 nats in the fits and runs available. Under repetition the gain arrives only at the end of the schedule, so intermediate checkpoints mislead. McCandlish et al., Bergsma et al., Tissue et al., Hägele et al.

Repetition alone keeps lowering loss here as compute grows. It stops when the target itself goes wrong: a model with capacity to spare, passed over the same text often enough, fits its particulars instead of its patterns. What happens then is set by the regularizer.

Regularization matched to the pass count

Weight decay is the regularizer that sets this. With the AdamW optimizer, the weights are a running average of recent updates with a forgetting time of about 1/(learning rate × weight decay) steps. Kim and colleagues tuned weight decay together with learning rate and pass count for sixteen combinations of model size and data size, and the chosen values ranged over a factor of 32. Converted to forgetting times, every one of them lies between 0.2 and 0.5 passes over the data, so a single value near 0.3 passes recovers each tuned weight decay to within a factor of two from that run’s learning rate. An independent grid by Xu, Wu and colleagues shows the same constancy at about one pass under a different schedule. The rule that larger models need more weight decay is mostly their smaller learning rates showing through. Wang and Aitchison, Kim et al., 2025, Xu et al., 2026

With weight decay set that way, larger models keep helping on fixed data. On 209 million tokens, Kim and colleagues’ best 1.4-billion-parameter model reached 3.46 nats against 3.75 for the standard recipe, which tunes learning rate and passes at the customary weight decay of 0.1 and keeps the best model size, 600 million parameters here. Their fitted law relating loss to data converts that gap to about twice the data, and the tuned 300-million-parameter model, half the size of the standard recipe’s best, is worth about 1.5 times. Set wrong, the same regularizer costs. At one fixed learning rate, raising weight decay from 0.1 to 1.0 made single-pass training worse by 0.10 to 0.17 nats and helped only beyond about four passes. At 300 million parameters, with the learning rate and passes held at their tuned values, eight times the tuned weight decay gave 3.98, 0.39 above the tuned 3.59 and worse than the 3.95 that the default 0.1 gives the same model. Kim et al., 2025, Lovelace et al.

Dropout and weight sharing behave the same way. Dropout of 0.1 raised a 2.8-billion-parameter model’s masked-token accuracy by 7.6 points after 256 passes over 134 million tokens, and cost 0.9 points on a single pass over fresh data. A looped transformer, which runs one block of layers several times, gained 0.02 to 0.03 nats at matched compute over larger unlooped models when 100 million tokens were repeated ten times. Against the same block run once, in a comparison with untuned weight decay, the gain specific to sharing the weights rather than adding depth was 0.01 to 0.04 nats. On fresh data the same sharing costs 0.03 to 0.12 nats. The precedent is in vision, where augmentation and dropout gained 4 to 15 points of accuracy on 1.3 million images and cost up to 13 points on 13 million at the same compute. Xue et al., Chen et al., Schwethelm et al., Steiner et al.

These regularizers behave alike. They carry no information about the data’s content; their right strength follows from how many times each example will be seen; they substitute for one another; and each has a measured cost where it is set for the wrong regime. That is why so many reported data-efficiency gains vanish against a tuned baseline. A method that arrives with its own regularizer, as masked diffusion does (a model trained to fill in randomly hidden tokens, with the hidden fraction redrawn every time), looks efficient next to a baseline that has none. No study has set a regularizer’s strength for language-model pretraining without a sweep on held-out loss. The marginal likelihood, the Bayesian way to pick a prior’s strength, has been shown only on networks under two million parameters. Immer et al.

One run against the average

The largest fixed-data gain measured after tuning regularization comes from training several copies of a model from different random seeds and averaging their predictions. Averaging five 1.4-billion-parameter models trained on Kim and colleagues’ 209 million tokens gave 3.27 nats, against 3.46 for the best single model of that size, a loss the standard recipe would need about 3.7 times the data to reach. The cost is about 47 times the training compute of the standard recipe’s best run and 12 times its inference compute. The figure below lists the measured points. Kim et al., 2025

Why averaging helps is an identity. When predictions are combined by averaging log-probabilities, as Kim and colleagues do, the ensemble’s loss equals the members’ average loss minus a term measuring how much they disagree. One trained model therefore sits above its procedure’s mean predictor, the average prediction of all the runs the procedure could produce, by the disagreement among seeds. That disagreement is large here. Single 300-million-parameter models agree on their loss to within 0.008 nats from seed to seed, because they are equally wrong on average, yet they disagree on individual predictions enough that an ensemble of eight reached 3.32: 0.27 below the best single model of that size at 3.59, and 0.39 below its own members, which were trained to be individually worse, at 3.71. The gap between one model and the ensemble shrinks as one over the number of members, which is what averaging any independent random predictors gives, so the fit can be extrapolated to unlimited members. For those eight members the fitted limit is 3.27, by coincidence the measured loss of the five-model ensemble above. Kim et al., 2025

That limit is the mean predictor defined above, and it belongs to the recipe, not to the data. With members kept at the single model’s settings it is 3.34; retuning them to be individually worse and more varied moves it to 3.27; growing them from 150 million to 1.4 billion parameters moves it from 3.49 to 3.17. The only direct comparison with a true posterior average, on small image and text classifiers, found a 50-member ensemble within 0.02 to 0.24 nats of it at about one 2,400th of the compute. So the extrapolated limit is a property of the ensemble, not the Bayesian ceiling of the first section. Kim et al., 2025, Izmailov et al.

Most of the gain can be moved back into one model. A 300-million-parameter student trained on the real text plus text sampled from the 8-model ensemble reached 3.36, keeping about 84% of the ensemble’s gain over the best single model of its size at an eighth of the inference cost. A student whose teacher is a single model of the same size, trained on the same data, reached 3.44, so self-distillation recovers about half of the ensemble gain. In Kim and colleagues’ follow-up, which retuned every recipe and so reports slightly different values, ensembles of self-distilled students converge to a fitted limit of 3.32, against 3.31 for ensembles of ordinary models in the same study, so self-distillation moves one model toward the average without raising the limit. The student trained at the default weight decay, so the other reading, that the teacher’s samples stand in for regularization, has not been excluded. In both cases the real text has to stay in the mix: a self-distilled student trained on its teacher’s samples alone ended at 4.07, worse than the teacher’s 3.71. Averaging weights instead of predictions works only within one training run. Weight averaging and exponential moving averages give a better single model by hundredths of a nat, and averaging the weights of independently trained models collapses downstream accuracy to chance. Kim et al., 2025, Kim et al., 2026

The distillation result also says what was binding at this scale. A 300-million-parameter model can hold a predictor at 3.36 nats, and training it directly on the sample found 3.59. The shortfall was in getting from the sample to a good model, not in the model’s capacity. Across data sizes the ensemble’s advantage shrinks at a fixed member size of 1.4 billion parameters, from about 3.7 to 2.8 times the data as the sample grew eightfold. Both a variance account and a Bayesian one predict that: the more the data determine the model, the less independent runs disagree. Kim et al., 2025

What regularization forgets

Everything above is measured in held-out loss, and held-out loss is nearly blind to the long tail. A fact stated once in 200 million training tokens has about a 2% chance of appearing in a 4-million-token validation set at all. Under a Zipf law with exponent 1, if a fact needs about 100 sightings to be stored, one pass over a corpus stores about 1% of its distinct facts, whatever the corpus size, and those facts account for about two-thirds of the fact mentions. The whole once-seen tail is worth a few hundredths of a nat per token in Kim and colleagues’ setting, against recipe gains of 0.3 to 0.5. Loss and recall can move in opposite directions: in a controlled study, adding training facts at fixed model size lowered loss on unseen facts while the share of training facts the model could reproduce fell from near 100% to about 40%. Allen-Zhu and Li, 2024, Lu et al.

The Gaussian prior that weight decay imposes has the wrong shape for that tail, and the tuned strength makes it worse. Models store about 2 bits of factual information per parameter when each fact is seen a thousand times, about 1 bit at a hundred, and almost nothing at one. Kim and colleagues’ tuned recipe halves any weight the gradient does not refresh within a fraction of one pass (the forgetting time above, times the log of 2), where the capacity experiments used decay a thousand times gentler. In public models the pattern is visible. BLOOM-176B answered about 25% of trivia questions backed by ten training documents and over 55% of those backed by ten thousand, and a 560-million-parameter model answered about 2% of questions backed by fewer than a hundred. Nobody has measured what tuned weight decay, ensembling or distillation cost in rare-fact recall at fixed data; the downstream checks in those papers are commonsense and reading tasks. Allen-Zhu and Li, 2024, Kandpal et al.

The remedy is the oldest one in language modeling: use the sample twice, once as a fit that shares statistics across contexts and once as a store of what each context contained, and weight the store by how much evidence the sample holds for that context. That is the Pitman-Yor prediction with a network in place of the backed-off estimate. In the terms of the four-part figure above, a store enlarges the model family to include the sample itself, which is what a long-tailed prior’s posterior already contains. Bengio and colleagues’ neural language model of 2003 scored its headline gains over the best n-gram, 24% on Brown and 8% on AP News, only when mixed half and half with an interpolated trigram, which helped in every configuration. Khandelwal’s nearest-neighbour store is the modern form, and what it recovers is more than the once-seen tail: its gains concentrate on rare names and near-duplicated passages seen a few times. About half of its 13.6% perplexity cut comes back without any store, from a second trained output head on the same representation, so part of what the store supplies is a better parametric head. Bengio et al., Khandelwal et al., Xu et al., 2023

The store’s benefit shrinks as the parametric model improves, and shrinks again when the model has already trained on the store’s text. With the Pile’s training set as an unbounded n-gram store, perplexity on the Pile test set fell by about a third for a 1.6-billion-parameter GPT-2, which never trained on the Pile, by 13% for GPT-Neo 1.3B and by 10% for GPT-J 6.7B, which did. On Wikipedia articles written after both corpora were collected, Llama-2 13B gained 0 to 5%. Fine-tuning on fifty retrieved training neighbours before each test document cut bits per byte by about 20% for a GPT-2 that had not seen the Pile and about 4% for the Pile-trained GPT-Neo. The perplexity gains also do not carry to generation: a store that cut GPT-2’s WikiText perplexity from 14.8 to 12.6 raised the probability of the correct next token for only 42% of tokens and left generated text no better by automatic or human judgment. Liu et al., Hardt and Sun, Wang et al.

The combination the argument asks for has not been run: regularization set hard for the patterns, a store over the same corpus for the particulars, and rare-fact recall measured by exposure count.

The same facts in more forms

Back to the opening result. Rewriting each biography five ways added no information about the people and changed where the model stored it. Without rewriting, a linear probe could read each attribute from the hidden state only just before the attribute appeared in the biography; with it, every attribute could be read at the person’s name. The lesson transfers: adding 100,000 other people with rewritten biographies raised accuracy on the unrewritten people from 4.4% to 86.8%, in a variant whose baseline sat lower than the opening’s 9.7%, though their own text was untouched. It also works only in the direction the text was written. With unlimited rewrites that kept the name first, models could not name a person from their attributes, and GPT-4 predicts the sentence after a given sentence of Pride and Prejudice 65.9% of the time and the sentence before it 0.8%. Rewriting raises the sample’s usable information for this learner, which stores facts by position unless the data forbid it. The data-processing inequality that forbids any gain is a statement about the other quantity. Allen-Zhu and Li, 2023a, Allen-Zhu and Li, 2023b

A view, meaning a prediction problem derived from the same text, works the same way, in two roles. The first is access. A model trained to fill in a missing span of code, by moving the span to the end of the document, raised its single-line infilling pass rate from 0.19 to 0.60 while its left-to-right loss moved by less than 0.4%. A byte-level model spelled words correctly 99.9% of the time where a token model trained on the same data managed 1.1%. A next-token model already holds the answer to “what fills this gap”; the extra view puts it one forward pass away. The second role is regularization again. On a single pass, extra views are close to neutral: fill-in-the-middle training moved left-to-right loss by −0.33% to +0.24% across model sizes, and a multi-token prediction head left DeepSeek-V3’s held-out bits per byte unchanged. Under heavy repetition they lower the original view’s loss by tenths of a nat against a weakly regularized baseline and by hundredths against a tuned one. Adding a next-token loss on a randomly masked copy of each sequence gained 0.006 to 0.03 nats on every one of five seeds, at twice the compute per step, on a baseline whose own tuning had gained 0.46. Bavarian et al., Pagnoni et al., DeepSeek-AI, Xu et al., 2026

Masked diffusion is the limiting case of the second role, and its reported advantage on repeated data is its built-in regularizer. Its loss, with the mask rate drawn uniformly and weighted by one over the rate, is the average loss of a model writing the tokens in a random order. It trains every conditional of the joint distribution at every amount of visible context, and it averages over nothing. Given the same tuned weight decay, an autoregressive model matches it at every size tested. For helping a left-to-right model, light masking beats the full range: in Xu, Wu and colleagues’ grid, masking at rates up to 0.5 gave 3.308, rates up to 0.99 gave 3.353, and no masking gave 3.340. Ou et al., Xu et al., 2026, Prabhudesai et al.

Coarse views deserve a separate statement, because they look like they should add something. For a summary z computed from a document x, the probability of the document factors exactly into the probability of the summary times the probability of the document given it. A model that first learns the summary and then the text given the summary, a coarse-to-fine model, learns the same thing in two steps. What such a model can change is the estimation: fewer symbols, lower-variance statistics, a second training signal. No language-model test holds the data fixed with compute free; the comparisons that exist are compute-matched or import a teacher. Training the first two-thirds of a run on averages of four token embeddings halved compute at equal loss, and its small loss advantage reversed below about 300 training tokens per parameter, the regime of a fixed sample. A 1.6-billion-parameter model predicting sentence embeddings lost to a 1.4-billion-parameter token model on the same data. The only coarse-concept method that lowered loss took its concepts from a pretrained teacher. In images, generating each pixel’s high bits before its low bits beat random order on CIFAR-10 by 0.05 bits per dimension, and the same authors report no gain on text. Shao et al., Barrault et al., Tack et al., Hoogeboom et al.

Who writes the new text settles what a rewriting result measures. Text sampled from a model trained on the same corpus is averaging, bounded by the ensemble limit above. Text written by a model trained on far more is more data. In Kim and colleagues’ follow-up, ensembles of 300-million-parameter models converge to 3.31 on the real corpus and to 3.18 with rewrites from Llama 3.1 8B Instruct; of the drop from a tuned single model at 3.55 to 3.18, about two-thirds is reachable by averaging models trained on the corpus alone. A 1-billion-parameter writer produced nothing useful, 8 billion worked and 70 billion was no better. The largest reported gains have the strongest writers: GPT-4o-mini’s plain rewording of 480 million tokens of web math matched 16.7 times as much fresh math text on the MATH benchmark, and its added reasoning raised the score to 19.4 from 11.1. No one has shown a writer trained only on the fixed corpus doing the same on natural text. Kim et al., 2026, Ruan et al.

Which parts to learn from

Choosing which parts of a fixed sample to train on, how often, and in what order cannot add information either, and the evidence says it does not help. In every comparison that held the pool fixed, discarding lost. No pruned subset of ImageNet beat training on all of it, and a loss-based selector that spent half the data on its own training finished about seven points behind uniform sampling. Faster-than-power-law curves from pruning plot error against the examples kept, with each kept set drawn from a larger pool. Filtering wins only while the pool holds more tokens than the run will process; on one fixed pool, unfiltered data overtook a 10% filter after about three and a half passes. Repeating a subset is worse than repeating everything, because it changes what the objective rewards: an 800-million-parameter model with 10% of its tokens drawn from a 0.1% subset repeated a hundred times ended near the loss of a 340-million-parameter model. Deduplication corrects the likelihood’s assumption that examples are independent. At matched passes it cut memorized output tenfold and left perplexity on unseen text unchanged. Curricula have no controlled positive evidence, and the only measured fixed-data effect of data order is as a source of ensemble diversity. Sorscher et al., Mindermann et al., Goyal et al., Hernandez et al., Lee et al., Kim et al., 2025

Computation at answer time

The part of the loss that the model family cannot represent has a reading at inference: the family is whatever the model can compute before it answers. Sort answer-time methods by where their information comes from. Methods that use only the question and the weights are bounded by what the weights hold. Writing out an intermediate step lets a model combine two facts it stored but never saw together. GPT-4o answered 92.8% of such two-step questions when it wrote the middle entity and 7.6% when it answered directly, and in a synthetic setting a transformer trained with written middle steps answered 99.4% of held-out two-step questions where one trained to answer directly never learned them. The limit is the lookups: 96% of failures with written steps were a wrong first fact, and two-step training did not extend to three steps. At matched compute, a small model with answer-time search beat a model 14 times larger only on the easiest two fifths of questions, ranked by the small model’s own pass rate. Yang et al., Yao et al., Snell et al.

Methods that put the training set back in at answer time recover some of what training saw and did not keep, the few percent of loss that a store over the training set recovers. The context window is the weaker of the two ways to do it. In a synthetic world, in-context examples select among concepts training stored and do nothing for concepts it did not. Putting retrieved training documents in GPT-2 Large’s prompt moved its bits per byte from 1.07 to 1.06, where a gradient step on the same documents gave 20%. Test-time training also extracts more from a task’s own demonstrations than reading them does: with the same ten examples, Llama 3.1 8B averaged 50.5% on BIG-Bench Hard with them in its prompt and 57.8% after a few gradient steps on rearrangements of them. The demonstrations are new information, so that is not extraction from fixed data. It is a measure of how much the prompt fails to extract. Xie et al., Hardt and Sun, Akyürek et al.

What is settled, what is interpretation, what is open

Settled, at scales under 10 billion parameters and under 15 billion unique tokens:

• Four passes over the text cost 0.6 to 0.8% in loss against four times the fresh text, with optimization held fixed. Smaller batches and annealing add bounded amounts.
• Weight decay set by the pass count, with a forgetting time between a fifth of a pass and one pass, lets larger models keep helping. The best tuned model is worth about twice the data of the standard recipe’s best, and a tuned model half that size about 1.5 times. Set for the wrong regime it costs 0.1 to 0.4 nats.
• Averaging five 1.4-billion-parameter models is worth about 3.7 times the data at 47 times the compute, a figure that already includes the tuned weight decay. A 300-million-parameter model distilled from an 8-model ensemble keeps about 84% of that ensemble’s gain, and self-distillation keeps about half. Averaging composes with rewrites from an outside writer and not with self-distillation.
• The same facts rewritten in several forms become extractable where one form left them stored and unusable, in a synthetic corpus.
• A store over a training corpus cuts perplexity by about a third for a model that never trained on it and by 10 to 14% for one that did, at up to 7 billion parameters and by less as the model grows. In loss that is about 14% and about 5%, and it does not improve generation.
• Selecting or reweighting a fixed sample does not beat uniform repetition of all of it, and ordering has no controlled evidence either way.

Interpretation, mine unless noted:

• That the methods above all act on the procedure’s shortfall or enlarge the family, and leave the sample’s own limit untouched. The decomposition is standard; placing each result in it is this digest’s reading, and the one measured case of a prior changing a learning curve’s slope is in images.
• That weight decay under repetition acts as a memory control on the optimizer rather than as a Bayesian prior; the data-size trend cannot separate the two, and the rule above only assigns its strength to the procedure.
• That self-distillation is averaging rather than a stand-in for regularization; the student trained at the default weight decay, and the control has not been run.

Open, with the experiment that would settle each:

• Whether strong regularization plus a store over the same corpus keeps both patterns and particulars. Insert synthetic facts at exposure counts from 1 to 256 into a 200-million-token corpus, train at weight decays from 0 to 3.2 with three seeds, and measure recall by count with and without the store.
• Whether any lever changes the data exponent in a language model. Run the tuned, averaged and rewritten recipes at four data sizes, fit each curve, and compare slopes with seed-based intervals; four points per recipe cannot do it.
• Whether a coarse view helps at fixed data. Add a coarse-level auxiliary target to Kim and colleagues’ tuned recipe and measure the fine view’s loss against the masked-copy control, five seeds.
• Whether a writer trained only on the corpus clears the threshold that an 8-billion-parameter outside writer clears, and at what corpus size.
• Whether any of this survives at the scale where the data limit binds, hundreds of trillions of tokens, where models are far from over-parameterized and seed disagreement should be smaller.

Sources

• Allen-Zhu and Li, 2023a. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. Rewritten biographies make stored facts extractable.

• Allen-Zhu and Li, 2023b. Physics of Language Models: Part 3.2, Knowledge Manipulation. Facts are recalled in the direction they were written.

• Allen-Zhu and Li, 2024. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. Bits per parameter against exposures.

• Akyürek et al., 2024. The Surprising Effectiveness of Test-Time Training for Abstract Reasoning. Gradient steps on a task’s demonstrations against reading them.

• Bahri, Dyer, Kaplan, Lee, Sharma, 2021. Explaining Neural Scaling Laws. The data exponent and what moves it, on images.

• Barrault et al., 2024. Large Concept Models. Sentence-level prediction against a token model on the same data.

• Bavarian et al., 2022. Efficient Training of Language Models to Fill in the Middle. A second view of the text at no cost to the first.

• Bengio, Ducharme, Vincent, Jauvin, 2003. A Neural Probabilistic Language Model. The first neural language model, mixed with counts.

• Bergsma et al., 2025. Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training. Batch size at fixed tokens.

• Chen et al., 2026. How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents. Looping on repeated data, with an untied control.

• DeepSeek-AI, 2024. DeepSeek-V3 Technical Report. The multi-token prediction ablation.

• Feldman, 2020. Does Learning Require Memorization? A Short Tale about a Long Tail. Under a long tail, failing to fit rare examples costs test error.

• Gal and Ghahramani, 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. Averaging over dropout samples against a single pass, on the Penn Treebank.

• Goyal et al., 2024. Scaling Laws for Data Filtering. Filtering on a fixed pool against compute.

• Grünwald, 2004. A Tutorial Introduction to the Minimum Description Length Principle. The worst-case guarantee that does without a prior, and how close a posterior average comes to it.

• Hägele et al., 2024. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. What a cooldown is worth.

• Hardt and Sun, 2023. Test-Time Training on Nearest Neighbors for Large Language Models. Gradient steps on retrieved training documents.

• Hernandez et al., 2022. Scaling Laws and Interpretability of Learning from Repeated Data. Repeating a subset.

• Hoogeboom et al., 2022. Autoregressive Diffusion Models. Coarse-to-fine ordering on images and text.

• Huang and Renals, 2010. Hierarchical Bayesian Language Models for Conversational Speech Recognition. Teh’s model against Kneser-Ney on up to 211 million words of meeting transcripts.

• Hutter, 2021. Learning Curve Theory. Where a learning curve’s exponent comes from.

• Immer et al., 2021. Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning. Setting regularization without a sweep, on small networks.

• Izmailov et al., 2021. What Are Bayesian Neural Network Posteriors Really Like?. A true posterior average against deep ensembles.

• Kandpal et al., 2023. Large Language Models Struggle to Learn Long-Tail Knowledge. Recall against the number of supporting documents.

• Khandelwal et al., 2020. Generalization through Memorization: Nearest Neighbor Language Models. A store over the model’s own training set.

• Kim, Kotha, Liang, Hashimoto, 2025. Pre-training under infinite compute. Tuned weight decay, ensembles, distillation and self-distillation on 209 million tokens.

• Kim et al., 2026. Data-efficient pre-training by scaling synthetic megadocs. Rewrites from an outside model, and which gains compose with ensembling.

• Lee et al., 2022. Deduplicating Training Data Makes Language Models Better. What deduplication changes at matched passes.

• Liu et al., 2024. Infini-gram. An unbounded n-gram store over the Pile.

• Lovelace et al., 2026. Prescriptive Scaling Laws for Data Constrained Training. The single-pass cost of strong weight decay.

• Lu et al., 2024. Scaling Laws for Fact Memorization of Large Language Models. Loss and recall moving apart.

• Maddox et al., 2019. A Simple Baseline for Bayesian Uncertainty in Deep Learning. Averaging over an approximate posterior against its mean, on two word-level benchmarks.

• McCandlish, Kaplan, Amodei, 2018. An Empirical Model of Large-Batch Training. The critical batch size.

• Mindermann et al., 2022. Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. Loss-based selection with all data used.

• Muennighoff et al., 2023. Scaling Data-Constrained Language Models. Repetition with optimization held fixed.

• Ou et al., 2024. Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data. The masked-diffusion loss as random-order prediction.

• Pagnoni et al., 2024. Byte Latent Transformer. Byte-level access to spelling.

• Prabhudesai et al., 2025. Diffusion Beats Autoregressive in Data-Constrained Settings. The comparison at weight decay 0.1.

• Rathmanner and Hutter, 2011. A Philosophical Treatise of Universal Induction. The posterior predictive’s guarantee and its dependence on the prior.

• Ruan et al., 2025. Reasoning to Learn from Latent Thoughts. Rewording and reasoning from an outside writer.

• Schwethelm et al., 2026. How Much Is One Recurrence Worth?. The cost of weight sharing on fresh data.

• Shao et al., 2024. Patch-Level Training for Large Language Models. Coarse units as a compute saving.

• Snell et al., 2024. Scaling LLM Test-Time Compute Optimally. Answer-time search against a larger model.

• Sorscher et al., 2022. Beyond neural scaling laws: beating power law scaling via data pruning. Pruning against the examples kept.

• Steiner et al., 2021. How to train your ViT?. Augmentation and dropout against data size.

• Tack et al., 2025. LLM Pretraining with Continuous Concepts. Coarse concepts from a pretrained teacher.

• Teh, 2006. A Hierarchical Bayesian Language Model based on Pitman-Yor Processes. Interpolated Kneser-Ney as approximate Bayesian inference.

• Tissue et al., 2024. Scaling Law with Learning Rate Annealing. What annealing is worth.

• Villalobos et al., 2024. Will we run out of data?. The stock of public text against the growth of training runs.

• Wang and Aitchison, 2024. How to set AdamW’s weight decay as you scale model and dataset size. Weights as a running average with a forgetting time.

• Wang et al., 2023. kNN-LM Does Not Improve Open-ended Text Generation. Perplexity gains that do not reach generation.

• Xie et al., 2022. An Explanation of In-context Learning as Implicit Bayesian Inference. The context selects among stored concepts.

• Xu, Zhao, Song, Stewart, Ermon, 2020. A Theory of Usable Information under Computational Constraints. Information relative to a predictive family.

• Xu et al., 2023. Why do Nearest Neighbor Language Models Work?. Half the store’s gain from a second output head.

• Xu et al., 2026. Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws. Tuned weight decay, and masking as a second view.

• Xue et al., 2023. To Repeat or Not To Repeat. Dropout under extreme repetition.

• Yang et al., 2024. Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?. Two-step questions with and without written steps.

• Yao et al., 2025. Compositional Generalization from Learned Skills via CoT Training. Written steps and composition.