Generalizing beyond the training distribution

Wu and colleagues asked GPT-4 to add two two-digit numbers, working step by step, in five number bases. In base 10 it was right on 98.2% of a thousand problems. In base 9, where a carry happens at 9 instead of at 10 and nothing else about the procedure changes, it was right on 38.6%. It was not confused about the question: asked what number comes after a given one in base 9, it answered correctly 90% of the time. Shown sixteen worked base-9 examples before each problem, it reached 88.4%. Wu et al.

Raventós and colleagues trained a small transformer to do linear regression from the examples in its prompt: sixteen input-output pairs from one unknown linear function of eight variables, predicting each output from the pairs before it. Each training run drew its training sequences from a fixed set of functions, and the test used functions the model had never seen. With 32 training functions, the model’s error on new functions was 0.964. That is within 0.003 of the best possible predictor that assumes the test function is one of the 32, and nearly three times the 0.344 of ridge regression, the best predictor for a fresh function. With 32,768 training functions, the same architecture trained on the same number of sequences scored 0.345 on new functions, within 0.001 of ridge regression. It had stopped behaving as a lookup over the training functions and started doing regression. Raventós et al.

Both results are about inputs that differ from the training data in a named way: a base the data never used, a function the data never contained. Out of distribution, as this digest uses the phrase, means just that: the test inputs move along some axis that the training data held fixed or covered only in part. The first result is what failure looks like. The model has a procedure for addition, but the procedure works best in the base its data used and worse in bases rarer in text, by the authors’ reading: bases 8 and 16 score 60.2% and 74.0%, bases 9 and 11 score 38.6% and 56.5%. A model that had only memorized base-10 sums would score near zero in base 9, since every problem was chosen to have a different answer there, so this is partial transfer, not none. The second is what repair looks like. Once the training data varied the function enough, the general procedure became what the network learned, and it then covered functions it had never seen.

The question this digest answers is how to get more of the second case. The argument runs as follows. Nothing generalizes beyond its training data without an assumption about what training and test share, and the training data cannot check the assumption. Given one, a model is right on a new distribution when two things hold: the model class contains a function that is right there, and the training data vary enough that every function in the class that fits them is also right there. The class decides how much variation is needed. On this digest’s reading of the theory, the information that fixes a model’s behaviour off its training data comes from three places: what is built into the class, including the computation the model runs at answer time; variation in the labelled training data along the axis the test will move on; and labels or checks obtained from the test side. Every measured success this digest could trace lands in one of these. The results that looked like more, from CLIP’s robustness to frontier scores on the Abstraction and Reasoning Corpus, rest on closer inspection largely on coverage, meaning that the training data already held examples along the axis the test moved on: by measurement for CLIP, and by the benchmark maintainers’ own judgement for ARC. That is the discouraging half of the answer. The other half is that covering an axis is cheap compared with covering its instances, and so is supplying the test side: a few tens of thousands of distinct training functions taught a procedure that handles a continuum of new ones, and sixteen worked examples in a prompt, at answer time, recovered most of what base 9 had lost.

What has to be true for anything to generalize

Suppose the rule linking inputs to outputs is the same at training and test time and only the inputs move. It is tempting to think the model will then transfer. It need not. Ben-David, Lu, Luu and Pál proved otherwise. Take any learning algorithm that sees labelled training data and unlabelled test inputs. Give it an unchanged rule, and add either a small distance between the two input distributions or the existence of one classifier that is good on both. The algorithm can still have a test error of at least one half, with probability at least one half. The reason is simple. When test inputs never coincide with training inputs, “the same rule” says nothing about the rule’s values at the test inputs, so the training data cannot say which of the many functions that fit them is the one that continues correctly. The standard bound for transfer states the same fact from the other side. It puts the test error below the training error plus a distance between the distributions plus the error of the best single function on both. That last term is an assumption, and checking it needs test labels. And when the test inputs lie off the training data and the model class can tell the two apart, the distance term reaches its maximum, so the bound says nothing. Ben-David, Lu, Luu and Pál, Ben-David et al.

So every claim that a method generalizes out of distribution is a claim conditional on an assumption about what training and test share, and the first question to ask of any such claim is which assumption, and who checked it.

Given such an assumption, that one function in the class is right on both distributions, the problem has a definite shape. A finite training sample is fit by many functions in the model class. They agree on the training data and disagree elsewhere, and the one that training returns is chosen by the architecture, the optimizer, the regularizer, the order and form of the data, and the random seed. D’Amour and colleagues measured the seed’s share. Fifty ResNet-50s trained on ImageNet with different seeds scored 75.9% on ImageNet with a standard deviation of 0.11 points, and 19.7% on pixelated ImageNet-C with a standard deviation of 2.4 points, individual models ranging from about 15% to 26% (read off the paper’s figure). A hundred BERT models fine-tuned on MultiNLI, a dataset of sentence-pair inference, scored 83.4% to 84.4% on its own test set. On HANS, a test built so that the shortcut “a hypothesis reusing the premise’s words is entailed” gives the wrong answer, the same hundred models scored about 57% to 67%, and the rank correlation between the two scores was 0.07. Two things follow. Equally accurate models do differ off distribution, by up to twenty times their in-distribution spread on the hardest stress tests and by two to nine times on milder ones. And in these stress tests the seed decided how badly a model failed, not whether it did: the drop every seed shared on pixelated images, 56 points, was more than twenty times the spread between seeds. A later section shows a case where the data left far more to chance. The function off the training data is set mostly by systematic properties of the class and the training, and partly by chance. D’Amour et al.

Many functions fit the same training data, and three things narrow them

1What the training data leave open

Every curve fits the training points and they disagree everywhere else.

2Vary the training data

Points along the axis the test will move on leave less room to disagree.

3Restrict what the model can express

If only straight lines are allowed, the fits agree everywhere. Right only if the truth is a line.

4Get labels from the test side

An answered example where the test lives pins the curves down there.

What the systematic part does is known in one controlled case. Xu and colleagues proved that a two-layer ReLU network trained by gradient descent, in the infinite-width limit, approaches a straight line along any direction away from the origin the farther it gets from the training data; in finite networks of two or four layers, more than 96% of fits along random outward rays had a coefficient of determination above 0.99. A network of this kind can therefore extrapolate a target function only if the target is itself close to linear along every direction out of the training range. Quadratic, cosine and square-root targets were not extrapolated; linear ones were. And a linear target is extrapolated when the training data cover every direction from the origin, and in their experiments usually not otherwise: with one input dimension held constant, or some dimensions only positive, the fit was usually wrong outside the data. Xu and colleagues read that second condition as what a spurious correlation does. When camels appear only in deserts, some combination of features is missing from the training data, the condition fails, and the model is free to extrapolate wrongly. Xu et al.

Putting the two halves together gives the principle the rest of this digest tests. A model generalizes to a new distribution when its class contains a function that is right there, and the training data vary enough that every function in the class that fits them is right there too. The two halves are not independent: the class decides which directions remain free, and the data must cover those directions. Xu’s graph networks show each half failing on its own. Asked for the largest degree in a graph, trained on graphs of 20 to 30 nodes and tested on 50 to 100, a network that adds up its node summaries must learn “maximum” with its internal layers, which continue linearly, and its error on the large graphs was 70.6% (single runs, read off the paper’s figures). Replace the sum with a maximum, so that the only nonlinear step is built in and the layers need learn only linear maps, and the error was 0.0%. Keep that architecture and train it only on path graphs, every one of which has largest degree 2 and smallest degree 1, so that the four degree statistics the theorem requires to vary (the largest and the smallest degree, each alone and each weighted by how many nodes have it) span only two of their four dimensions, and the error was 94.5%. The right class with the wrong variation fails too. Xu et al.

The information that picks the right function off the training data can come from three places, and the sections below take them in turn. It can be built into the class: the architecture, the regularizer, the invariances, and the computation the model performs at answer time, since a model that writes out intermediate steps is a different class of functions from one that answers in a single pass. It can come from variation in the labelled training data along the axis the test will move on. Or it can come from the test side: labelled examples of the new task, answered examples in the prompt, a checker that says whether an answer is right. Three things that look like further sources are not. Unlabelled test inputs by themselves add nothing, because the impossibility theorem above holds for learners that see them (the same paper shows a learner that reweights its training data toward the test inputs driven to the worst possible error on a task it cannot tell apart from one where the reweighting helps). And computation at answer time adds no information about the test distribution; it changes the form of the function the model computes, so it belongs to the first source, as a prior. Its benefit, if any, is that a function of that form is pinned down by less variation. And scale is not a source, as a later section shows: for ordinary trained models on most of the shifts tested, accuracy on shifted data is a fixed function of accuracy on the training distribution, and what lifts a model above that relation is the data it was trained on. Ben-David, Lu, Luu and Pál

Vary the axis the test will move on

The regression experiment above is the most controlled measurement of the second source, so it is worth working through in full. A task is a vector of eight numbers. A training run fixes some number of such vectors, drawn from a standard Gaussian. Every training sequence is built from one of them: sixteen random inputs, each labelled with the vector’s dot product plus noise of variance 0.25. The model is an eight-layer transformer with 128-dimensional states, trained to predict each label from the pairs before it. In the main experiment it sees 128 million training sequences whatever the number of tasks, which is four million per task when there are 32 and about 7,800 per task when there are 16,384. Two reference predictors bracket what is possible. The first is the best possible lookup: it knows the training vectors and assumes the test task is one of them, weighting each by how well it explains the examples so far. The second is ridge regression, which assumes nothing about the training set and is the best predictor for a task drawn fresh from the Gaussian. Error is mean squared error divided by eight, averaged over the sixteen positions. Ridge scores 0.344 on fresh tasks whatever the training set, since it does not use it. Raventós et al.

Training tasks	Transformer, new tasks	Lookup, new tasks	Transformer, its own training tasks	Lookup, training tasks
32	0.964	0.967	0.122	0.122
1,024	0.396	0.606	0.314	0.227
4,096	0.348	0.520	0.343	0.254
32,768	0.345	0.434	0.344	0.289
about 1,000,000	0.345	0.367	0.345	0.327

With 32 tasks the transformer is the lookup. It scores 0.122 on the tasks it has seen, far below the 0.31 that ridge, which stores nothing, scores on those same 32 tasks, and 0.964 on new ones, where the lookup scores 0.967. With 32,768 tasks it scores 0.345 on new tasks, within 0.001 of ridge, and pays for it on its own training tasks, where the lookup now does better (0.289 against 0.344). The lookup never catches up: with a million stored tasks it is still 7% worse than ridge on new ones, and the gap widens with dimension. The transformer stopped being a lookup long before the lookup became good. By 4,096 tasks it was within 0.004 of ridge while the lookup over those same 4,096 tasks was still half again as bad. The numbers in the table were taken from the coordinates of the published figure, and the paper reports no seeds for these figures. Raventós et al.

Why does this happen, when the training objective rewards the lookup? The lookup minimizes training loss; the table shows it (0.254 against the transformer’s 0.343 at 4,096 tasks). Two comparisons decide what a network ends up computing. One is which solution is cheaper at convergence, counting the training loss together with the penalty on weight size that training imposes: a lookup’s cost grows with the number of tasks it must store, and the procedure’s does not. The other is which solution is faster to learn, which decides what one sees at a finite budget. Adding distinct tasks moves both comparisons against storing, because each added task adds to the lookup’s size and to the time needed to store it, while the procedure’s size and learning time do not grow. On this account the number of distinct tasks is the quantity to vary, and it is the one these studies varied. Weight decay is the penalty term made explicit, and it acts as the account predicts: raising it from 0 to 1 lowered the number of tasks at which the general solution takes over from about 2^14.5 to about 2^10.5. The same comparison runs over training time in grokking, where a small network on modular arithmetic memorizes its table first and only later finds the general circuit: there the table is the faster solution and the circuit the cheaper one at convergence. In Nanda and colleagues’ setting the circuit appears only with weight decay or dropout, and the delay scales as one over the weight decay. Raventós et al., Nanda et al., Liu et al., 2022

Three refinements keep this from being overstated. First, the switch from lookup to procedure has two parts, and the table shows only one. At a fixed training budget the error on new tasks falls smoothly over seven doublings and is near ridge by 4,096 tasks. Raventós and colleagues then trained on two and four times as many sequences, by raising the batch size at a fixed number of steps, and found that the extra data pushed the model in opposite directions on the two sides of a threshold between 16,384 and 32,768 tasks. Below it, more data moved the model back toward the lookup: at 4,096 tasks, four times the sequences raised the error on new tasks from 0.348 to about 0.361. Above it, more data moved the model toward ridge. That is the two comparisons above seen in one experiment. At 4,096 tasks the general solution is the one found first and the lookup is the one training converges to, so the near-ridge behaviour in the table is temporary and more training erodes it; past the threshold the general solution is also the cheaper one, and more training improves it. A sharp switch in the limit of unlimited data per task is the authors’ extrapolation from three batch sizes. In a synthetic string-rewriting task with a million examples held fixed, the switch is sharp outright: fewer than 300 distinct rules never generalized to new rules and 1,000 or more always did. In reinforcement learning the curve is gradual. Cobbe and colleagues trained agents on the platform game CoinRun for a fixed 256 million steps, spread over different numbers of generated levels, and tested on new levels. Success was 66.6% with 100 levels, 84.8% with 1,000, 95.9% with 4,000 and 98.3% with a fresh level every episode. The gap between training and test levels fell from 32.8 points to 1.8. Across the sixteen games of the later Procgen benchmark, test return rose roughly linearly in the logarithm of the level count, and the gap closed somewhere between about 3,000 and 100,000 levels depending on the game, with several games still improving at 100,000 (read off the paper’s figures). One reconciliation, which none of these papers tests, is that each sub-skill switches sharply and the aggregate is smooth. Raventós et al., Zhang, Wang and Charton, Cobbe et al., 2019, Cobbe et al., 2020

Second, capacity pulls in two directions. In the regression setting a larger model needed more tasks: the four-layer model switched between 2,048 and 4,096 tasks and the eight-layer model between 16,384 and 32,768, and widening the four-layer model raised its threshold further. A bigger table is affordable, so the lookup stays competitive longer. Nguyen and Reddy, however, found a sharp switch at about 10,000 tasks in a model with room to memorize every task they tested. There both solutions fit, so what decided the outcome was which was learned first, not capacity. And in the two large-scale cases in this digest, bigger models needed less variation rather than more. In CoinRun the larger network closed the gap with about a quarter of the levels the smaller one needed, for reasons the paper does not examine, and the largest of the PaLM models below flattened earliest. Near the switch, which solution a run finds depends on its seed: runs split into two clusters, one storing and one generalizing. Raventós et al., Nguyen and Reddy, Cobbe et al., 2019, Kirsch et al.

Third, and most important, the general solution covers only what the training tasks span. Every “unseen task” in this literature is a new member of the family the training tasks came from: a new regression vector, a new handwritten character, a new level of the same game. Inside the family, meaning the distribution the training tasks were drawn from, the reach is wide. Goddard and colleagues trained on regression directions within 120 degrees of one pole, a cone which as a set of directions spans the whole space, and found the model accurate even near the opposite pole. Kwon and colleagues proved for a linear attention model that error off the span of the training tasks grows with the angle to that span, and falls to the noise floor within it. Outside the family the same kind of model fails. A transformer trained on sparse linear functions, whose vectors together span the whole space but each have only a few nonzero entries, scored 0.554 on dense ones where a dense-trained model scored 0.225. One trained on linear functions and on cosines fit each class but not their sum. And even within the family, scaling a regression model’s inputs to three times the training range raised its error from 0.0006 to 0.58. The variation that matters is variation along the axis the test moves on. Chan and colleagues showed that the axis can be subtle. A transformer learned to classify from its context only when two things held at once: the training sequences were bursty (the same class recurring within a sequence) and the classes were numerous, each shown in varied images. With burstiness fixed, more classes helped; with the class count fixed, more burstiness helped; and a class represented by a single repeated image never produced it. In their reading, natural text’s Zipf distribution over words, with a few frequent items and a long tail of rare ones, is what lets one model store the frequent items in its weights and still learn to read the rare ones from context. Goddard et al., Kwon et al., Yadlowsky et al., Garg et al., Chan et al.

The same principle, with the axis named, explains a result that was reported as something else. Chu and colleagues trained a language model on a card game, described in text, in which the rule for scoring face cards was fixed, then tested with a new rule. Supervised fine-tuning on the fixed rule took the model’s success on the new rule from 11.5% at the shared starting checkpoint down to 3.4%, while reinforcement learning took it to 15.0%, and the paper’s title says that supervised fine-tuning memorizes and reinforcement learning generalizes. Lin and colleagues reran it in their own training setup. Shown the new rule but scored by the old one, the fine-tuned Llama model still succeeded about as often as in distribution (a Qwen model did not), so it had learned to ignore the rule text, which never varied. Varying the rule during supervised fine-tuning, with no reinforcement learning, raised success on unseen rules from between 0 and 25% to between 64 and 72%. And the variation transferred only along the axis it varied: face cards worth 11 to 13, values the training hands never contained, stayed at 6 to 10%. The original comparison rests on single runs with no seeds, and its 15.0% is within the spread of the paper’s own training curve; the rerun chose its checkpoints on the test splits. The solid finding is the decline under a fixed rule. Chu et al., Lin et al., 2025

At the scale of real training sets, where the number of examples and the number of distinct tasks grow together, the separation is lost, and what remains measurable is where the curve flattens. Chung and colleagues fine-tuned PaLM on growing sets of instruction tasks at a fixed number of steps and measured a normalized average over four held-out benchmarks. The 8-billion-parameter model went from 6.4 with no tuning to 8.3, 14.8, 20.5 and 21.9 with 9, 89, 282 and 1,836 tasks; the 540-billion-parameter model went from 49.1 to 52.6, 57.0, 57.5 and 58.5. The last six and a half times as many tasks were worth about a point, and the largest model flattened earliest. The authors offer two untested explanations: that the added tasks were not very different from the earlier ones, or that the model was mostly learning to express what pretraining had already taught it. Three caveats apply: the 1,554 tasks added last came from one collection, each cell is a single run, and the checkpoint was chosen on the held-out benchmarks. If the second explanation is right, the variation that matters came from pretraining. And in the task-diversity studies of this section, from the regression experiment to CoinRun, held-out usually means a new task of the same kinds. Chung et al.

The vision literature on spurious correlations says how little variation along the right axis can be enough, and how much along the wrong axis is useless. Kirichenko and colleagues trained networks from scratch on images that paste a handwritten digit above a photographed object, with both predicting the label. With the correlation perfect, the object could be decoded from the representation at 62%, no better than from a network trained on the digits alone (61.4%); with 5% of training images breaking the correlation it was decoded at 90%, the optimum (both read off the paper’s figure). In a pairing where the second feature was as simple as the first, it was learned even under perfect correlation. In the Camelyon17 benchmark, where a pathology classifier trained on slides from some hospitals is tested on another, a single labelled slide from the test hospital added to the training set (which is the test-side source of a later section) moved accuracy from 71.0% to 82.9%. The benchmark iWildCam, by contrast, has over 200 training camera traps and still loses 16 points of F1 score, a combined measure of precision and recall, on new ones. Across these benchmarks, what decided the result was whether any training data broke the specific correlation that shifted, not how many environments there were. Kirichenko et al., Koh et al.

Build the axis into the model

The first source of information, what the model class assumes, shows most clearly on the axis of input length, because length is an axis no finite training set can cover. Liu and colleagues trained transformers to track the state of a simple automaton, parity of a bit string for instance, on sequences of length 40, where they reached 100%. Tested at lengths up to 128 their per-token accuracy fell to about 70%, which on the positions beyond 40 is close to the 50% that guessing would give. An LSTM trained the same way stayed at 100% at every length. The failure is not in the position encodings alone: randomly shifting the positions during training, so that every test position had been seen, improved length generalization but did not complete it, and removing position encodings gave similar results. The failure follows an internal quantity. At a fixed length, Anil and colleagues found a model’s parity accuracy fall to chance on strings whose count of ones it had never seen, and Liu’s transformers lost accuracy on such strings too. That fits one explanation the authors offer for what a shallow transformer builds: not the automaton run step by step but a parallel shortcut, here “count the ones, then take the count modulo two”, in which the count is a quantity that grows with length and the modulo is learned only over the counts the training data contained. Liu et al., 2023, Anil et al.

This is the first section’s theorem about ReLU networks again, carried by analogy to one learned sub-computation at a time; the theorem itself covers two-layer networks, not transformers. On this digest’s reading it gives the rule that organizes the length-generalization record: a model extrapolates as far as every quantity its learned sub-computations receive stays inside the range those sub-computations were fitted on. A sub-computation whose answer does not depend on magnitude, such as a comparison, is the exception that extrapolates on its own. The methods that have been shown to extend length generalization each do one of three things. They cover more of the range, they remove the quantity that grows with length, or they lower the error of each repeated step. Izzo and colleagues prove a version of this rule for small idealized transformers. Izzo et al.

Covering the range is what varying the training data does here, and the record says what counts: the range of training lengths, not the number of examples. Zhou and colleagues trained on addition problems of up to 10, 20, 30 and 40 digits and found the longest length at which accuracy held grew from 1.0 to 1.25 to 1.5 to 2.5 times the training length. In Delétang and colleagues’ parity test, by contrast, a transformer trained a hundred times longer at the same lengths stayed at chance beyond them. Zhou’s 2.5 times is the best of ten runs. At 100 digits those ten runs, all with the same near-zero training loss, scored about 98, 96, 94, 71, 68, 46, 28, 27, 4 and 0% (read off the paper’s figure), so the median run reached about 57%. That is the first section’s picture pushed to its limit: with nothing in the data to fix the function beyond the training lengths, the seed decided most of the outcome, from near perfect to none. Model size did not help: among Zhou’s models from 5 million to 268 million parameters the smallest did best, and Anil and colleagues’ models from 244 million to 64 billion failed alike on parity. Zhou et al., 2024, Delétang et al., Anil et al.

Removing the growing quantity is what architecture does. Delétang and colleagues trained a range of networks on formal-language tasks at lengths up to 40 and tested at 41 to 500. On reversing a string, a recurrent network with a stack reached 100%; on duplicating one, a recurrent network with a tape reached 100%; plain recurrent networks, LSTMs and transformer encoders scored 50 to 62% on both, which for these tasks is close to guessing, and a transformer encoder with 5 to 25 layers trained for up to ten million batches never left that range (an autoregressive transformer in the same study did solve duplication in its best run). Those perfect scores are the best of 30 runs, with means of 96 and 90 and large spreads, so the memory structure makes the right solution findable rather than inevitable. For addition, the growing quantity is the digit position, and two encodings that give each digit its place value rather than its position in the sequence hand the model the alignment its designer knows. One moves 21-to-100-digit accuracy from about 4% to 93% after training on up to 20 digits, and to 99% with two further changes; the other reaches 200 digits after training on numbers of up to 30 digits, with checkpoints chosen on 200-digit validation loss. Both count each number’s positions from a random starting index, so the indices seen in training run far past the numbers’ lengths, and both stop at the largest position index the training data ever used, which is the same rule once more. Delétang et al., McLeish et al., Cho et al.

Writing out intermediate steps belongs here too, as a change in the class of functions the model computes rather than as new information. Without intermediate tokens, a constant-depth transformer cannot track even the composition of permutations of five items at large length (Liu and colleagues’ example), unless a widely believed conjecture in complexity theory is false; with one written token per step, a single network can recognize any regular language at every length. That is an existence result, and learning is another matter. A scratchpad, meaning intermediate steps written out before the answer, did not by itself fix Liu’s transformers and was worse on bracket matching; what fixed them was a scratchpad together with attention biased toward recent tokens, which fixes what each step reads. Anil and colleagues found the same in large models: fine-tuned with a scratchpad on parity problems of up to 21 bits, a model wrote an entirely correct solution for about 8% of 30-bit problems (read off the paper’s figure); in separately trained models with the distracting earlier tokens masked, the solutions were perfect. Each written step is a sub-computation, and it generalizes when what it reads stays in range. The remaining limit is per-step reliability. A model that is right on 83% of 20-step problems is right on about 99.1% of individual steps, and step errors compound. In one example they compounded faster than independence would predict. With chain-of-thought prompting, PaLM at 540 billion parameters concatenated the last letters of two, three and four words at 99.4%, 94.8% and 63.0%; the per-step rate from the two-word case would predict 98.8% at four words, not 63.0%, though the four-word problems also differ in form from the two-word examples. The collapse of reasoning models on the Tower of Hanoi at eight disks is consistent with compounding errors, though no single per-move error rate fits the whole curve. The collapse survived the published rebuttals and a replication: the output limit the rebuttals cited comes to about 2,550 tokens of moves at eight disks, within the models’ budgets, and asking for a few moves at a time did not help. The sound part of the rebuttals was that the companion river-crossing puzzles, in which pairs of travellers must cross a river under constraints, were unsolvable above five pairs. Merrill and Sabharwal, Liu et al., 2023, Anil et al., Wei et al., Shojaee et al., Lawsen, Dellibarda Varela et al.

What this buys, and what it costs, follows from where the structure is needed. Data augmentation supplies an invariance for the transformations actually sampled; architecture supplies it for all of them. So built-in structure matters most on open-ended axes such as length, size and magnitude, which no sample covers, and least on bounded axes such as rotation, which a sample can. On a bounded axis the two meet: Brehmer and colleagues found augmentation matched an equivariant architecture only with small data and many epochs, and the comparison was made in distribution only. And a structural bet costs where it is wrong. On a shape-classification benchmark a model built to ignore rotation scored 60.1% against 78.1% for an unconstrained one, because the benchmark’s shapes were consistently oriented and orientation carried information; on handwritten digits the same constraint cost under a point; and on rotated test shapes the unconstrained model fell to 16.9%. The right structure for an axis is a statement about the test distribution, and like every such statement it can be false. Brehmer et al., Lawrence et al.

In most modern practice the class is not designed but inherited: it is whatever pretraining on broad data produced, and the fine-tuning procedure decides how much of it survives. Three results fix the picture. On the Waterbirds benchmark, a bird classifier is trained on images in which the background predicts the label and tested on images where it does not. Refitting only the last layer of an ordinary network, fine-tuned on Waterbirds from ImageNet weights, on about 500 examples with the correlation broken raised worst-group accuracy from 74.9% to 92.9%, above the 91.4% of group distributionally robust optimization, the method built for the problem. The feature the right rule needs was already in the representation. But it was there because pretraining put it there: features trained on ImageNet only, with just the last layer fit on the balanced Waterbirds examples, gave 88.7%, and the same method without pretraining gave 53.9%. Second, full fine-tuning can destroy what pretraining supplied. Averaged over ten distribution shifts, a linear probe on frozen pretrained features scored 66.2% out of distribution, full fine-tuning 59.3%, and fine-tuning that starts from the fitted probe 68.9%. Third, the destruction is recoverable by weight averaging: for CLIP’s largest model fine-tuned on ImageNet, averaging the fine-tuned weights with the zero-shot weights raised ImageNet accuracy from the fine-tuned model’s 86.2% to 86.8% and lifted the average over five shifts from 68.6% to 76.9%, above the zero-shot model’s 73.4%. The probe and the averaged weights add no new information; they keep what pretraining built while taking what fine-tuning adds, and a later section returns to what pretraining built. The refitted layer is different: it uses 500 new labelled examples, which is the test-side source of the section after next. Kirichenko et al., Izmailov et al., Kumar et al., Wortsman et al.

The same loss appears in language-model post-training, where it has been measured as forgetting. Chen and colleagues measured it across several models. A Llama model of 8 billion parameters trained on one benchmark’s questions with another model’s outputs as targets lost 38.5 points on unrelated tasks; trained with its own fresh samples scored by a reward, it gained 0.2. The freshness matters: in a Llama model of 1 billion parameters, data sampled once from the starting model and reused for two epochs cost 34.6 points on one benchmark, more than another model’s outputs did there, and in two Qwen models resampling each epoch removed most of the loss. What protects the inherited class is training on the model’s own distribution, which moves it less. On this digest’s reading the same closeness limits the gain: staying close to the base model helps only where the base already behaves well. Yue and colleagues’ sampling results bear that out: reinforcement learning raised a model’s single-sample accuracy on competition mathematics from 34.5% to 76.3%, while its accuracy with 256 samples moved from 96.2% to 95.4%, so the training sharpened what the base model could already produce. Chen et al., 2025, Shenfeld et al., Yue et al.

Facts stored in the weights are not procedures

The second source of information, variation in the training data, works on procedures: a rule learned from enough values of an axis covers the axis’s other values. It does not work the same way on facts stored in a network’s weights, and the difference explains a family of failures that looks like a failure of understanding.

Berglund and colleagues fine-tuned models on invented facts of the form “The composer of Abyssal Melodies is Uriah Hawthorne” and asked both ways. Asked who composed the piece, 96.7%. Asked who Uriah Hawthorne is, 0.1%. Their training set included 30 other facts stated in both orders, which the authors list among the things that did not help; Allen-Zhu and Li found the same with inverse questions in training for half of 100,000 invented people, after which inverse lookup on the other half scored 0 to 1.7%. So this is not a case of the data failing to vary the direction. A fact is not a point on an axis but an item. The reading these experiments support is that a transformer stores what it learned about an item for the role it was stated in, meaning the position the item occupied (the thing asked about or the answer), and, by Wang and colleagues’ circuit analysis of two-hop questions below, for the layer depth at which it was stored, so that a second step of reasoning, which runs in the upper layers, cannot reach a fact stored only in the lower ones. The storage is not absolute: in a small closed world of family relations, strong weight decay let a model infer held-out reverse facts from its weights at 0.98 to 1.0 in most seeds, where ordinary settings gave about 0.2. Put the same facts in the prompt and models reverse them at 97% to 100%. Frontier models show the asymmetry on real facts: in Berglund’s test GPT-4 named a celebrity’s parent 79% of the time and the celebrity from the parent 33%, and in Allen-Zhu and Li’s it recalled the next sentence of an Austen passage 65.9% of the time and the previous one 0.8%. Berglund et al., Allen-Zhu and Li, Kim et al., 2025, Lampinen et al.

Composition shows the same split between cheap and expensive. New combinations of familiar parts, each part in a position it has occupied before, are learned from few examples: on the SCAN benchmark of commands and action sequences, 4% of the commands gave about 93% on the rest, and a transformer in Ramesh and colleagues’ synthetic setting learned all 3,125 compositions of five functions from 100 of them, when it wrote each intermediate result out; asked for the final answer alone, it failed even with 2,000 of the compositions in training. A part never seen in a role fails in that role: SCAN’s “jump”, seen only alone, scored 0.08 to 1.2% in combinations, and adding 8, 16 or 32 combined examples raised it to 38.3%, 77.8% and 88.4%; functions applied in an order never trained on were at chance. Lake and Baroni’s meta-learning, which trains a 1.4-million-parameter transformer on a stream of episodes in which the benchmark’s words and actions are re-paired at random each time, so that it learns to compose from a few examples in context, cut the “jump” error from 99.27% to 0.22%, and left outputs longer than any in training at 100% error. It gained the variation its episodes rehearsed and nothing beyond it. On the COGS benchmark of sentence meanings, structures absent from training altogether, such as a prepositional phrase modifying a subject when training had them only on objects, scored 0.00 to 0.02 for every model tested. Lake and Baroni, 2018, Ramesh et al., Lake and Baroni, 2023, Kim and Linzen

Two-hop questions over stored facts are the stored-knowledge case of composition, and Wang and colleagues measured where it stops. A transformer trained on atomic facts and on two-hop questions over some of them answered two-hop questions over held-out combinations of trained facts well after a long grokking period, about fifty times the steps needed to fit the data, but questions built from atomic facts that had never appeared inside any two-hop example stayed at 0% through two million steps. Their circuit analysis puts the first hop in the lower layers and the second in the upper ones, and a fact reaches the upper layers only if it appeared inside a two-hop example during training. In a follow-up by Yao and colleagues, writing the bridge entity, the one that links the two facts, out as an intermediate step took that 0% to 99.4%. In the knowledge-manipulation experiments of Allen-Zhu and Li, comparing two people’s stored attributes went from 53.9% to 99.6% when the model wrote the attributes down first. Written steps bring the parts into the context, where a role is cheap, and extend depth by a bounded amount: one model trained on synthetic grade-school problems of up to 15 operations scored 91.8% at 20 and 76.8% at 23 (figure labels). Wang et al., 2024, Yao et al., 2025, Allen-Zhu and Li, Ye et al.

So the rule for stored knowledge is the rule for procedures with the axis replaced by the item: a fact is available in a role when that fact was trained in that role, or when it is in the context and a procedure the model has runs over it. Lampinen and colleagues turned this into a method. Fine-tuning Gemini 1.5 Flash on Berglund’s dataset gave about 0.00 on reversals; the whole dataset in context gave 0.98; fine-tuning on the dataset plus the model’s own in-context inferences about it (its reversals and deductions, written out and added to the training set) gave 0.99. On a richer invented hierarchy of categories the three scored 0.52, 0.78 and 0.90 on reversals. The hardest split held only one fact about each test category, its parent, so that a question may need a chain of more than two steps; there the three scored 0.33, 0.40 and 0.40 against a chance level of 0.25 (all six values read off the paper’s figure). A deduction that needs a longer chain than any the data rehearsed was barely available by any of the three routes. Lampinen et al.

Answers from the test side

The third source is information from the test distribution itself: answered examples of the new task, whether in the prompt or used for a few gradient steps, and checkers that say whether an answer is right. This is new information in the strict sense, and it is the one source the impossibility theorem leaves open: with enough labelled test examples, the training data stop mattering at all. Ben-David et al.

The reversal results above are the simplest case, because the same facts move from one source to another. The facts are the same; what changes is where they enter. Stored in the weights, a fact stated one way is largely unavailable the other way; read from the context at answer time, it is available both ways at 97% to 100%, because the model has a procedure for reading relations off text, which its pretraining data varied widely. Wu’s arithmetic shows the same source at work on a procedure instead of a fact. GPT-4’s base-9 addition went from 38.6% with no examples to 48.1%, 67.0%, 79.7%, 85.8% and 88.4% with 1, 2, 4, 8 and 16 worked base-9 examples in the prompt, while base 10 moved from 98.2% to 99.9%. One example already raised base 9 by nearly ten points, and sixteen recovered 84% of the gap to base 10’s no-example score with the curve flattening, so what the examples supply is a correction to a procedure the model already has, and the correction is partial. Berglund et al., Wu et al.

The same two-digit addition in five number bases

GPT-4, accuracy on 1,000 problems per base. Filled bar: asked to work step by step, no examples. Outlined bar: the same request after sixteen worked examples in that base.

Base	Accuracy	Knows the base
8	60.299.9	98%
9	38.688.4	90%
10the usual one	98.299.9	100%
11	56.586.9	91%
16	74.088.7	100%

The Abstraction and Reasoning Corpus, whose first version is called ARC-AGI-1 and whose second and third versions are discussed below, is where this source has been pushed hardest, because every task in it is a new rule shown through a few demonstration pairs, and the demonstration pairs are labelled examples from the test task. Akyürek and colleagues took a Llama model of 8 billion parameters, fine-tuned on synthetic tasks of the same kind, and at test time trained a small adapter for each puzzle on its own demonstration pairs, with the pairs rearranged and geometrically transformed to make more of them, then produced an answer under each transformation and voted. On the public evaluation set this took the model from 18.3% to 47.1% of puzzles solved in two attempts. The ARChitects, who won the 2024 competition with the same idea, give the fullest breakdown of their own pipeline on the same set: 18.3% for the fine-tuned model (by coincidence the same starting figure), 44.5% after training on each task’s demonstrations, 62.5% after sampling one answer under each of sixteen transformed versions of the task, 67.6% after ranking candidates by their probability across fresh transformations, and 71.6% after a depth-first search over high-probability tokens. Akyürek et al., Franzen et al.

Using the test task's own examples, then its family's symmetries, on the Abstraction and Reasoning Corpus

Share of 400 public evaluation tasks solved in two attempts, as each stage is added to the previous ones, one run per stage. The label after each bar says where the stage's information comes from.

1. Fine-tuned model, two sampled answers18.3
2. Train on the task’s own demonstration pairs44.5 +26.2 labels from the test task, with transformed copies
3. Answer under sixteen transformed views, keep the two most probable62.5 +18.0 a prior about the task family
4. Rank by probability across fresh transformations67.6 +5.1 a prior about the task family
5. Depth-first search over high-probability tokens71.6 +4.0 computation

• Same team’s 2024 competition system, semi-private set of the first benchmark56.0
• That system on the second benchmark at its launch2.5

Three things in that ladder say what the test-side source is and is not. First, the step that uses the test task’s own labels, training on its demonstrations, is the largest single step, 26 points, and it is the one the impossibility theorem permits. That step already trains on transformed copies of the demonstrations, so two sources are mixed inside it, and since the ablation is cumulative and single-run, each step’s size depends on its order. Second, the next largest step, 18 points from transformed versions of the task, is a prior: the knowledge that rotations, flips and colour permutations preserve an ARC rule is a statement about the task family, true of this family and not of others. Fitting a network to a single puzzle with no such prior scores 0.75% (with puzzles whose output size differs counted as failures); a network built around the family’s symmetries, trained with no pretraining at all to find the shortest description of the puzzle, scores 20%. Third, the whole pipeline is tied to the family it was built for. The benchmark keeps two test sets its developers never see, a semi-private set for scoring systems that call external models and a private set for the competition, and this digest calls both hidden. The same team’s 2024 competition system scored 53.5% on the first benchmark’s private competition set and 56.0% on its semi-private set, both under the competition’s compute limit, and 2.5% on the second benchmark at its launch, under that benchmark’s 5% noise floor; a year later the winning entry on that benchmark, built on the same approach of synthetic tasks and test-time training, scored 24%. Public scores on the first benchmark run above hidden-set scores for nearly every system, by up to 18 points (Akyürek’s ensemble with a program-synthesis system scored about 62% public and 47.5% semi-private). The public set is easier, was used for development and has been public since 2019, so the gap cannot be split among those causes. Akyürek et al., Liao and Gu, Franzen et al., Chollet et al., 2025, ARC Prize Foundation, 2025

The same source appears in the spurious-correlation literature under two other names. Refitting the last layer on 500 examples that break the correlation is test-side labels applied where they are cheapest. And model selection is test-side labels too: on the spurious-correlation benchmarks, the published gains of nearly every method depend on choosing hyperparameters or checkpoints with group-labelled validation data from the shifted distribution, which was worth 10 to 24 points to every method. On the DomainBed benchmark, with hyperparameters chosen on validation data from the training domains, plain training scored 67.0 on the seven-dataset average and the best of eight alternatives 67.2. (The authors’ later table with thirteen alternatives has 66.6 against 67.5.) Under the other selection rule that never sees the test domain, the best lead was 1.8 points. On the ten datasets of the WILDS benchmark the invariance and robustness objectives tied or lost on nine. An invariance objective is a bet about which features will stay predictive, which the first section counts as part of the class; on these two benchmark suites the bets did not beat plain training on average, with gains on single datasets (CivilComments, one office-image benchmark) and losses on others. Kirichenko et al., Idrissi et al., Gulrajani and Lopez-Paz, DomainBed results table, Koh et al.

Two limits bound this source. Unlabelled test inputs alone are not part of it: the impossibility theorem holds for learners that see them. In the original test-time-training work, adapting on a single unlabelled image gained about 2 points of accuracy averaged over fifteen image corruptions and a stream of thousands about 13, and both gains came through a self-supervised task whose relevance to the shift is an assumption, which makes them a case of the first source. And a checker, which is part of this source, amplifies only what the model already produces. Brown and colleagues drew up to ten thousand samples per problem: on a software-repair benchmark with an exact test, solved problems went from 15.9% at one sample to 56% at 250, roughly linearly in the logarithm of the sample count; on competition mathematics, where the checker was a majority vote or a learned reward model, 98.4% of 128 problems had a correct answer among ten thousand samples (lucky guesses included) and the selectors found it for 41.4%. Snell and colleagues matched compute between a small model given more samples, which had also been fine-tuned on the benchmark’s training set, and a model about fourteen times larger given one: the small model won on the easier questions at generous budgets, and on the hardest both were near zero (about 6% against 0 to 4%, within noise on about a hundred questions). Sampling with a checker finds what is in the model’s support; it cannot put something there. Sun et al., Brown et al., Snell et al.

Scale moves both accuracies together

Where does scale sit among the three sources? The vision record answers with one measurement repeated across hundreds of models. Miller and colleagues plotted out-of-distribution accuracy against in-distribution accuracy for 219 ImageNet models on a replication test set collected the same way as the original, and after a probit transform the points lay on one line with a coefficient of determination of 0.996. On CIFAR-10 and its replication set, changing a model’s size, its optimizer, its training length, or the fraction of the data it saw moved it along the line. Taori and colleagues had earlier asked what moves a model above the line, which they call effective robustness, across 204 published models: robustness interventions rarely did, and some models trained on far more data gained 1 to 2 points on the replication set and up to 29 on a test set of renditions (drawings, sketches and cartoons of the same objects), while others gained nothing. The model family that sat well above the line on every shift was CLIP: at 76.2% on ImageNet, matching a ResNet-101, its zero-shot classifier was about 33 points higher on the average of four shifts (an average computed for this digest from the paper’s figure), and fitting it to ImageNet narrowed that gap (a linear head gained 9.2 points on ImageNet and lost 4.7 on renditions). Miller et al., Taori et al., Radford et al.

Fang and colleagues then asked which of CLIP’s differences caused the gap. They ruled out language supervision at training time: on the same ImageNet images, contrastive language-image training and ordinary classification landed on one line. They ruled out a pretrained language encoder and the prompts. And an image-only model trained contrastively on the same web images, with class labels assigned by matching words in the captions and no language model at all, matched CLIP’s robustness (35.7% on ImageNet against 37.9%, 18.8 against 19.9 on the average shift). What remained was the training distribution. Two later measurements say what about it. Shi and colleagues measured CLIP’s effective robustness, on standard ImageNet test sets with their own measure, against a baseline that accounts for its training distribution as well as ImageNet: across many models trained on the LAION web collection the average gap fell from 5.96 points to 0.00, and across models trained on the YFCC collection from 2.59 to 0.77. Mayilvahanan and colleagues removed renditions from LAION, a web image collection, and trained on photographs only: accuracy on rendition test sets fell from 40.7% to 22.0%, with accuracy on photographs unchanged. On this evidence CLIP sat above ImageNet’s line because its data contained the test sets’ axes, and measured against its own data it sits on or near a line like other models; the removal experiment was run for renditions only. Two other candidate causes were ruled out: fewer ImageNet classes left effective robustness unchanged in Taori’s tests, and mixing two web sources gave a line between the two sources’ lines rather than above both. Fang et al., Taori et al., Shi et al., Mayilvahanan et al., Nguyen et al.

So scale is not a source. It raises in-distribution and out-of-distribution accuracy together, which is worth having, and it does not change the relation between them. What lifts a model above its line is coverage of the shifted axis by the data. What keeps it there is a fine-tuning procedure that preserves the pretrained class, which is why the probe and the averaged weights of the earlier section leave the line; the refitted readout leaves it with new labels. The line is not universal: Miller and colleagues found that on the pathology benchmark Camelyon17, models at 95% in-distribution ranged from about 50% to 95% out of it, and Teney and colleagues found datasets on which the two accuracies trade off. Language models show the same shape, with zero-shot and in-context predictions above the fine-tuned line and more fine-tuning examples lowering a model toward it. Miller et al., Teney et al., Awadalla et al.

Reason or recite

The controversy is whether large language models solve problems by running procedures, which is what reasoning would mean, or by retrieving and adapting answers to problems like those in their training data, which is reciting. The argument so far lets it be stated as an empirical question with several answers rather than one. The question is: along which axes do the training data of a frontier model vary, and along which do they hold still? Where they vary, the model has a procedure and uses it; where they hold still, performance drops by an amount that depends on how far the test item sits from what was varied. Both sides of the debate describe real behaviour of the same models on different kinds of test item, and the results of the last three years say which kinds are which. Wu et al.

Fresh items from a familiar distribution cost frontier models little. On a grade-school arithmetic set written from scratch to match a public benchmark, frontier models moved by at most about 2 points in either direction while small open models lost up to 8. Changing the names and numbers in the public benchmark’s problems cost GPT-4o 0.1 points and the first o1 model 3.3, against a standard error of about 2 on a hundred-question set, and whether such drops count as significant depends on the test used: under the authors’ own test most do, under a two-sample test they do not. Changing names alone cost nothing even for a small model. Frontier models released before the 2026 American Invitational Mathematics Examination scored 92 to 98% on it, by the MathArena live table in October 2026. Whatever these models do on a new problem of a familiar kind, it is not a lookup of that problem. Zhang et al., Mirzadeh et al., Balunović et al., MathArena

Short changes to a stated rule, which broke models in 2023, largely do not break reasoning models, the models trained to write a long chain of thought before answering. McCoy and colleagues gave the first o1 preview a Caesar cipher with a shift of 12 instead of the shift of 13 that appears in text everywhere: it decoded 0.90 of messages against 0.92 for the familiar shift (read off the paper’s figure), where GPT-4, run without chain of thought, had scored 0.00 against 0.51. The gap came back on hard items, 0.27 against 0.47, and on long procedures run in an unfamiliar notation: a Gemini thinking model fell from 99% to 56% on multi-digit addition when each digit was replaced by a letter. No one has rerun Wu’s full suite of counterfactual tasks on a reasoning model, and the base-9 figure in this digest’s opening is a 2023 result. The summary the evidence supports is that reasoning models extend how long an unfamiliar procedure can be run before it fails, without removing the limit. McCoy et al., Yang Yan et al.

What still fails, and fails by large margins, is the link between a problem’s surface and the method it calls for. Kai Yan and colleagues took classic puzzles and altered them so that the classic answer is wrong: o1 fell from 86% on the originals to 30% on the altered versions, and DeepSeek-R1 from 86% to 23%, and longer thinking did not help. One irrelevant clause added to a grade-school problem (“five of them were a bit smaller than average”) cost the first o1 model 17.5 points and GPT-4o 32. Variants of competition problems that keep the original method cost reasoning models about nothing; variants that break it cost 14 to 16 points, though the method-breaking variants are also harder, so that is an upper bound on how much is anchoring to the familiar method. A test of this kind has not yet been given to people, and the clause that misleads a model is written to sound relevant, so part of what it measures is how strongly the convention “every stated quantity matters” is held. But that convention is itself a link between surface and method that training data never varied, which is what the argument above predicts would fail. Kai Yan et al., Mirzadeh et al., Huang et al., 2025

Two results say what the procedures are made of. Ruis and colleagues traced which pretraining documents most influenced a model’s answers to reasoning questions, and found them shared across questions of the same type, stating a procedure (code, worked arithmetic), and rarely containing the answer; the result is correlational, on two models and three simple tasks. Saparov and colleagues trained small transformers to find paths in graphs. Lookahead here is how many steps ahead a model must look to choose the right next move. Within the range of lookahead the training data covered, balanced data taught search that held at every lookahead, where a distribution with few long lookaheads taught a shortcut. Beyond that range, a model trained to lookahead 12 held to 14 and was near chance by 18 (figure labels), which is the length result of the earlier section in a new task. Ruis et al., Saparov et al.

The benchmark built to settle the question has, in its maintainers’ judgement, been reached with the help of coverage they cannot precisely quantify. The Abstraction and Reasoning Corpus was designed so that each task is a new rule, with a private test set and tasks unlike anything on the web. Its first version is now scored above 97% by several frontier models at under a dollar a task; on its second version, where the best 2025 competition entry scored 24% on the private set and the strongest 2025 model 54%, the best score is now 95%. The maintainers’ 2026 report says how. A model with “non-zero fluid intelligence” can be trained on an enormous number of tasks that densely sample the task space, “possibly” generated automatically, and if the public training set and the private test set are too similar, that is enough to shortcut the benchmark; one way is to generate tasks, solve them, verify the solutions and train on the traces, “enough to provide high coverage density of the target domain at training time, thus considerably reducing or outright removing the need for test-time adaptation”. “We believe this has happened to ARC-AGI-1 and ARC-AGI-2 with frontier LRMs – either incidentally or intentionally”, they write, LRMs being their term for large reasoning models, and their 2025 report adds that they cannot precisely quantify the effect. One leaderboard number fits that reading: a model that scores 95% at its highest reasoning setting scores 59.6% on the second benchmark at the setting the leaderboard labels “None”, where most 2026 models score 0 to 2%. The opposing observation sits beside it: reasoning effort moves most models on the same benchmark from near 0% to 70 to 90%, which says computation at test time matters too. The 2026 report also states this digest’s argument in the maintainers’ words: language models “remain bound to task-specific training, albeit now over task-specific reasoning chains instead of the literal task data.” Their third benchmark is interactive, with most of its mechanics absent from the public set, and it put every frontier model under 1% at launch in March 2026. By September one model scored 62.7% under the standard evaluation setup and 99.9% under an adapter supplied by its provider that carries the model’s hidden reasoning state between steps. The published record cannot yet say how much of that recovery is coverage of the new family and how much is computation at test time. On problems new to everyone, the reach of computation at test time can be measured directly: on 68 open problems in mathematics whose solutions are checked by a proof assistant, the strongest model solved 2, and a far larger budget of attempts, which the authors say not to read as a score, raised that to 5. The leaderboard figures in this paragraph are from the ARC Prize site as of October 2026. ARC Prize Foundation, 2026, ARC Prize Foundation, 2025, Chollet et al., 2025, ARC Prize leaderboard, Adamczewski and Bloom

What to do

The question was how to make models generalize further beyond their training distribution. The record answers it with a rule that has three clauses and a limit, and the digest closes by stating each with its status.

To generalize along an axis, vary that axis in the training data. This holds in the controlled tests the digest found, with the caveats given above (single seeds in several, checkpoints chosen on test splits in one): the regression experiment, CoinRun’s levels, the string-rewriting rules, the card game’s rule text, the 5% of counterexamples that made a network learn the photographed object as well as the digit. What counts is coverage of the axis the test moves on; the number of distinct tasks is the best measure of it in the controlled studies, and the number of examples, or of environments that do not vary that axis, is not. The variation needed is set by the model class, and it buys the axis’s other values, not other axes: face-card values never dealt stayed unsolved, and sentence structures never seen in training stayed at zero. This is the cheap option, because finite variation teaches a procedure that covers a continuum, and the large-scale results examined here used it: CLIP’s renditions by measurement, and ARC’s task space by its maintainers’ judgement.

Where the axis cannot be covered, build it into the class. Length, size and magnitude are the clear cases, and the measured fixes all remove the quantity that grows (recurrence, memory, position relative to the number’s end, written steps with fixed retrieval) or extend the range a sub-computation was fitted on. This is measured for the synthetic tasks and is interpretation beyond them. It costs where the structure is wrong, by amounts that range from under a point on handwritten digits to 18 points on oriented shapes, and in the modern setting most of the class is inherited from pretraining, so the practical form of this clause is to keep what pretraining built: probe before fine-tuning, average weights back toward the pretrained model, train on samples drawn fresh from the model as it trains.

Where neither is possible, get labels or checks from the test side. A few answered examples in the prompt, a few gradient steps on the test task’s own demonstrations, a readout refitted on examples that break the correlation, a checker over many samples. It can be very cheap: in one benchmark a single labelled slide from the test hospital, added to training, did what two hundred training camera traps could not in another. This is the one source the theory leaves fully open, and it is what moved the 2024 ARC competition. Its limits are measured too: unlabelled test inputs alone add nothing; examples correct a procedure the model has rather than supplying one it lacks; a checker finds what is in the support and cannot put something there; and the whole apparatus is tied to the task family whose symmetries it assumed.

The limit is the one the first section stated. Nothing in the published record generalizes beyond what one of the three sources covered: the span of what the training data varied, the axis the model class was built to extend, or what the test side supplied. The results that seemed to go further rested largely on coverage: CLIP’s robustness by measurement against the right baseline, and frontier scores on ARC by its maintainers’ judgement. The open questions are therefore specific. Whether natural data produce a switch from storing to generalizing as sharp as the string-rewriting rules did is untested; the regression curve at a fixed budget is smooth and CoinRun’s is gradual, and both are synthetic. Whether the general solution, once found, persists under much longer training is untested too, and the cheapest test is a long run just above the regression experiment’s threshold. Whether reasoning models have closed Wu’s counterfactual gaps is untested, because no one has rerun the suite. Whether a human panel also fails the irrelevant-clause problems is untested, and would decide what that benchmark measures. And how much of the 2026 recovery on interactive ARC is coverage and how much is computation at test time cannot be split from the public record. A reader who wants a model to understand something out of distribution should ask which axis the something moves along, whether anything in training varied it, and if not, which of the other two sources can.