Evaluating mid-training
· 53 min read
Digests are written primarily by LLMs with only readability edits by me. If this one reads like slop, please shoot me an email at max.zhuang@berkeley.edu.
Kornblith, Chen, Lee and Norouzi trained eight ResNet-50 image classifiers on ImageNet with ordinary softmax cross-entropy and eight with cosine softmax, a loss that normalizes the features and the class weights before comparing them. The architecture, data and optimizer were shared; each loss had its own stopping epoch, chosen on held-out ImageNet images, and the cosine loss its own temperature. On ImageNet the cosine-softmax models were better: 77.9 percent top-1 accuracy against 77.0, a statistically significant difference. The authors then used the models as starting points for eight other image datasets, in two of the ways the paper tests. When a linear classifier was fitted to each model’s frozen features, the softmax models were better by about ten points on average (75.95 against 65.84 percent), and by 23.6 points on Stanford Cars. When the whole network was fine-tuned instead, with hyperparameters tuned separately for each loss and dataset, softmax led by about a point (86.2 against 85.1 percent, three pretraining runs each), most of it on one dataset, and the authors describe the choice of loss as having little effect. Kornblith et al.
Cosine softmax minus softmax, in percentage points. A bar to the right favors cosine softmax; to the left, softmax.
One pair of training recipes received three verdicts: one from the checkpoint’s own score, and one from each of two ways of using it. Every comparison of mid-training recipes is in the same position. Mid-training, the late stage of pretraining in which the learning rate finishes decaying on a curated data mix, produces a model that is not deployed. It is the starting point of supervised fine-tuning, reinforcement learning (RL), or another stage, which this digest calls the next stage. The mid-training digest describes what the stage contains. This one asks what it means for one recipe to be better than another, how to measure it, and which answers hold across fields that train in stages: language models, vision, speech, and reinforcement learning.
What “better” means
A mid-training recipe is better than another, for a particular next stage, when the pipeline that starts from it ends better. Four things have to be fixed before that sentence has a definite answer.
The first is the next stage itself: its algorithm, its data, its budget, and how its hyperparameters are chosen for each starting checkpoint. The opening example shows the next stage changing the verdict. Reinforcement learning from an offline start shows how far the next stage alone can move the result. In Nakamoto and colleagues’ experiments, policies pretrained offline with conservative Q-learning (CQL) averaged a normalized score of 42 out of 100 over 11 tasks. Fine-tuned online with CQL, they averaged 71. Pretrained the same way and then fine-tuned with soft actor-critic, a method without CQL’s penalty on actions outside the offline data, they averaged 29, below where they started (the fine-tuning curves use six seeds). Nakamoto et al.
The second is the evaluation protocol: the benchmarks, and also the prompt template, sampling temperature, number of samples, length limit and answer extraction. Sampling temperature, the length limit and the prompt template have each moved reasoning scores by as much as many published training gains; temperature alone moved AIME 2024 scores by up to 15 points. When each problem is sampled once, changing only the random seed does the same. In the OctoThinker study of mid-training before RL, one mid-trained checkpoint scored 0.2 or 33.0 on MATH500 at the start of RL (read off the paper’s figure), depending only on the prompt template. OctoThinker, Hochlehnert et al.
The third is the cost that is held equal, and it has to be named. Comparisons of language models usually hold training tokens or compute equal. Offline-to-online RL holds environment interactions equal and does not count offline compute; work that reuses an earlier agent treats that agent’s compute as already spent. The choice can decide which pipeline wins. Ball and colleagues’ RLPD (reinforcement learning with prior data) trains online from scratch with the offline data mixed into every batch. Counted in environment steps, it beat pipelines that pretrained offline first on sparse-reward maze and hand-manipulation tasks, and matched them on locomotion. Counted in gradient steps, it was the more expensive: it took 20 per environment step where the baselines took one, about 5 to 20 times as many in total even counting the baselines’ offline training (an estimate from the reported settings; the paper does not count them). Held to one gradient step per environment step, a version of the same agent fell behind the pretrained pipeline on both large mazes (read off a figure). Ball et al., Agarwal et al.
The fourth is how the outcome is summarized: the score after a stated budget of the next stage, or the whole learning curve. Offline-to-online papers report both, and the two can disagree: in Nakamoto and colleagues’ tables, CQL ends higher than IQL, another offline method (71 against 69), but falls further short of a perfect score averaged over the run (a cumulative regret of 0.41 against 0.38).
With those fixed, one recipe is better than another without qualification when, after the next stage, it is at least as good on every evaluation that matters (each target evaluation, each capability that should be kept, each other target domain) and better on at least one by more than the run-to-run noise of the whole pipeline. Any other result is either a tie within noise or an exchange, more on one evaluation for less on another, and should be reported at the setting where it was measured.
One consequence of the definition runs through the rest of this digest. A measurement on the mid-trained checkpoint, taken before the next stage, is a prediction of the pipeline’s result, and it is valid only where it has been checked against that result.
Equal-cost controls
A comparison is only as strong as the alternatives it includes. Four are standard, and each changes how a published result reads.
- The same learning-rate schedule on the original data, which separates the decay from the data. The OLMo 2 example below is this control.
- The same number of extra steps on ordinary data, which separates the content from the extra training. In Hu and colleagues’ study of pre-pretraining, 500 steps of a formal language before ordinary pretraining lowered a 160M-parameter model’s text loss at the end of that pretraining by 0.039 nats, against no first stage. Five hundred steps of ordinary text in the same position lowered it by 0.026, two-thirds as much, so most of the gain came from the extra steps. In Yamaguchi and colleagues’ larger setting the verdict changed. At 500M to 3B parameters, with a batch about 30 times larger in tokens, so that the same 500 steps carried about 30 times more formal-language text, and a downstream benchmark average as the measure, the ordinary-text control reproduced at most about a third of the gain (at 1B, 0.4 of 1.4 points). There most of the gain belonged to the formal language; which of the changed conditions made the difference was not tested. Hu et al., Yamaguchi et al.
- The same data placed elsewhere: mixed into pretraining from the start, or given to the next stage. In the two direct tests found, mixing into pretraining did slightly better than a separate later stage: 79.9 against 78.1 on SuperGLUE for T5-base, and a word error rate of 13.19 against 13.91 in speech at the largest amount of added data, where the speech authors describe the two strategies as performing similarly overall. In both, the separate stage used more compute and restarted training after a finished run. Neither tested a shift of the data late within one learning-rate schedule, which is what language-model mid-training does. Aribandi et al., Hsu et al.
- No intermediate stage, with its compute given to the next stage. In the one example found that holds total compute fixed, Zhang, Neubig and Yue’s 100M-parameter models on synthetic arithmetic, giving the compute to RL instead of mid-training won at the larger budgets (see the section on the next stage’s budget).
OLMo 2 shows how much the first control can absorb. The 7B model’s checkpoint at 4 trillion tokens, before its learning rate was decayed to zero (by the stated schedule it was then about a fifth of its peak), scored 69.6 on the multiple-choice average of OLMES, the evaluation suite used for the OLMo models, 63.2 on its generative-task average, 59.8 on MMLU, and 28.5 on 200 problems the authors drew from the test set of GSM8K, a grade-school math benchmark. A 50-billion-token decay of the learning rate to zero on a mix of filtered web text (58 percent), math (24 percent), academic and encyclopedic text (11 percent) and instruction data (7 percent) raised these to 75.7, 70.2, 63.1 and 46.5. The same decay on the ordinary pretraining mix gave 74.0, 64.5, 61.8 and 27.0.
Measured against the checkpoint before the decay, the new mix gained 6.1, 7.0, 3.3 and 18.0 points. Measured against the same decay on the original data, it gained 1.7, 5.7, 1.3 and 19.5. The decay accounts for 72 percent of the multiple-choice gain and 61 percent of the MMLU gain, but only 19 percent of the generative gain; on math, all of the gain is the data. Each of these is a single run. The math score is out of 200 problems, which gives a standard error of about 3 points, so the 19.5-point gain is clear. The multiple-choice and MMLU gains of 1.7 and 1.3 points come from much larger benchmarks, but with one run each nothing separates them from run-to-run variation. The mix contained GSM8K’s training split, but two mixes with the same share of it and less math text (about 6 percent of the mix, against 24) scored 30.5 and 28.5 on the math problems, against 27.0 for the decay on the original data and 46.5 for the full mix, so the training split alone did not produce the gain. OLMo 2
- Decaying the learning rate on the original data
- Switching to the new mix, beyond that
Points of accuracy · each blue bar starts where the amber bar ends · OLMo 2 7B from its 4-trillion-token checkpoint · each decay 50 billion tokens · one run each
The baseline with no intermediate stage matters because a stage can also leave the next stage worse off. Training first on another task lowered the target score in about half of the intermediate–target pairs tested: 62 of 110 for RoBERTa-large (means of three restarts), 534 of 1,056 for BERT-base with full data in both stages (single runs), and 203 of 462 for adapters on RoBERTa-base with targets of at most 1,000 examples (the first two counted for this digest from the papers’ figures and tables). A count with no threshold for noise cannot separate small losses from noise, so these counts show that a gain from the intermediate stage was far from assured, not how often it hurt. A clear loss comes from vision. A ResNet-18 trained with plain SGD at a constant learning rate and no data augmentation, to convergence on half of CIFAR-10 and then on all of it, reached 51.7 percent validation accuracy, against 56.2 when trained from scratch on all of it, though both fit their training data. Pruksachatkun et al., Vu et al., Poth et al., Ash and Adams
A gain can also be reported in compute, as the extra training a baseline would need to match it. Davidson and colleagues call this the compute-equivalent gain, with the baseline training further on its original data. Every such measure is a horizontal distance from the candidate to a baseline curve, so it depends on the baseline, on where along the curve it is read, and on the curve’s slope. In Hu and colleagues’ study above, the 500 formal-language steps saved 7.15 later training steps for each step they took, against no first stage, in language-modeling loss at 160M parameters, read at 10,000 steps of later training. Five hundred steps of ordinary text in the same position saved 6.65 for each step. By its content, an ordinary-text step is worth one ordinary-text step, so most of such a multiplier comes from the position of the extra steps and from where the saving is read on the baseline’s curve, and the formal language’s own contribution is small. Davidson and colleagues warn that a large compute-equivalent gain can mean only that more training had stopped helping the baseline. The defensible report is the pair of curves, or a multiplier with its budget, evaluation and noise attached. Davidson et al.
What a checkpoint’s own scores can tell you
Whether a score measured on the checkpoint predicts the pipeline’s result depends on how the candidates differ, and on what the score measures. Kornblith, Shlens and Le retrained 16 image architectures with one shared recipe. Across them, ImageNet accuracy predicted average transfer accuracy over 12 datasets with a correlation of 0.99 for linear classifiers on frozen features, and 0.96 after full fine-tuning (Pearson correlations, on accuracies in log odds). Across publicly released checkpoints, each trained with its own regularization, the frozen-feature correlation was 0.37 and not significant. Within each of five Inception architectures, regularization settings that changed ImageNet accuracy by under a point moved frozen-feature transfer by most or all of the gap between the worst and best of the 16 architectures. Kornblith, Shlens and Le
Language-model studies show the same weakness, less cleanly. Yano and colleagues pretrained 1B and 8B models under seven learning-rate schedules, one never decayed and six decayed, one run each. The base models’ average over seven zero-shot tasks did not predict the schedules’ order after supervised fine-tuning: the rank correlations were −0.57 at 1B and −0.07 at 8B, neither distinguishable from chance with seven runs. The same paper also ran eight pipelines, four pretraining schedules each followed by a mid-training stage whose learning rate was held or decayed to zero. Their scores after mid-training predicted the order after fine-tuning at 0.98 (1B) and 0.83 (8B), but two benchmarks, GSM8K and DROP, appear in both averages; without them the correlations are 0.56 and 0.99. At both sizes the pipeline that led before fine-tuning came second after it. These correlations are computed for this digest from the paper’s tables. Yano et al.
A measurement built for the next stage ranked very different models well. Yu and colleagues scored each of ten base models from seven families by the per-byte likelihood it assigned to the decisive action in a strong agent’s successful attempt at a coding task, the first action after which the task’s tests pass. That score ranked the families’ released post-trained models on SWE-bench Verified at a rank correlation of 0.964 when the attempts came from a different source (DeepSWE), and 0.915 when they came from SWE-bench Verified itself. Which action counts as decisive, and which tokens are scored, were chosen on the same ten models, and each family’s score comes from its own lab’s post-training. Yu et al.
Across the reviewed studies, general-purpose scores on the checkpoint, such as benchmark averages, loss and single-try accuracy, mostly ranked candidates well when the candidates differed widely: in size, in architecture under one recipe, or in how far training had gone. Shen and colleagues’ 36 chess models, at four sizes and several training lengths on one data distribution, were ranked after RL at 0.93 to 0.99 by their pretraining loss, depending on the RL compute at which the outcome was read. Nine checkpoints along one cosine learning-rate schedule in Bansal and colleagues’ study kept their order through RL on GSM8K (a rank correlation of about 0.98, computed for this digest from values read off a figure). The one exception found among checkpoints of one run, Springer and colleagues’ below, was small. Those scores were least reliable when the candidates were different procedures applied to one model, and most of the reversals found came from such comparisons. Comparing mid-training recipes is that kind of comparison. There, measurements built to read what the next stage uses did better in the few studies that tested them (see the section on proxies). This is a pattern read across a handful of studies, mostly single runs, not a tested rule. No study explains it as a whole: in vision, one instance has been traced to how tightly the last layer clusters the training classes (below), and the explanations offered for the language-model reversals rest on correlations. Shen et al., Bansal et al.
The studies show five ways a checkpoint’s score can mislead: the next stage can erase a difference, hide one, or reverse a ranking; a measured change can belong to something other than the data’s content; and a measurement can sit at a floor or a ceiling. Most of the examples are single runs, and each gives its run count.
The next stage can erase a difference the checkpoint shows. Across 56 base models, Dominguez-Olmedo, Dorner and Hardt estimated that models trained after November 2023 scored 7.3 points higher on MMLU and 19.1 higher on GSM8K than older models at equal pretraining compute. After every model was fine-tuned on the same task data, the differences were −0.5 and +0.5 points and not significant. To test why, the authors fine-tuned copies of older models on that task data. The copies then led their untouched originals by about as much as newer models had led older ones, and when copies and originals were both fine-tuned further, that gap closed. Dominguez-Olmedo et al.
The OLMo 3 team briefly fine-tunes each full candidate mix to check, in its words, that gains seen after mid-training “yield improvements beyond base model capabilities”. One candidate mix led the previous one by 1.0 points on the base-model evaluation average and by 0.1 on the post-training evaluation average after that fine-tuning; for another, 2.4 points became 2.0 (one run each, so the 1.0-point lead may itself be within run-to-run noise). OLMo 3
In the PRISM study of mid-training recipes, switching Mistral-7B’s 27-billion-token mid-training mix from math and code to math, code and science raised MATH500 from 47.70 to 70.71. After the same RL, the pipeline without science data scored 84.18 and the one with it 84.15 (the best RL step of one run each). PRISM
The checkpoint can hide a difference the next stage reveals. Allen-Zhu and Li trained small language models on synthetic biographies of 100,000 invented people. Models that saw one fixed-order biography per person, and models that saw five reworded and shuffled biographies, both predicted the facts in their biographies with next-token accuracy well above 99 percent, the authors report. After the same question-answer fine-tuning on half the people, the two 124M-parameter models answered questions about the other half at 9.7 and 96.6 percent. Across 31 fine-tuning configurations, in each of four combinations of model size and biography style, models trained on one biography per person never exceeded 11.0 percent, against 2.7 for always guessing the most common answer. In OctoThinker, three math web corpora used for the same 20-billion-token mid-training of Llama-3.2-3B scored 19.2, 20.2 and 20.4 on few-shot MATH500 before RL, and about 10.4, 31.1 and 35.7 after the same 400 steps of RL (read off the paper’s figure, one run each). Allen-Zhu and Li, OctoThinker
Two more hidden differences surfaced only in later stages. Watts and colleagues mid-trained OLMo-2-1B with a sharpness-aware optimizer, which left its ten-benchmark base average where AdamW left it (42.9 against 43.2). When the two were fine-tuned to the same fine-tuning loss, the sharpness-aware model had lost 22 to 35 percent less of that average on three of four datasets (one seed). Lu, Tang and Cavallaro forked OLMo-2-1B early in its training, at 49 billion tokens, and gave the branches different 500-million-token final windows of data. After the same fine-tuning, two branches were within about a point of each other on instruction following, refusal and capability. Under the same preference training on general preference data, they lost 24.0 and 32.2 points of refusal of harmful requests (three seeds). A comparison after fine-tuning is itself a checkpoint-level measurement when preference training or RL comes next. Watts et al., Lu, Tang and Cavallaro
The ranking can reverse. Kang and colleagues fine-tuned Mistral-NeMo-12B-Instruct on the same 25,000 math examples for 8 epochs or for 2. Eight epochs scored higher after fine-tuning, 34.52 against 32.51 (single-try accuracy averaged over seven math benchmarks), and lower after the same RL recipe, 36.72 against 42.57. Each RL figure is the best of at least four runs of that one recipe, each evaluated at four or more checkpoints. Kang et al.
Springer and colleagues found a smaller reversal between two public checkpoints of one OLMo-1B run. The 3.09-trillion-token checkpoint beat the 2.35-trillion-token one on five standard benchmarks by 0.17 to 1.54 points before instruction tuning and trailed it by 0.48 to 1.21 after, with the fine-tuning learning rate tuned for each checkpoint (read off the paper’s Figure 2, three fine-tuning runs each). At the two smallest fine-tuning learning rates plotted, which left both checkpoints well below their best scores, the later checkpoint scored higher on most benchmarks instead. In two other model families the instruction-following score kept rising with more pretraining. Springer et al.
Maskey and colleagues found a larger reversal, most of it from one failure the pretraining measures did not show. They forked a 30B mixture-of-experts run at 6.7 trillion tokens: one branch kept its learning rate constant, and the other decayed it to a tenth of its peak over the last 800 billion tokens. At that point the decayed branch was better on every standard pretraining measure: lower training and validation loss, and a pretraining benchmark aggregate of 0.440 against 0.415. After identical mid-training, long-context training and fine-tuning, it scored 0.247 against 0.360 on a separate suite of six post-training benchmarks. The downstream schedule had been chosen on the decayed branch and reused for the other, and even the best of the decayed branch’s 27 tuning runs reached only 0.294. On HumanEval+, which scored the last code block of each answer, the decayed branch kept repeating code blocks and scored 4.9 percent against 64.6. Without HumanEval+ the two scored 0.287 and 0.303, a gap one seed cannot establish. Even that gap includes the same failure to finish an answer: all of the decayed branch’s AIME answers and 99 percent of its GPQA answers ran to the length limit, against 30 and 55 percent for the other branch. Maskey et al.
In the three cases found where language-model checkpoints swapped order after later training (Springer, Yano and Maskey), the checkpoints sat at different points of a learning-rate schedule, some decayed further than others, which is the difference the same-schedule control removes. Kang and colleagues’ reversal after RL, and Aphale and Liu’s below, came instead from the number of fine-tuning epochs.
A measured change can belong to something other than the content of the new data. The decay in the OLMo 2 example and the extra steps in Hu and colleagues’ study are two cases. Restarting training is a third, and it shows up as a loss. Ibrahim and colleagues found that re-warming the learning rate while continuing on the same data raised a 405M-parameter model’s loss on that data by 0.1 nats at its peak, against 0.35 when it continued on new English web text with the same schedule. Part of the forgetting measured during a stage can therefore be the restart itself. Ibrahim et al.
Contamination is a fourth case: a gain can come from test items in the data rather than from what the data teaches, though its effects vary by benchmark. In OLMo 3’s one matched pair of mid-training runs with and without decontamination, the run on uncleaned data scored 13.9 points higher on DROP, but the cleaning had also removed over 60,000 DROP training examples, so that difference reflects both the contamination and the lost training data. On GSM8K the run on uncleaned data scored 1.6 points lower, within run-to-run noise, even though every GSM8K test item was in its data. OLMo 3
The measurement can sit at a floor or a ceiling. In Zhang, Neubig and Yue’s synthetic setting, five pretraining recipes all scored about 0 on the hardest held-out problems at pass@128 before the same RL, and between about 6 and 26 after it (read off a figure, one run each). Aphale and Liu fine-tuned Qwen2.5-Coder-3B for about 1, 2, 3, 4 and 6 epochs, a ladder of five checkpoints, and ran the same RL from each, three seeds per checkpoint. Pass@64, the share of evaluation problems solved at least once in 64 tries, sat between 0.897 and 0.932 for every checkpoint, while their best pass@10 during RL ranged from 0.481 to 0.806. Pass@64 still ranked them the right way overall (rank correlation +0.57) and picked out the best and the worst, but could not separate the three in the middle. Zhang, Neubig and Yue, Aphale and Liu
What each next stage reads from its starting point
Each kind of next stage changes the model in its own way, and a starting point is good for it to the extent that this way of learning can use what the starting point has. General-purpose scores such as mean accuracy and loss average away part of what each next stage uses.
Reinforcement learning
Reinforcement learning with a verifiable reward trains on the model’s own samples, each scored right or wrong. Group-relative methods such as group relative policy optimization (GRPO) draw several samples per problem and weight each sample’s update by its reward minus the group’s mean reward (the original GRPO also divides by the group’s standard deviation). A problem whose samples are all right, or all wrong, gives every sample an advantage of zero, so it contributes nothing to the reward-driven part of the update. With success probability p on a problem and g samples per group, the expected variance of the rewards in a group is p(1 − p)(g − 1)/g, largest at p = 0.5. A group drawn for a problem is more likely than not to be entirely wrong once p falls below 1 − 2^(−1/g), which is 0.083 for 8 samples and 0.042 for 16. Aphale and Liu
That describes where group-relative RL gets the most signal from each problem. Where RL can end up is described by an idealization. Suppose RL maximizes the expected reward, 1 for a correct answer and 0 otherwise, minus β times the KL divergence from the starting model, and reaches the exact optimum for each problem separately. Then its result is the starting distribution reweighted: every correct answer’s probability is multiplied by the same factor, c = e^(1/β), and the distribution is renormalized. A problem solved with probability p then ends at p·c / (p·c + 1 − p), which follows in one line (see the derivation below) from the reweighted solution that Korbak, Perez and Buckley give. Korbak, Perez and Buckley
Two checkpoints show what follows. Checkpoint A answers each of five problems correctly with probability 0.5 and never answers the other five. Checkpoint B answers each of ten problems correctly with probability 0.25. Both score 25 percent on single tries. With c = 9, which corresponds to β of about 0.46, A’s five solvable problems rise to 0.9 and its other five stay at 0, an average of 45 percent; B’s ten rise to 0.75, an average of 75 percent. B ends higher for every c above 1, and as the KL penalty shrinks toward zero the idealized results approach 50 and 100 percent. The reason is diminishing returns: at the optimum, each extra unit of starting probability on a problem adds less the more that problem already has, so for a fixed single-try score the most even spread across problems ends highest. Under GRPO with 16 samples per problem, B also gives a learning signal on more problems: on average about 9.9 of its 10 problems produce groups that are not all right or all wrong, against 5.0 of A’s. The single-try score could not separate the two checkpoints. The share of problems solved at least once in 16 tries could: 0.50 for A, 0.99 for B.
Ten problems. Each bar is one problem's probability of a correct answer in a single try. The starting values are invented, and the values after RL are computed, not trained.
A third checkpoint shows the limit of that share. Checkpoint C answers each of the ten problems correctly with probability 0.05. Its share of problems solved at least once in 16 tries is 0.56, above A’s 0.50, yet with c = 9 it ends at 32 percent, below A’s 45. Sixteen tries credit more than c = 9 does: pass@16 counts a problem solved 5 percent of the time as 0.56 solved, while the optimum at c = 9 raises it only to 0.32. Its single-try score is 5 percent, only about 5.6 of its 10 problems produce mixed groups of 16 on average, and its expected reward variance per group is about a quarter of B’s. At the idealized optimum, C ends below A for c below 17 and above A beyond it, so which of the two is the better start depends on the strength of the KL penalty. Neither the single-try score nor the share solved within 16 tries settles it.
In general, accuracy at the idealized optimum equals the starting model’s pass@k averaged over k drawn from a geometric distribution with mean c. The whole curve of pass@k against k therefore fixes the idealized outcome for every β. No single point on the curve matches it. For c of 5 or more, the point with the smallest worst-case error over starting probabilities is pass@k at k between about half and two-thirds of c, and even that point can miss the idealized result by 7 to 10 points of accuracy (computed for this digest). In the example, with c = 9, pass@k orders the three checkpoints correctly for every k from 2 to 13 and puts C above A from k = 14 on. Near a crossover any single k can misorder close candidates: for c just above 17 up to 20, C ends slightly above A, but every pass@k in the half-to-two-thirds range still puts A first.
Derivation
The objective E[r] − β·KL(π ‖ π₀), maximized over output distributions π for a fixed prompt, has the solution π*(y) = π₀(y)·exp(r(y)/β) / Z, where π₀ is the starting model and Z normalizes. Korbak, Perez and Buckley give it for unconditional sequences (their Eq. 5); the version for one prompt is the same derivation. For one problem with reward 1 on correct answers and 0 otherwise, correct answers carry total probability p under π₀ and p·c before normalizing, with c = e^(1/β), while incorrect answers carry 1 − p. Normalizing gives p·c / (p·c + 1 − p).
This value is concave in p for c > 1: its second derivative is −2c(c − 1) / (1 + p(c − 1))³. So among checkpoints with the same average p, the one with the same p on every problem ends highest at the optimum.
Let K follow a geometric distribution on 1, 2, 3, … with P(K = k) = (1/c)(1 − 1/c)^(k−1), whose mean is c. Then E[1 − (1 − p)^K] = 1 − (1/c)(1 − p) / (1 − (1 − 1/c)(1 − p)), which simplifies to p·c / (p·c + 1 − p). Since pass@k on one problem is 1 − (1 − p)^k, the identity holds problem by problem and so for any average over problems.
For the three checkpoints: A averages 0.5·c/(c + 1), B averages c/(c + 3), and C averages c/(c + 19). B exceeds A by c(c − 1)/(2(c + 1)(c + 3)), which is positive for every c > 1. A and C are equal at c = 17. These are this digest’s arithmetic, checked by script. The idealization assumes exact optimization over all output distributions, a KL penalty measured from the starting model with a fixed β, a reward of 0 or 1 on the final answer, and no sharing of learning between problems.
The idealization leaves out what real runs do. Most of the RL runs with verifiable rewards in the studies reviewed here used no KL penalty or a coefficient of 0.001. Taken literally, that makes c effectively infinite (e^1000 for 0.001), and the idealized result is then the share of training problems the starting model ever solves, where C would end level with B and above A. Real runs stop far short of that limit. In the one study that reports the comparison, after 450 steps without a KL penalty, Qwen2.5-7B answered 42.5 percent of its own training problems on one try, while before RL it had solved at least 67.2 percent of them within 256 tries. On held-out problems from the same benchmark, single-try accuracy rose from 10.2 to 28.3 percent while the share solved within 256 tries fell from 69.1 to 63.9. And the parameters are shared across problems, so learning on problems the model partly solves can raise problems it never solved and lower others it rarely solved. In Zhang, Neubig and Yue’s synthetic arithmetic setting (100M-parameter models, one run per condition), RL raised pass@128 on harder held-out problems only when it trained on problems just beyond the pretraining range, which the base model still solved some of the time; RL on problems the base model never solved raised nothing. Yue et al., Zhang, Neubig and Yue
Which k to read in a real run is therefore not settled. A real run has no single c, and whether a longer run acts like a larger c is untested. Taken literally, the small KL penalties of real runs point to large k. The number of samples RL draws per problem is another candidate, since a problem solved far less often than once per group rarely yields a group with mixed rewards. On Aphale and Liu’s ladder, pass@10 before RL, near its group size of 8, ranked the checkpoints the wrong way, while pass@64 got the overall order right (both computed for this digest from the paper’s tables); in Kang and colleagues’ study, pass@64 ranked RL results well (see the section on proxies).
Gandhi and colleagues’ experiment shows both what the idealization captures and what it misses. They ran PPO, an RL method with a learned baseline rather than a group mean, on Llama-3.2-3B playing Countdown, an arithmetic puzzle. Unprimed, the model scored 0 at the start and about 0.34 after RL (read off the paper’s figure; the authors describe it as about 30 percent accuracy). Problem by problem, the idealization would leave a problem the model never solves at 0, so either success was too rare to register at the start or learning carried across problems. Fine-tuned first on search traces whose final answers were all wrong, a step the authors call priming (the paper does not give the size of that set), the model scored about 0.29 at the start and 0.74 after RL, level with priming on the standard traces (0.73; one run per condition). Correct answers in the priming data were not needed. Every primed model that reached the plateau near 0.7, well above the unprimed model’s 0.34, already solved some problems before RL (scores of 0.18 to 0.42, read off the figures), so the experiment does not separate the search behavior from the starting success rate. Gandhi et al.
On Aphale and Liu’s ladder, single-try accuracy before RL ranked the checkpoints’ best results during RL backwards (rank correlation −0.75 over five checkpoints and three seeds). The checkpoint fine-tuned longest was ahead before RL on single tries (0.187 against 0.151 for the shortest) and ended furthest behind (a best pass@10 of 0.481 against 0.806). The entropy of the first generated token at the sampling temperature, which mostly fell as fine-tuning went on (from 0.227 to 0.120 nats), ranked them the right way (+0.69), a correlation over five checkpoints rather than a tested cause. During RL the longest-trained checkpoint’s success rate fell from 0.085 at step 50 to 0.020 at step 200 (one seed), and at that rate about 85 percent of groups of 8 were entirely wrong, so most groups gave no learning signal; the authors attribute the fall to the collapse of entropy. The inversion appeared on one of the paper’s two base models: on DeepSeek-Coder-6.7B, the best RL results stayed between 0.861 and 0.891 across its ladder. Aphale and Liu
Raising entropy during pretraining did not make a better start for RL in the one test found. Wu and colleagues pretrained with a modified objective that raised next-token entropy. Before RL, its pass@64 on math was below that of ordinary cross-entropy training at two of three model sizes. After RL, its mean accuracy over six math benchmarks was within a point of ordinary training’s at two sizes and 2.4 points below at the largest, and 1.7 and 3.7 points below a variant tuned toward lower entropy at two sizes (one run each). Its higher entropy did not last: at one size, its entropy during RL fell from 2.18 to about 0.2 early in training (read off a figure). Wu et al.
So what RL reads at the start is, in part, how the chance of a correct answer is spread across the problems it trains on, measured the way RL samples (its prompt template, temperature and length limit). Problems that are neither always nor never solved carry the most weight, and how far learning on them carries to the target evaluation matters too. That snapshot is not the whole of it. On Aphale and Liu’s ladder, checkpoints with similar pass@64 ended far apart, and the order they ended in went with their entropy before RL, not with their success rate.
Problem by problem, the starting probability is a weak guide. Across 18 post-training runs (16 of them RL) on two 1.5B math models, DeepSeek-R1-Distill-Qwen-1.5B and Qwen2.5-Math-1.5B, the base model’s pass rate on rarely solved problems (solved in 1 to 25 of 256 tries) ranked those problems by how much they improved, averaged over runs, with rank correlations of 0.19 and 0.29. Squared, and divided by the square of the highest correlation that run-to-run noise allows (0.95 and 0.91 here), those correlations account for about 4 and 10 percent of the variation in gains that the runs reproduce (computed for this digest using the paper’s formula). Averaged over hundreds of problems, the same kind of information has ranked whole checkpoints well in one study and failed in another; the section on proxies gives the conditions. Duan
Supervised fine-tuning
Supervised fine-tuning (SFT) trains on outputs it is given, so it does not need the model to produce the target answers first. It does supply the answer format. In Dominguez-Olmedo and colleagues’ comparison above, the newer models’ advantage came from more training on data resembling the benchmark tasks, which the authors call training on the test task. On MMLU it was mostly skill at answering in multiple-choice form: scored as a cloze task, with no answer letters, the newer models had no significant advantage. Fine-tuning every model on the same task data, multiple-choice questions for MMLU and worked math problems for GSM8K, removed the advantage on both.
What SFT supplies poorly is information stored in a form its questions cannot retrieve. In Allen-Zhu and Li’s biographies, described above, how the facts were written during pretraining decided how much fine-tuning could extract, and none of the 31 fine-tuning settings tried, full fine-tuning and LoRA at several ranks, changed that. Jiang and colleagues found a similar pattern with a pretrained model: how the documents were learned mattered more than the instruction tuning that followed. They trained Llama-2-7B on the opening sections of 1,976 new Wikipedia film articles and tested it on questions about 256 of them, in two orders that used the same documents and questions:
- all the articles, until their perplexity was about 1, then instruction tuning on questions about the other 1,720: 30.3 percent correct;
- questions about the 1,720 articles first, then those questions together with their articles, and only then the 256 test articles: 48.1 percent.
Keeping the test articles in the instruction-tuning mix of the first order, to rule out forgetting, gave 30.2. Unlike Allen-Zhu and Li’s small models, Llama-2 did extract some knowledge through instruction tuning, which the authors attribute to its broad pretraining. Allen-Zhu and Li, Jiang et al.
How facts are stored also decides which direction of question fine-tuning can answer. Lampinen and colleagues fine-tuned Gemini 1.5 Flash on one-sentence facts about fictional celebrities. Asked for a name given its description, the reverse of how the facts were written, it scored 0.000 on a four-choice test where guessing scores 0.25, because it always picked names it had seen as training targets, against 0.987 with the facts given in context. Adding statements that a prompted language model inferred from the facts, such as rephrasings and reversals, to the fine-tuning data raised it to 0.993 (values read from the paper’s figure). Facts embedded in a larger structure of related facts were partly reversible after plain fine-tuning (0.525). Lampinen et al.
A measurement on the checkpoint can see this property when it asks for the information the way the next stage will. In Allen-Zhu and Li’s setting, a probe given only a person’s name (a linear classifier on the frozen transformer’s last-layer state, with a small low-rank update to the input embeddings) tracked the accuracy after fine-tuning across 16 variants of the pretraining data. The authors say the two correlate closely, and the correlation is above 0.9 whether the probe’s scores are averaged over the six attributes (birth date, birth city, university, major, employer and the employer’s city) or pooled attribute by attribute (computed for this digest from the values printed in the paper’s Figure 7). The authors also state that next-token accuracy on the facts in the biographies was above 99 percent in every run.
What SFT does to what the model already knew depends on the checkpoint and on SFT’s own learning rate. Rofin, Varre and Flammarion fine-tuned checkpoints from along the pretraining of four models of 1B to 3B parameters. At a fine-tuning learning rate of 3e-4, the drop in average accuracy on six general benchmarks after 500 SFT steps grew with pretraining: for OLMo-1B, from about 8 to about 20 points, and for SmolLM3-3B only once its pretraining learning rate decayed. At 1e-5 or below it stayed between about 1 and 5 points without a steady rise (values read off the paper’s figure). A comparison of checkpoints from different points of pretraining therefore has to state SFT’s learning rate. Rofin, Varre and Flammarion
Two explanations have been offered for that growth, and neither is established. Springer and colleagues’ theory, for a linear network that learns features one at a time, makes each feature learned in pretraining one more direction along which fine-tuning can damage what pretraining fitted. It accounts for forgetting, not for a worse result on the fine-tuning task itself, though their small-model experiments showed that too for some tasks. Rofin and colleagues attribute the growth to the sharpening of the loss surface, its curvature rising, that comes with learning-rate decay, a link they call correlational. Springer et al.
Frozen features and fine-tuned features
A next stage that fits a classifier to frozen features reads whether those features already separate the new classes. Kornblith and colleagues traced the opening example to how tightly the features cluster the pretraining classes. Every alternative loss that raised ImageNet accuracy also raised the penultimate layer’s class separation on ImageNet, measured as R², one minus the ratio of the average distance between features of the same class to the average distance between any two (0.641 for cosine softmax against 0.349 for softmax). Across nine losses, higher separation went with worse frozen-feature transfer (rank correlation −0.93). Tighter clusters, the authors argue, discard variation that new classes need. Networks trained with the different losses had similar representations except in the last third of the network, the layers that change most during fine-tuning, which the authors suggest is why fine-tuning nearly erased the difference. Kornblith et al.
MAE and MoCo v3, two self-supervised vision methods, show a similar reversal, between a linear probe and fine-tuning. Under a linear probe on frozen features, MoCo v3 scored 77.6 percent on ImageNet against MAE’s 73.5 (ViT-L models). With one transformer block fine-tuned, MAE led, 81.0 against 79.9, and on COCO object detection, with hyperparameters searched for both, MAE led by 4.0 points of box average precision. With 14 frozen vision encoders and three ways of learning a robot policy on top, Hu, Wang, Li and Gao found that the best encoder depended on the method: R3M, an encoder pretrained for robot manipulation, was best under behavior cloning and second worst under a reward computed from distances between embeddings. He et al., Hu, Wang, Li and Gao
Online reinforcement learning from an offline start
Offline RL trains a policy and value estimates from logged data, and online fine-tuning then collects new data. Methods that keep value estimates deliberately low during offline training often got worse when online fine-tuning began and spent interactions recovering. The authors’ account is that during online exploration, actions that were actually worse returned more than the underestimated values of the learned policy, so they looked better, and the policy unlearned its offline start. The checkpoint’s own offline score could not separate the pipelines: four offline methods that each scored about 0.5 on a visual pick-and-place task ended between about 0.6 and 0.9 after 50,000 online steps (read off a figure). Nakamoto et al.
Zhou and colleagues showed that the next stage decides whether such starts fail. Fine-tuned online with their own algorithms and without the offline data, CQL and IQL fell to nearly 0 percent success on one kitchen task and did not recover. Under a next stage that also discards the offline data but begins with rollouts of the frozen policy, starts from the value estimates of all three offline methods tested improved on three kitchen tasks, and which was best changed from task to task. The authors call the three equally good, though their figure shows final scores as far apart as about 0.9 and 0.44 on one task (read off a figure). Zhou et al.
Retained and usable knowledge
Across these settings, the property a next stage needs belongs to the pair of checkpoint and next stage, not to the checkpoint alone. Robotics adds a last case, in which what a model keeps and what its next stage can use come apart. Hancock and colleagues fine-tuned one vision-language model into two robot policies that differed only in how actions were written as tokens: as ordinary text, or as ten tokens the model almost never produces. On MMMU the text-action policy kept 42.7 and the other 45.9, against 46.0 before; across twelve vision-language benchmarks the authors call the two comparable, each ahead on some. On a robot task that required recognizing a cartoon character, the text-action policy succeeded in 18 of 30 trials and the other in 9. The policy that kept less on MMMU did better on the robot. Knowledge that is retained and knowledge the next stage’s outputs can use are different measurements. Hancock et al.
Cheap proxies for the full pipeline
A proxy here means a measurement cheaper than the full pipeline, taken on the checkpoint or from a short run of the next stage, that is used to rank candidates. The proxies that predicted the pipeline’s result in the reviewed studies had two things in common, and a third condition decides whether one can be trusted for mid-training. This is a pattern read across a small number of studies, not a tested rule, and a failure described below shows that the two shared features are not enough.
They read what the next stage acts on, under the next stage’s own conditions. Kang and colleagues measured pass@64 with RL’s prompt and sampling temperature on supervised fine-tuning variants of two instruction-tuned models, on the same benchmarks used to score the outcome. It ranked the variants’ results after RL (GRPO without a KL penalty; the best of at least 16 evaluations each) with rank correlations of 0.92 to 0.98, where the score after fine-tuning ranked them at 0.69 to 0.78. Both sets of correlations are computed within groups of similar fine-tuning compute, where the candidates are close. Pooled across budgets, the score after fine-tuning ranked one model’s twelve data-subset variants well too (0.97, computed for this digest), though it got the 8-epoch and 2-epoch models above the wrong way round. Allen-Zhu and Li’s name-only probe, and Yu and colleagues’ likelihood of the decisive action, are two more cases. Kang et al.
The outcome was an average over many problems. On problems the two 1.5B math models in Duan’s study rarely solved, about 57 to 67 percent of the variation in a single post-training run’s per-problem gains was specific to that run (computed for this digest from the paper’s variance shares). Averaged over hundreds of problems, that problem-by-problem variation largely cancels, though variation between runs in the overall score remains. Duan
The condition for trusting a proxy in a mid-training decision is that it has been checked on the same kind of comparison. Kang and colleagues’ variants were procedures applied to one model, the kind of comparison mid-training needs. Yu and colleagues’ likelihood was checked across released model families, and Bansal and colleagues’ and Shen and colleagues’ scores across checkpoints and sizes of one training setup; none of these has been shown to rank recipes applied to one model.
A proxy can be right and still too coarse. On Aphale and Liu’s ladder, pass@64 ranked the RL results in the right direction, but within a band of 0.897 to 0.932 it could not separate the middle three checkpoints.
Measuring under the next stage’s conditions was not enough on that same ladder. Single-try accuracy before RL, measured at RL’s sampling temperature on the outcome’s own problems, ranked the RL results backwards (−0.75). It reads the average chance of a correct answer, which the worked example above shows can hide what RL uses. The measurement that went with the outcome was entropy, which bears on how the chance of success changes during RL. The range of single-try accuracy across the ladder, 0.151 to 0.187, was as narrow as pass@64’s, so the width of the band does not explain which of the two worked. Aphale and Liu
A short run of the next stage is a candidate proxy that the reviewed studies suggest but have not tested against the full pipeline. OLMo 3 runs a quick fine-tuning pass on each full-length mid-training candidate to check that base-model gains survive it. Duan’s results point the same way: information from an actual post-training run predicted per-problem gains better than anything measured before post-training. The per-problem gains of an existing post-trained checkpoint from another model family predicted a held-out run’s per-problem gains at rank correlations of 0.52 and 0.34 on the two models, averaged over such checkpoints. The best single signal available before post-training, a 7B sibling’s pass rate, reached 0.38 and 0.22, and a cross-validated regression on all such signals reached 0.33 and 0.17, below the best single one.
The reviewed studies report few checks of any proxy against the full pipeline for recipes applied to one base model, and none for short trial anneals, which the OLMo reports use to choose mid-training data. OLMo 2 chose its math data with 19 such anneals, which its authors call microanneals, each reported as a single run and judged on MMLU and on 200 GSM8K questions. On 200 questions, the 2.5-point differences behind two of its choices are within question-sampling noise (an estimate computed for this digest). OLMo 2
The one reviewed comparison of short and long anneals, at the base-model level, found rankings of data sources reversing as the anneal grew. Magnusson and colleagues’ DataDecide suite shows something different for pretraining corpora: rankings from 150M-parameter runs, at about 2 percent of a 1B run’s compute, matched the 1B rankings for about 80 percent of pairs, on base-model accuracy with no post-training. Ostapenko et al., Magnusson et al.
Trade-offs between what is gained and what is kept
A mid-training recipe moves several quantities at once: each target evaluation, each capability that should be kept, each other target domain, and how readily the model learns afterwards. Many settings move a recipe along a single curve on which more gain comes with more loss. Training longer did this in Biderman and colleagues’ code and math fine-tuning of Llama-2-7B, until long instruction tuning began to lose on both measures. Raising the rank of a LoRA adapter usually did too, though in code instruction tuning rank 256 learned more than rank 64 at the same retention. Raising the peak learning rate did it in Ibrahim and colleagues’ continued pretraining. Across Öncel and colleagues’ 191 continued-pretraining runs of six open models, scored on questions about recent news, the share of questions a model newly got wrong rose with the share it newly got right, both as shares of all the questions, by a fitted 0.60 per unit (correlation 0.68). Biderman et al., Ibrahim et al., Öncel et al.
How readily the model learns afterwards is the quantity least often measured. A released checkpoint will often meet next stages that were not designed when it was chosen, such as a later recipe or someone else’s fine-tuning, so how readily it learns something new under a fixed budget matters apart from its result under any one next stage. Hernandez-Garcia and colleagues measure it as the area under the validation-loss curve while a copy of the checkpoint trains on a held-out language for a fixed number of tokens. Whether mid-training changes it is among the open questions at the end. Hernandez-Garcia et al.
A few changes moved the curve instead. Replaying a small share of the original data has the most support. Ibrahim and colleagues continued a 405M-parameter model from 300 billion tokens of English text (the Pile) onto 200 billion tokens of German. With no replay, and the learning rate re-warmed to its pretraining peak of 3e-4, the final losses were 3.56 on the Pile and 1.11 on German. Changing the peak learning rate moved along a curve: keeping it at its low final pretraining value gave 3.22 and 1.21, keeping more of the Pile by learning less German. Replacing 1 percent of the German tokens with Pile tokens left the curve, giving 2.83 and 1.12. Larger replay shares traded again: at 50 percent, 2.24 and 1.22. Single runs. Ibrahim et al.
- Learning rate varied, no replay
- Share of Pile data replayed, starting from the 3e-4 run
- Trained from scratch on both datasets
Final validation loss in nats per token · the two axes have different scales · 405M parameters · 300 billion Pile tokens, then 200 billion German tokens · one run each
Thede and colleagues found the same in all five of their continued-pretraining settings: the smallest replay share tried, 10 or 17 percent, cut forgetting by 0.29 to 0.50 nats of validation loss and by 2.9 to 10.7 points of benchmark accuracy, while changing the loss on the new data by at most 0.017 nats (computed for this digest from the paper’s table; single runs). Thede et al.
Two other changes helped on one side at little cost on the other, each in single-run reports. He and colleagues distilled a larger model only on the tokens where it was most confident. At the mid-trained checkpoint, with a 7B teacher, this scored 44.7 on an average of reasoning benchmarks, against 36.1 and 38.8 for ordinary distillation of every token with a forward and with a reverse KL loss. Its factual recall, 29.3, fell 1.0 point short of ordinary next-token training’s 30.3, against shortfalls of 1.3 and 1.8 for the two forms of ordinary distillation. PRISM averaged the weights of a mid-trained model with those of its base model, 85 percent mid-trained and 15 percent base, then trained for 1,000 steps on long sequences. That recovered more of the lost long-context score (on RULER) than the same training alone, at every length from 16,000 to 128,000 tokens. He et al., PRISM
A merge by itself traced its own trade-off curve, and whether it beat replay depended on how forgetting was measured. In Thede and colleagues’ study, replay at a fixed share beat merging on both adaptation and loss-based forgetting in 18 of 20 comparisons (five settings, four merge weights each). On forgetting measured by benchmark accuracy, replay beat merging in only 4 of the 20, because lightly merged models kept nearly all their benchmark accuracy while giving up most of the adaptation (counted for this digest from the paper’s table; single runs). Thede et al.
Three reporting rules follow.
- Compare recipes as curves, each swept over one setting such as duration or learning rate, rather than as single points.
- Report each capability separately. After PRISM’s math, code and science mid-training of Granite-3.3-8B, a six-task average rose from 66.15 to 66.48 while HellaSwag, WinoGrande and TruthfulQA each fell about 5 points and GSM8K rose about 18.
- Measure the trade-off after the next stage as well as on the checkpoint. In Xu and Zheng’s synthetic logic domains (Qwen3-8B, five seeds), two allocations of the same mid-training token budget across five domains were 19.7 points apart in overall accuracy on the checkpoint and 0.16 apart after the paper’s main fine-tuning and RL, well inside the seed spread. A second fine-tuning recipe in the same paper, which the paper describes as matched to the first in epochs and token budget, kept 18.9 of the 19.7 points; the paper does not explain the difference. In He and colleagues’ study, the confident-token model’s deficit in factual recall at the checkpoint closed during fine-tuning, because the model trained without distillation lost more recall there.
PRISM, Xu and Zheng, He et al.
How a gain changes with the next stage’s budget
A gain from mid-training is one of two kinds, defined against the next stage’s budget. A head start is a gain the baseline would match with an affordable amount more of the next stage, and it is worth the next-stage compute or data it saves. A change in reach is a gain that no affordable amount of the next stage closes. Telling them apart means measuring the gap at several next-stage budgets while enlarging whatever limits the next stage: fresh data when data is scarce, steps when compute is. Repeating epochs over a fixed fine-tuning set tests something different from enlarging the set, and a test at one larger budget often cannot tell which kind a gain is.
Where the gap was measured at more than one next-stage budget, it usually shrank as the budget grew, but it closed within the budgets tested in only one study. Zhang, Neubig and Yue split a fixed compute budget between mid-training and RL in 100M-parameter models on synthetic arithmetic. On the hardest held-out problems, giving 20 percent of the budget to mid-training beat RL alone on pass@128 by about 13 and 11 points at the two smallest budgets (about 1 and 2 billion tokens’ worth of compute), was about level at 4.2 billion, and trailed by about 3 to 8 points from 8.4 billion up (read off the paper’s Figure 18, single runs). On the problems both stages trained on, giving mid-training a larger share of the budget gave a higher pass@128 at every budget. In domain-adaptive pretraining of RoBERTa, the gain on one biomedical task was 3.2 F1 points with 500 labelled examples and 0.4 with the full set of about 180,000. In vision, an intermediate self-supervised stage on images from the source and target domains added 7.4 to 8.8 points of target accuracy with one label per class and 1.3 to 2.5 with 68. Zhang, Neubig and Yue, Gururangan et al., Reed et al.
Gaps persisted, or no setting tried closed them, in three situations. The first was a next stage whose data came from another domain. Adding audio from the target domain to speech pretraining cut the word error rate by 34 to 51 percent with 10 hours of fine-tuning labels from another domain, and by 31 to 46 percent with 100 hours. Hsu et al.
The second was a next stage that had stopped improving. Doubling MiniCPM-1.2B’s fine-tuning tokens changed seven benchmarks by −0.7 to +0.5 points, while adding high-quality data and fine-tuning data to the learning-rate decay changed them by −2.4 to +8.2 (single runs). Baek and colleagues fine-tuned each pipeline on its domain data until the test loss stopped falling. A pipeline that mixed the domain data into pretraining reached the baseline pipeline’s best domain loss with 57 to 71 percent as many pretraining tokens, depending on the domain, so the gap remained after fine-tuning had stopped helping. On the one domain tested with replay in fine-tuning, replay helped the baseline but did not close the gap. MiniCPM, Baek et al.
The third was a difference in how information was stored. In Allen-Zhu and Li’s biographies, no fine-tuning setting retrieved facts stored without augmentation, and Jiang and colleagues report that more epochs of instruction tuning usually did worse. Neither study measured the gap at a larger fine-tuning budget.
A single larger budget seldom settles which kind a gap is. In Akter and colleagues’ study, doubling the fine-tuning data of a model pretrained without reasoning data gained 4.09 points and closed 55 percent of its gap to the weakest reasoning-pretrained model, after fine-tuning only and in single runs. Whether more data would close the rest was not tested. (The 4.09 and 55 percent are computed for this digest from the paper’s tables; the paper reports a gain of 7.39 against a baseline fine-tuned on a different set, and does not say which set it doubled.) Akter et al.
How the base model changes a recipe’s value
The same mid-training data was worth different amounts on different base models. In three of the studies below, it tended to be worth less to a base that had already seen more of its kind, or more training overall, though two of the three results are weak, as noted. In a fourth, on RL, the direction reversed.
Lu, Tang and Cavallaro added the same 500-million-token window of safety text last to OLMo-2-1B branches forked at 49 billion, 504 billion and 4 trillion tokens. Under later preference training it protected 9.1, 3.0 and about 0.5 points of refusal relative to a window of web text (three post-training seeds; at 504 billion, the standard deviation of the difference across seeds was 4.5 points, larger than the difference itself). Lu, Tang and Cavallaro
In Baek and colleagues’ controlled sweep with 160M models, the relative reduction in test loss after fine-tuning that came from mixing translation data into pretraining rose from about 2 to 5 percent as the target language’s share of ordinary pretraining fell from 10 to 0.1 percent. Gururangan and colleagues’ gains from domain-adaptive pretraining were ordered, on the test split, by how little vocabulary each domain shared with a sample like RoBERTa’s original corpus: computer science 7.95 F1, biomedicine 1.35, reviews 0.90 and news 0.80. The development split did not keep that order, and the domain with the least overlap also had the smallest labelled sets. Baek et al., Gururangan et al.
With RL as the next stage the direction reversed: a base that had seen more math gained more from RL on math. Bansal and colleagues found that adding 8.5 to 10 billion math tokens to a 1B model’s pretraining raised its gain from RL on MATH problems more than quadrupling the parameters did. A result measured on one base model may not hold, or may be much smaller, on another. Bansal et al.
Run-to-run noise
The definition of better asks for a difference larger than the run-to-run noise of the whole pipeline, and each stage adds its own. These are the sizes found.
- Mid-training. In OLMo 3, two 32B mid-training runs of one mix differed only in data order. A merge of the two beat them by 2.9 and 1.6 points on the math benchmark average and by 5 and 2 points on GSM Symbolic, so the runs differed from each other by about 1.3 and 3 points.
- RL. In Bansal and colleagues’ study, two RL seeds from the same early checkpoint both started at about 25 on GSM8K pass@32; one ended at about 13 and the other between 34 and 38 (read off a figure). At 8B, Xu and Zheng’s five seeds of a synthetic pipeline had standard deviations of 1.5 to 2.6 points after RL, and Khatri and colleagues’ three repeated runs of one RL recipe varied by at most ±0.015 in their fitted final pass rate.
- Evaluation. One sampled pass over AIME 2024’s 30 problems had a standard deviation of 5 to 6 points for three 1.5B models, and running the same evaluation on different GPU clusters moved scores by up to about 4 points, even with identical software containers.
Most comparisons in the studies above are single runs of each stage, so differences of a few points between them are not established. OLMo 3, Bansal et al., Xu and Zheng, Khatri et al., Hochlehnert et al.
A protocol for comparing two recipes
The studies above support a procedure. Where a step rests on a study not discussed above, the step names it.
- Fix the next stage before training anything: its algorithm, data and budget, and a hyperparameter search for each candidate over the same grid with the same budget, selected on data not used for the comparison. Report the result at one shared setting as well. The best fine-tuning learning rate moved with the checkpoint in Springer and colleagues’ sweeps, and a shared rate created or hid their reversal. Tuning can also leave a comparison unchanged: when Dominguez-Olmedo and colleagues gave the newer models their best rate from a five-value sweep, the newer models’ advantage on MMLU after fine-tuning stayed near zero and not significant. And a search can overfit its selection data: Liu and Wang found that hyperparameter optimization for fine-tuning often failed to beat a plain grid search, for that reason and for lack of time.
- Fix the evaluation protocol completely (benchmarks, templates, temperature, samples per problem, length limits, answer extraction, software version), and keep some evaluations out of every mix decision. Sampling temperature alone moved AIME 2024 scores by up to 15 points.
- Name the costs held equal: the compute of each stage, unique and repeated data, and whatever is scarce in the next stage.
- Build the equal-cost controls: the same schedule on the original data, the same number of steps on ordinary data, the same data placed elsewhere under a stated schedule, and no stage at all with its compute moved to the next stage.
- Decontaminate the mid-training data against the test and validation splits of every evaluation, and report which benchmark training splits are in the mix; a training split in the mix needs a control like OLMo 2’s. Before RL on a model family with suspected contamination, add a clean, newly generated benchmark to the evaluations and run a control with random rewards. In Mingqi Wu and colleagues’ study, random rewards raised Qwen2.5’s scores on a standard math benchmark but gave no reliable gain on a clean generated one, which the authors attribute to contamination. A random-reward gain alone does not prove contamination: Shao and colleagues trace such gains to the RL objective amplifying behavior the model already had.
- Choose a cheap proxy that reads what the next stage acts on, measured under the next stage’s conditions, and check that it separates the candidates. Then run the full pipeline on a few candidates of the same kind and report how often the proxy orders pairs the same way.
- Replicate every stage that adds noise, at the sizes in the previous section, and test the difference between candidates, not each score separately. Check that outputs terminate and that answers are extracted correctly before reading an aggregate.
- Report each target evaluation and each retained capability after the next stage, as curves over one setting of the recipe and over the next stage’s budget, and state where the curves cross.
Springer et al., Dominguez-Olmedo et al., Liu and Wang, Hochlehnert et al., Mingqi Wu et al., Shao et al.
What is settled and what is open
The definition this digest uses, that one recipe is better than another only after a named next stage, under a fixed evaluation protocol and at a stated equal cost, is a choice rather than a finding; the opening example and the equal-cost controls show what goes wrong without it. Within the reviewed evidence, mostly from single runs, five things are shown.
- General-purpose checkpoint scores mostly ranked candidates well when the candidates differed widely, in size, in architecture under one recipe, or in how far training had gone, with one small reversal found between checkpoints of one run. They were least reliable when the candidates were procedures applied to one model. This is a pattern across a handful of studies, not a tested rule.
- Each next stage reads a different property of its start. The evidence is direct for fine-tuning on questions, which reads whether stored information can be retrieved by those questions, and for a linear probe, which reads whether frozen features separate the new classes. For RL it is partial: how the chance of a correct answer is spread across its training problems under its own sampling matters, and so does how that spread changes during RL.
- The same-schedule control can absorb most of a mid-training gain. In the one released-model report that ran it for a whole mid-training stage (OLMo 2, one run each), it accounted for most of the multiple-choice and MMLU gain, a fifth of the generative gain and none of the math gain.
- Small replay shares moved the gain-retention curve rather than trading along it, in Ibrahim and colleagues’ study and in all five of Thede and colleagues’ settings.
- These patterns appeared across the fields reviewed. The next stage changed which start was better in vision, offline-to-online RL and language models, and gains shrank as the next stage’s budget grew in vision, speech, natural-language fine-tuning and synthetic language-model experiments.
Five questions are open.
- How often a short trial anneal orders mid-training recipes for one model the same way as the full pipeline. No reviewed study measures it. For base-model scores across mid-training data choices for one model, the one correlation found is limited in scope: across six allocations of Qwen3-8B’s mid-training data among five synthetic logic domains, scores after mid-training correlated at 0.975 with scores after fine-tuning and a short RL stage. That held under the paper’s second fine-tuning recipe, the one that kept the checkpoint gap; under its main recipe, a 19.7-point gap at the checkpoint shrank to 0.16.
- Whether mid-training changes what RL can reach, or only gives a head start, at production scale. The clearest budget sweep is at 100M parameters with single runs.
- How large the run-to-run spread of a post-RL comparison is above a few billion parameters with a full-length RL stage. The two measurements found at 8B, in the section on noise, are limited in scope: three runs of one RL recipe, and a synthetic pipeline whose RL stage added 1 to 2 points.
- Whether tuning RL separately for each candidate changes the ranking of language-model starting points. Every such comparison reviewed used one RL configuration for all candidates, while in offline-to-online RL, changing the number of gradient steps per environment step changed which start came out ahead.
- Whether mid-training changes how readily a checkpoint learns something new. No study found compares a mid-trained checkpoint’s adaptability with that of the same model before mid-training, or with an equal-compute alternative. The nearest, by Öncel and colleagues, continued the pretraining of six released base models, the mid-trained OLMo 2 models among them, and found that the models pretrained on fewer tokens absorbed more new knowledge.
Most numbers above come from single runs; seed counts are given where the sources report them.
Further reading
Selected sources available through 12 October 2026, with first-preprint dates.
- What the next stage reads: Kornblith et al., October 2020; He et al., MAE, November 2021; Hu, Wang, Li and Gao, April 2023; Korbak, Perez and Buckley, May 2022; Chen et al., a theory of how pretraining’s coverage of good answers enables later stages, October 2025; Aphale and Liu, June 2026; Allen-Zhu and Li, September 2023; Nakamoto et al., Cal-QL, March 2023.
- Predicting the pipeline from the checkpoint: Kornblith, Shlens and Le, May 2018; Kang et al., October 2025; Duan, October 2026; Yu et al., October 2026; Dominguez-Olmedo et al., July 2024.
- Reversals and hidden differences: Maskey et al., September 2026; Springer et al., March 2025; Yano et al., March 2026; Watts et al., May 2026; Lu, Tang and Cavallaro, July 2026.
- Controls and units: OLMo 2, December 2024; Hu et al., pre-pretraining on formal languages, February 2025; Davidson et al., December 2023; Hernandez et al., scaling laws for transfer, February 2021; Zhang, Neubig and Yue, December 2025.
- Trade-offs: Biderman et al., May 2024; Ibrahim et al., March 2024; Thede et al., September 2026; Xu and Zheng, September 2026; Hernandez-Garcia et al., June 2026.
- Measurement: Magnusson et al., DataDecide, April 2025; Heineman et al., signal and noise in benchmark evaluations, August 2025; Hochlehnert et al., April 2025; Khatri et al., October 2025; Mingqi Wu et al., July 2025; Shao et al., June 2025.
- Earlier intermediate stages: Gururangan et al., April 2020; Pruksachatkun et al., May 2020; Hsu et al., April 2021; Ash and Adams, October 2019.