Mid-training

OLMo 2’s 7B model read 3.9 trillion tokens of ordinary pretraining data. Then it read 50 billion more, a little over one percent, from a different mix: about half filtered web text, the rest math problems, instruction data, academic papers, and encyclopedia entries, while the learning rate fell to zero. On a grade-school math test, GSM8K, its score went from 24.1 to 67.5. Its average across ten evaluations went from 53.0 to 62.9. How much of the math jump reflects math ability is uncertain: the new mix included GSM8K’s own training problems, and a later audit by the OLMo 3 team found matches to GSM8K’s test questions in the same math data. Team OLMo, OLMo 2, OLMo 3

That stage is mid-training. Most published training recipes since 2024 include one, under different names. Meta’s Llama 3 report calls it annealing. NVIDIA’s Nemotron-4 calls it continued training. Qwen3 calls it a reasoning stage. Microsoft’s Phi-4, Ai2’s OLMo, and Hugging Face’s SmolLM3 call it midtraining or mid-training. The term is recent: OLMo 2’s authors say they took it from Microsoft’s Phi-3 reports and from OpenAI, and theirs is the first open report to define it, as a shorter second stage, 5 to 10 percent of the training compute, where the learning rate decays linearly to zero on a curated mix. OLMo 2, Tu, Zhang, et al., survey

This digest covers what the stage contains, why the end of a training run is where it goes, how it relates back to pretraining, and what it does to post-training. Throughout, one distinction matters: in every production recipe the learning-rate decay and the data change happen at the same time, but they do different jobs.

What the stage contains

Mid-training is continued next-token training on a much smaller and more curated dataset than pretraining, done while the learning rate finishes its decay, and usually paired with an increase in context length. Three things change; the objective does not. What separates it from continued pretraining, the older practice of training further on a single new domain, is that mid-training keeps a large share of the original pretraining data in the mix and continues the optimizer and schedule rather than restarting them. Liu, Neubig, and Xiong put it as a chain of sizes: the pretraining set is larger than the mid-training set, which is larger than the post-training set, and continued pretraining is the special case where the weight on general data drops to zero. Tu, Zhang, et al., survey, Liu et al.

Where mid-training sits, and what changes there

What changes	Pretraining	Mid-training	Post-training
Data	Web, code, books, papers. Trillions of tokens.	A curated mix: math, code, question-and-answer pairs, long documents, some instruction data. Tens to hundreds of billions of tokens.	Instructions, preference pairs, reward signals. Millions of examples.
Objective	Predict the next token.	Predict the next token, on the new mix.	Imitate, then optimize preferences or rewards.
Context length	Short: 4K to 8K tokens.	Often extended here: 32K to 128K tokens.	Usually unchanged.
Learning rate	High and steady, or a slow decline.	Decays to near zero.	Small.

The published recipes agree on the shape and differ on the details. The table lists what each report states; where a report gives no number, the cell says so.

Model, report date	Pretraining tokens	Mid-training tokens	What changes in the mix	Context length
OLMo 2 7B, Jan 2025	3.9T	50B, run three times and averaged	half filtered web; the rest math, instruction data, papers, encyclopedia	4K, unchanged
OLMo 3 7B, Dec 2025	5.9T	100B, then 50B for long context	web 28%, code 20%, math 19%, question-and-answer pairs 14%, reasoning traces 8%, instruction data 6%, PDFs 5%	8K to 65K
Llama 3 405B, Jul 2024	15.6T	about 800B for long context, then a short annealing	upsampled high-quality code and math	8K to 128K in six steps
Phi-4 14B, Dec 2024	about 10T	250B	30% long documents, 70% pretraining data	4K to 16K
Qwen3, May 2025	over 30T	about 5T, then hundreds of billions for long context	more STEM, code, reasoning, and synthetic data	4K to 32K
DeepSeek-V3, Dec 2024	14.8T	final 500B at two reduced learning rates, then two short phases for long context	no change described	4K to 32K to 128K
SmolLM3 3B, Jul 2025	11.2T	100B for long context, then 35B of reasoning traces	math, code, and reasoning upsampled; then reasoning traces in chat format	4K to 64K

For most of these the stage is 1 to 3 percent of the tokens. Qwen3’s reasoning stage is the outlier at about 14 percent, and Llama 3’s long-context stage is about 5 percent. Only Llama 3 and OLMo describe averaging checkpoints at the end. Qwen2.5 and Kimi K2 have equivalent stages that the table leaves out, because their reports give no data-mix change for them.

The learning rate finishes decaying

The learning rate is the step size of each weight update. A pretraining schedule warms it up over a few thousand steps, holds it high or lets it decline slowly for most of the run, and lowers it toward zero at the end. The older cosine schedule declines the whole way. The warmup-stable-decay schedule, popularized by MiniCPM, holds the rate constant and then decays it over the final stretch, and Hägele and colleagues showed that a constant rate with a short cooldown matches the cosine schedule’s final loss when the cooldown is less than 20 percent of training. Two facts about that final stretch matter here. MiniCPM, Hägele et al.

First, most of the visible improvement arrives during the decay. In MiniCPM’s runs the loss barely moves during the stable phase and then drops sharply once the rate begins to fall, to a value at or below the cosine curve. The weights change less during the decay than before it, yet the loss falls faster. A decay covering about 10 percent of the tokens was enough for their small models, and 2.5 percent was not. MiniCPM

Most of the final loss drop happens during the cooldown

• Constant rate, then cooldown
• Cosine schedule

Vertical axis: training loss, falling downward

Second, the drop comes from the weights settling rather than from learning anything new. Wen and colleagues describe the loss surface as a river valley: a high learning rate makes the weights bounce between the steep sides while still moving along the river at the bottom, and the decay stops the bouncing so the weights settle to the valley floor, revealing progress that was already made. Hägele and colleagues found the same thing empirically: interpolating weights between the checkpoint before the cooldown and the one after reproduces the loss drop smoothly, which they read as a transition into a basin. Wen and colleagues also found that the tokens whose loss falls most during the decay are the less predictable ones, while the highly predictable tokens, which tend to carry facts, are learned better in the stable phase. Wen et al., Hägele et al.

The decay therefore improves the model on any data. OLMo 2 measured by how much. From the 7B model’s 4-trillion-token checkpoint they ran two 50-billion-token decays, one on the ordinary pretraining mix and one on the mid-training mix, and evaluated all three checkpoints. The decay on the pretraining mix lifted the multiple-choice average by 4.4 points and MMLU by 2.0 points, and lowered its score on a math development set, 200 GSM8K problems, by 1.5. The decay on the mid-training mix produced somewhat larger general gains and raised the math score from 28.5 to 46.5. These are single runs, so treat the small differences as noise and the math difference as the finding. OLMo 2

The learning-rate decay and the new data do different jobs

• Pretraining checkpoint, no decay
• Decay on the pretraining mix
• Decay on the mid-training mix

Evaluation	Pretraining checkpoint, no decay	Decay on the pretraining mix	Decay on the mid-training mix
OLMES, multiple choice	69.6	74.0	75.7
OLMES, generative	63.2	64.5	70.2
MMLU	59.8	61.8	63.1
GSM*, grade-school math	28.5	27.0	46.5

Accuracy in percent · OLMo 2 7B · each decay run is 50B tokens from the 4T-token checkpoint

The data changes

The mid-training mix has two parts. The larger part is pretraining data again, filtered for quality: about half of OLMo 2’s mix is filtered web text, 70 percent of Phi-4’s is drawn from the pretraining set, and OLMo 3 keeps web and PDF data from its first stage on purpose, because that “prevents excessive shift in training data distribution.” The smaller part is what the stage exists for: math problems and solutions, code, question and answer pairs, instruction data, and, since 2025, worked reasoning traces. OLMo 3 describes five targets for that part: math, code, knowledge elicitation through question answering, and “bootstrapping instruction following and reasoning ability ahead of post-training stages.” OLMo 2, Phi-4, OLMo 3

The form of that data matters as much as its content. OLMo 2 tested two versions of the same synthetic math set: one with answers written as Python programs, which lowered the math score from the 28.5 baseline to 25.0, and one with the same problems rewritten as prose explanations, which raised it to 65.5. OLMo 3 found that instruction data written with the chat template’s special tokens made the base model emit those tokens in its answers. That took its grade-school math score from 49 to 0, mainly because the evaluation could no longer parse the answers. The same template written as ordinary text cost about 3 points, and the final mix joined messages with plain newlines instead. OLMo 2, OLMo 3

How long the new mix should run has been measured once at a useful scale. Blakeney and colleagues trained a 7B model for one trillion tokens and, for the final stretch, removed the bulk web crawl and upsampled code and curated domain data. They compared this with a run that kept the original mix from the same checkpoint on the same learning-rate schedule, so the comparison isolates the data change. Done over the last 20 percent of training, the new mix raised MMLU from 35.7 to 42.6, GSM8K from 14.7 to 23.0, and HumanEval from 17.2 to 23.4 relative to that run. Their duration sweep found that MMLU peaked with a 10 percent window and the overall average with 20 percent, and that pushing to 30 percent kept improving math and code while general language scores slipped. Blakeney et al.

The context length grows

Nearly every recipe extends the context window during this stage, and the reason is cost: attention’s compute grows with the square of the sequence length, so Llama 3 trains on short sequences for as long as it can. Its 405B model then stepped the window from 8K to 128K tokens in six stages over about 800 billion tokens, advancing only when short-context scores had recovered and a needle-in-a-haystack test was solved at the new length. DeepSeek-V3 did it in two phases of a thousand steps each, 4K to 32K to 128K. Phi-4 spent its whole 250-billion-token stage going from 4K to 16K. Llama 3, DeepSeek-V3, Phi-4

Long documents are mixed with short ones. Qwen3’s long-context stage is 75 percent documents of 16K to 32K tokens and 25 percent shorter. OLMo 3 uses 34 percent long documents and 66 percent of its mid-training mix, because in a short trial the reverse ratio cost 2.5 points on a set of its short-context evaluations and the chosen one cost 0.8. Phi-4 found that documents that are naturally long beat short documents packed together to fill the window, and SmolLM3 found that upsampling long web pages, books, and code repositories beyond what occurred naturally did not improve its long-context scores. Where the extension sits also varies: Llama 3.1 extends context before its annealing, Qwen after, and GLM 4.5 after supervised fine-tuning. Qwen3, OLMo 3, Phi-4, SmolLM3

Extending the context has costs. OLMo 3’s long-context stage lowered its 7B model’s math average from 59.8 to 54.4. In the other direction, PRISM mid-trained a Granite 8B model at an 8K window and watched its score on RULER, a long-context benchmark, fall from 59 to 6 at 128K tokens. Training on long sequences for 1,000 steps brought it back to 38, and averaging in 15 percent of the base model’s weights before that training brought it to 42. SmolLM3 hit the same conflict from its reasoning stage and fixed it by merging its final model with a mid-training checkpoint at a 9-to-1 ratio. OLMo 3, PRISM, SmolLM3

How it connects back to pretraining

Mid-training is the tail of the pretraining run, not a separate procedure applied to a finished model. It uses the same loss, continues from the same optimizer state, and keeps most of the same data. That continuity protects general ability. In the controlled study by Liu, Neubig, and Xiong, continuing a 160M model on code alone raised its loss on the original web corpus above the untouched model’s, while a mix that was only 20 percent code lowered it. Kotha and Liang found the same thing for fine-tuning: replaying pretraining data during the target stage makes training on the target data more token-efficient, and a single schedule spanning both stages beats a fresh fine-tuning run by wide margins on their code and math sets. They ask model developers to release checkpoints and optimizer states from before the cooldown, since a decayed checkpoint is a worse starting point for anyone who wants to run their own mid-training. Liu et al., Kotha and Liang

Labs also use the stage to evaluate data for pretraining. Because a short decay from a late checkpoint shows the effect of a dataset in a few billion tokens, labs use it to decide what goes into the next pretraining run. Llama 3 rated candidate datasets by taking a half-trained 8B model and annealing it to zero on 40 billion tokens with the candidate at 30 percent weight, which the report calls “more efficient than performing scaling law experiments for every small dataset.” OLMo 2 ran 19 such microanneals, 130 billion tokens in total, with differences visible after less than 10 billion. Blakeney and colleagues removed datasets from their final phase to score them, and Hägele and colleagues note that a warmup-stable-decay schedule turns a scaling-law sweep into one long run per model size with cooldowns branched off it, which cut the compute of their sweeps in half. Llama 3, OLMo 2, Blakeney et al., Hägele et al.

Data seen at the end of training is remembered best, so this is the stage where benchmark contamination can do the most damage. OLMo 3 concentrated its decontamination on the mid-training and long-context data for that reason, and found complete copies of GSM8K in the sources it had inherited. Its one matched pair of runs with and without cleaning shows that the effect varies by benchmark. The copies did not raise the GSM8K score: the cleaned run scored 1.6 points higher, within run-to-run noise. The uncleaned run did score 13.9 points higher on DROP, a reading-comprehension benchmark, but cleaning had also removed over 60,000 of DROP’s training examples, so that gap may reflect lost training data as well as leaked test items. OLMo 3

Training late on a benchmark’s own training data is a related risk. Llama 3’s report shows how far it can move a score: annealing the 8B model on the GSM8K and MATH training sets raised its validation scores by 24.0 and 6.4 percent, an experiment they ran to measure annealing’s power, and one they kept out of the production mix. On the 405B model the same experiment gained almost nothing. Nemotron-4’s report describes a second data distribution for its continued-training phase containing “a small number of benchmark-style alignment examples.” A benchmark jump across this stage should be read next to the decontamination method that accompanies it. Llama 3, Nemotron-4

The boundary with pretraining is a matter of degree. Qwen3’s reasoning stage is 5 trillion tokens, more than OLMo 2’s entire 3.9-trillion-token pretraining. Llama 3 also changed its mix during pretraining, raising math and multilingual data partway through. And NVIDIA’s front-loading study asks whether the reasoning data belongs in a late stage at all. It pretrained an 8B model on one trillion tokens with reasoning data at 20 percent from the first token, then applied the same supervised fine-tuning and reinforcement learning to that model and to a baseline without it. The reasoning-pretrained model finished 18.7 points higher on average, and doubling the baseline’s fine-tuning data narrowed the gap but did not close it. The study has no mid-training arm, so it cannot say whether the same data inserted late would have matched front-loading, but it does show that the gains from pretraining-stage data survive post-training. Qwen3, Llama 3, Akter et al., front-loading reasoning

How it connects forward to post-training

Post-training changes the objective. Supervised fine-tuning (SFT) trains the model to imitate chosen responses, preference training learns from comparisons between responses, and reinforcement learning (RL) rewards sampled outputs, often with a verifiable check such as a math answer key. All of it uses far less data than mid-training, thousands to a few million examples, and all of it starts from the model mid-training produces. So the question for this section is what mid-training changes about that starting point.

It moves the model toward the post-training distribution. Liu, Neubig, and Xiong pretrained Pythia models from 70M to 1B parameters on 128 billion tokens of web text, inserted one of five mid-training mixes, then fine-tuned on four small target sets with five seeds each. Mid-training on code cut the fine-tuning loss on Python from 2.17 to 1.89, and mid-training on math cut the loss on grade-school math from 0.94 to 0.85. On a general chat set and a science question set, both close to web text already, the change was near zero. Liu et al.

When the mid-training data matched the fine-tuning domain, the model also ended fine-tuning with a lower loss on the original web text, in seven of eight comparisons; after math fine-tuning at 1B, for example, 3.02 against 3.13 for a control that continued on web text. This compares where the models ended, not how much each forgot: the paper does not report the web loss before fine-tuning, so a lower final loss may only reflect a lower start. Mixes that did not match the fine-tuning domain sometimes ended with a higher web loss than the control. Liu et al.

Timing also interacted with dose. At 160M, code introduced early worked best at 80 percent of the data, but introduced late, 80 percent did worse than no code at all, while 10 percent still helped. The study measures loss rather than accuracy, stops at 1B parameters, and uses only supervised fine-tuning. Liu et al.

It decides what reinforcement learning can amplify. RL mostly raises the probability of behaviors the base model already produces sometimes, a point the mode collapse digest covers at length. OctoThinker started from the observation that Llama base models went nowhere under RL for math, producing answers that ran to the length limit and repeated themselves, while Qwen2.5 models improved steadily. Their fix was a 200-billion-token mid-training stage on a math corpus followed by 20 billion tokens of question-and-answer data and reasoning traces. After RL, the plain 3B Llama scored about 10 on MATH500, while the same model after mid-training scored about 65, level with Qwen2.5-3B. The plain Llama run also used the simpler of the two prompt templates the paper tested, and the template alone can move scores a long way: on one of the paper’s mid-trained checkpoints, switching it moved the score before RL from 0.2 to 33.0. Part of the plain model’s failure may therefore come from the RL setup rather than the model. OctoThinker

Mid-training changed what reinforcement learning could reach

• Base model
• After reinforcement learning

Model	Base model	After reinforcement learning
Llama-3.2-3Bno mid-training	7.4 before, 10.0 after
OctoThinker-Long-3BLlama-3.2-3B after 220B tokens of math mid-training	25.8 before, 65.2 after
Qwen2.5-3Ba different base model, no extra training	38.2 before, 66.4 after
Scale

MATH500 accuracy in percent · 3B-parameter models · one RL run each

Two further findings from that study matter. Base-model evaluations did not show what RL would make of a checkpoint: three math web corpora, each used for an otherwise identical 20-billion-token mid-training run, scored within about a point of each other on MATH500 before RL, and about 10, 31 and 36 after the same RL. The corpus decided whether RL worked at all; with the weakest, the model’s answers collapsed into repetition. Adding long reasoning traces gave the largest further gain, though RL was unstable until its recipe was changed. Adding short question-and-answer pairs gave no gain. The results are one run per configuration, and the traces were distilled from DeepSeek-R1, so they do not show what mid-training without distillation can do. OctoThinker

PRISM repeated the pattern across seven base models from four families, 3B to 24B parameters, with about 27 billion tokens of mid-training each. Before any RL, math scores rose by 15 to 40 points, code by 5 to 12, and graduate-level science questions by 6 to 13, with a general benchmark or two slipping. RL applied directly to most base models left competition-math scores near zero, while RL after mid-training raised the six-benchmark average by another 8 to 12 points. The composition of the mid-training mix mattered more than the composition of the RL prompts: on Granite 8B, changing the RL mix moved the average by 0.2 points, and adding science data to mid-training moved it by 5.6. Each of these RL results is a single run. PRISM

Zhang, Neubig, and Yue tested mid-training before RL in a fully synthetic setting at 100M parameters and found that it helped only while compute was scarce. At a small budget, giving a fifth of it to mid-training on the RL task distribution and the rest to RL beat spending all of it on RL by about 11 points on the hardest held-out problems. The measure was pass@128, the share of problems solved at least once in 128 tries. At budgets four or more times larger, the same split trailed RL alone by 3 to 8 points. These are single runs too. Zhang et al.

Some of post-training’s format can move into this stage. MiniCPM added high-quality data and supervised fine-tuning data to the decay phase and then ran ordinary SFT on top: on the 2.4B model, MMLU went from 44.6 to 50.9 and GSM8K from 27.7 to 42.3 compared with decaying on pretraining data alone. On the 1.2B model, doubling the plain model’s SFT budget did not make up the difference, except on GSM8K, where the doubled model scored 2.6 points higher. SmolLM3 trained on 35 billion tokens of reasoning traces in chat format, four times over, before its supervised stage. OLMo 3 found that holding out its instruction and reasoning data from mid-training cost 1.9 points on the base-model average. None of these labs dropped the separate SFT stage; MiniCPM states that it was still necessary. MiniCPM, SmolLM3, OLMo 3

Three tensions remain open. The first is between domains. Xu and Zheng mid-trained Qwen3-8B on five synthetic logic domains in 30 different proportions, five seeds each. Measured right after mid-training, each domain scored best at a moderate share, between about 10 and 40 percent, and a domain given no share fell sharply. Whether that lasted depended on the fine-tuning that followed. Under the paper’s main fine-tuning and RL, the drop disappeared, and an allocation that gave two domains nothing ended level with a balanced one (65.9 against 66.1). Under a second fine-tuning recipe, which the paper describes as using the same data, token budget and schedule, the differences between domains did not narrow, and the paper does not explain why the two recipes disagree. Any continued training moved things, including a control on educational web text alone, which dropped the model’s strongest domain from 84 to 29 percent. Xu and Zheng

The second is between the decay and adaptability. Yano and colleagues pretrained 1B and 8B models with and without a learning-rate decay and found the undecayed checkpoints scored worse after pretraining and 0.3 to 1.4 points better after supervised fine-tuning, with the decayed models sitting in sharper minima. Zhu and colleagues make the related argument that the decay reduces the model’s ability to use the high-quality data it sees during the decay, and report that dropping the batch size at the data transition instead of only lowering the rate raised their 15B mixture-of-experts model’s average from 49.8 to 51.5. Both are single-run studies, and neither tests reinforcement learning, so the question remains open. Yano et al., Zhu et al.

The third is between mid-training and knowledge. He and colleagues distilled a 7B or 13B teacher’s next-token probabilities into a 1B student during mid-training on the OLMo 2 mix. Ordinary distillation raised the student’s reasoning scores to 1.2 to 1.5 times those of plain next-token training but kept only 94 to 96 percent of its factual recall. The authors trace the shortfall to the facts the student had not yet learned, which is where the teacher’s guidance was weakest. Restricting distillation to the teacher’s most confident tokens raised reasoning to 1.6 to 1.7 times the baseline and kept about 97 percent of the recall, and after post-training the recall gap closed while reasoning stayed 1.25 to 1.32 times higher. He et al.

A separate line of work uses the word midtraining for something much smaller. Anthropic researchers trained Llama and Qwen models on 8 to 41 million tokens of synthetic documents describing a model specification before alignment fine-tuning, and found it cut a measured misalignment rate on Qwen3-32B from 54 to 7 percent. A follow-up at 120 billion parameters found that documents about a constitution lowered a blackmail rate by about 18 points, and that the effect survived later fine-tuning. A stress test in September 2026 found the effect fragile: replacing 2 percent of the alignment fine-tuning data with conflicting examples reversed it, and under RL the difference from the control was 8 points. These budgets are a thousandth of a production mid-training stage, and the safety experiments often start from already post-trained models, so the shared word covers a different procedure. Li et al., Cho et al., Baines et al.

What the evidence supports

Three things happen at once in mid-training, and the one controlled separation at model scale says they do different work: the learning-rate decay lifts general evaluations on any data, and the curated data decides which specific skills arrive. The useful window for the new data is about 10 to 20 percent of a run before general scores start to fall.

For post-training, mid-training sets the starting point. It moves the model toward domains far from web text, and when its data match the fine-tuning domain, the model ends fine-tuning with a lower loss on its original web data. It also shapes what reinforcement learning can amplify, though in the one test at equal compute, a synthetic one, its advantage over RL alone reversed at larger budgets. The same stage can cost long-context ability, and a domain left out of the mix can fall sharply at the checkpoint; in the one study that followed this through post-training, the main recipe recovered it and a second recipe did not.

What the evidence does not include is a controlled comparison, at production scale, of putting the curated data in a late stage against spreading it through pretraining. Nearly every result above is one training run per configuration; only a few of the studies here report seeds. Benchmark gains across this stage are entangled with contamination unless the report says how it was checked. And the word itself now covers a five-trillion-token reasoning stage, a one-percent annealing mix, a chat-formatted reasoning stage, and a few million tokens of alignment documents, so a claim about mid-training can only be interpreted with the recipe attached.

Further reading

Selected sources available through 21 September 2026, with first-preprint dates.

• Definition and open recipes: OLMo 2, December 2024; COLM, 2025. OLMo 3, December 2025. SmolLM3, July 2025.
• Lab reports with a late stage under another name: Llama 3, July 2024; Nemotron-4 15B, February 2024; DeepSeek-V3, December 2024; Phi-4, December 2024; Qwen3, May 2025.
• Learning-rate schedules: MiniCPM, April 2024; Hägele et al., May 2024; NeurIPS, 2024. Wen et al., October 2024. Counterpoints: Yano et al., March 2026; ICLR, 2026. Zhu et al., May 2026.
• Data at the end of training: Blakeney et al., June 2024; Kotha and Liang, March 2026.
• Mid-training and post-training: Liu et al., October 2025; OctoThinker, June 2025; Akter et al., September 2025; Zhang et al., December 2025; PRISM, March 2026; Xu and Zheng, September 2026; He et al., September 2026.
• Surveys: Mo et al., October 2025; Tu, Zhang, et al., October 2025.
• Alignment midtraining: Li et al., May 2026; Cho et al., July 2026; Baines et al., September 2026.