So you want to turn your model into a reasoning model

DeepSeek-R1-Zero started from DeepSeek-V3-Base, a pretrained model with no instruction tuning. DeepSeek trained it with reinforcement learning on math, coding and logic problems, and the reward checked two things only: whether the final answer matched the reference, and whether the reasoning sat between <think> and </think> tags. Over 10,400 training steps, its score on AIME 2024, a 30-problem mathematics competition, rose from 15.6% to 77.9%. Its responses grew from a few hundred tokens on average to more than 10,000, although nothing in the reward asked for length. DeepSeek-R1

DeepSeek did not ship that model. Its reasoning mixed English and Chinese within a single response, and it scored 46.6 on IFEval, an instruction-following benchmark, against 86.1 for DeepSeek-V3, the company’s chat model built on the same base. The model DeepSeek released, DeepSeek-R1, went through four training stages instead of one.

The same paper found that a smaller model learned more by copying R1 than by repeating its training. On Qwen2.5-32B, more than 10,000 steps of the same kind of reinforcement learning reached 47.0% on AIME 2024. Fine-tuning the same model on 800,000 of R1’s responses reached 72.6%. DeepSeek-R1

Those results frame the practical question. A reasoning model is a language model trained to write a long chain of thought before it answers, in such a way that spending more tokens raises its accuracy. There are two ways to train one: have it imitate the chains of thought of a model that already reasons, or reward it for correct answers and let it find its own. Which works better depends on the model being trained and on whether a stronger reasoning model exists to copy. Either way, the result usually needs further training before it is usable, and some abilities the training never measured can get worse.

This digest takes those decisions in order. Nearly all the evidence comes from 2025 and 2026 technical reports and open reproductions, almost all of it on math and code. A large share of it uses one model family, Qwen2.5, whose peculiarities come up repeatedly.

What the training is for

A chain of thought is the text a model writes while working toward an answer: setting up the problem, trying an approach, checking a step, abandoning an approach that fails. Reasoning models write it inside marked tags, and most interfaces hide it or show a summary. What makes the text useful is that accuracy rises with it. Qwen3-235B, capped at about 1,000 tokens of thinking, scored about 42% on AIME 2024. Allowed 32,000 tokens it scored about 86%, and with thinking switched off about 40% (values read from the report’s figure). Qwen3

That accuracy costs tokens, and every token is generated one at a time and billed. On a set of mock AIME problems, Claude 3.7 Sonnet wrote 26,990 output tokens per problem on average with thinking enabled and 1,436 without, about 19 times as many. Epoch AI

Five parts recur in everything below.

• The starting model. Either a base model, straight out of pretraining, or an instruct model that has already been trained to follow instructions and chat.
• A teacher: an existing reasoning model whose chains of thought can be collected and copied.
• A verifier: a program that checks a final answer, such as comparing a number with the answer key or running code against test cases.
• Distillation: supervised fine-tuning of the starting model on the teacher’s chains of thought, so that it learns to write similar ones.
• Reinforcement learning with verifiable rewards (RLVR): the model samples its own answers, the verifier scores them, and training raises the probability of the responses that scored well.

The two training routes differ in what the model learns from. Distillation gives the model a target for every token, written by someone else. Reinforcement learning gives it one number per response, but for responses it wrote itself. That difference explains most of the tradeoffs that follow.

The shape the published recipes share

The labs that disclosed their stages used variations of one shape. First comes an optional supervised stage on a small set of long chains of thought, usually called a cold start. Then reinforcement learning on problems a verifier can check. Then a second supervised stage that mixes reasoning data, often sampled from the model trained so far, with ordinary chat and writing data. Last comes a short reinforcement-learning stage aimed at general helpfulness, often scored by learned reward models: networks trained to rate responses the way people would. DeepSeek-R1, Qwen3

The variations are in which stages each lab kept. DeepSeek-R1 and Qwen3 used all four. QwQ-32B reported the same stages without the second supervised one. Magistral Medium skipped the cold start, but began from an instruct model rather than a base model. Olmo 3 Think fine-tuned on a large set of chains of thought and added a stage of DPO (direct preference optimization, which trains on pairs of responses to raise the probability of the preferred one) before reinforcement learning. QwQ-32B, Magistral, Olmo 3

Six published recipes, stage by stage

Recipe	Starting model	Before reinforcement learning	Reinforcement learning	After it
DeepSeek-R1-Zero	DeepSeek-V3-Base	None	Rewards for a correct final answer and for the thinking tags; 10,400 steps	None
DeepSeek-R1	DeepSeek-V3-Base	Fine-tuning on thousands of R1-Zero chains of thought, rewritten to be readable	The same rewards, plus one for keeping to the language of the question	Fine-tuning the base again on about 800,000 examples, a quarter of them general; then 1,700 more steps, with learned reward models in the last 400
Qwen3-235B	Qwen3 base	Fine-tuning on a small set of QwQ-32B chains of thought	3,995 problems for 170 steps	Fine-tuning on thinking and non-thinking responses; then reinforcement learning across more than 20 kinds of task
Magistral Medium	Mistral Medium 3 Instruct	None	Math and code, with the response limit raised in stages from 16,000 to 32,000 tokens	None reported
Olmo 3 Think, 7B	Olmo 3 base, mid-trained on a mix that was about 8% reasoning traces	Fine-tuning on 2.27 million prompts; then preference training with DPO	1,400 steps over math, code, instruction following and chat	None
Qwen3, 0.6B to 14B	Qwen3 base	Fine-tuning on the flagship models’ responses	None; on-policy distillation from the flagship instead	None

DeepSeek measured a checkpoint after each stage of R1. On AIME 2024 the scores were 59.0% after the cold start, 74.0% after reinforcement learning, 78.1% after the second supervised stage, and 79.8% for the final model. Reinforcement learning added 15.0 points and the two later stages 5.8. The cold start’s own contribution cannot be read off, for two reasons. The paper gives no score for the base model under the same evaluation. And the cold start’s examples were R1-Zero outputs rewritten to be readable, so they carry what R1-Zero’s reinforcement learning had already found. DeepSeek-R1

The later stages mattered more for everything else. IFEval rose from 71.7 after the cold start to 72.0, 78.1 and 83.3 across the next three stages. AlpacaEval 2.0, which has a language model judge chat responses, rose from 50.1 to 55.8, 62.1 and 87.6. These are successive checkpoints of one pipeline, so they cannot show what a stage would have done in a different order. DeepSeek-R1

Whether the cold start makes the final model reason better is untested. DeepSeek made its examples by rewriting R1-Zero’s outputs into readable form, and its revised paper calls the motivation “primarily product-driven”. The final R1 scored 79.8% on AIME 2024 against R1-Zero’s 77.9%. Qwen3’s report advises keeping the cold start small and short, so reinforcement learning has room to work. No published study compares a cold start against none at matched compute. DeepSeek-R1, Qwen3

Olmo 3 Think released its training data for every stage. Its 7B model was fine-tuned on 2.27 million prompts with chains of thought from QwQ-32B and DeepSeek-R1. For DPO, the preferred response in each pair came from Qwen3-32B and the rejected one from the much weaker Qwen3-0.6B. Then it was trained with reinforcement learning for 1,400 steps. Olmo 3

Its scores show that not every benchmark rises at every stage. AIME 2025 went from 57.6 after fine-tuning to 62.7 after DPO and 64.6 after reinforcement learning. AIME 2024 went from 74.6 after DPO down to 71.6 after reinforcement learning, while IFEval over the same step rose from 75.9 to 88.2. These are means of three evaluation runs. Olmo 3

Copy a teacher or reinforce correct answers

When a stronger reasoning model is available, the first decision is whether to copy it. Three studies compare the two routes on a fixed starting model, and a fourth measures what reinforcement learning adds after distillation.

Copying a teacher against reinforcement learning, on the same starting model

Accuracy on AIME 2024, averaged over several samples per problem. Each group starts from one model; bars within a group are comparable, bars across groups are not.

• Distillation
• Reinforcement learning
• Both, distillation first
• Before either

0%20%40%60%80%

DeepSeek, from Qwen2.5-32B base

Reinforcement learning, over 10,000 steps

47.0%

Distillation on 800,000 R1 responses

72.6%

Qwen3, from Qwen3-8B after a first distillation

Starting point

55.0%

Reinforcement learning, 17,920 GPU hours

67.6%

On-policy distillation, 1,800 GPU hours

74.4%

Mistral, from Mistral Small 3 (24B)

Distillation from Magistral Medium

65.4%

Reinforcement learning only

65.8%

Distillation, then reinforcement learning

70.7%

NVIDIA AceReason 1.1, from Qwen2.5-Math-7B

Distillation from DeepSeek-R1

62.0%

Distillation, then reinforcement learning

72.6%

DeepSeek’s comparison is the one from the opening. On Qwen2.5-32B, reinforcement learning for more than 10,000 steps reached 47.0% on AIME 2024, and fine-tuning on 800,000 R1 responses reached 72.6%. The two arms also differed in data: the reinforcement learning used math, code and science problems, while the fine-tuning set included about 200,000 non-reasoning examples. DeepSeek did not report the reinforcement-learning run’s compute. DeepSeek-R1

Qwen3’s team controlled the most, though with a different kind of distillation. They took Qwen3-8B after a first round of ordinary distillation, at 55.0% on AIME 2024, and trained it two ways on the same math and code questions. Reinforcement learning reached 67.6% and used 17,920 GPU hours. On-policy distillation reached 74.4% and used 1,800, a tenth as much. Qwen3

In on-policy distillation the student writes its own responses, and the teacher’s probability for each next token in them becomes the student’s training target. It combines the two routes. The model learns from its own samples, as in reinforcement learning, but gets a target for every token, as in distillation.

Mistral’s comparison came out closer. Mistral Small 3 (24B), fine-tuned on outputs from Magistral Medium, Mistral’s own reasoning model, scored 65.4% on AIME 2024. Trained with reinforcement learning only, it scored 65.8%. The result depended on the benchmark: reinforcement learning alone was behind on AIME 2025 (51.9% against 55.6%) and on LiveCodeBench, and ahead on GPQA and MATH-500. Fine-tuning first and then reinforcement learning scored 70.7% on AIME 2024, the best of the three. Magistral

NVIDIA’s AceReason-Nemotron 1.1 measured the second step at 7B. Fine-tuning Qwen2.5-Math-7B on DeepSeek-R1’s outputs brought it to 62.0% on AIME 2024, and reinforcement learning on top raised that to 72.6%. AceReason-Nemotron 1.1

Read together, these say that when a stronger teacher exists, copying it has reached higher scores than reinforcement learning, or matched it, and reinforcement learning afterwards has added several points. Only Qwen3’s comparison measured cost, and there distillation used a tenth of the compute.

Distillation moves an ability rather than creating it. Each teacher above got its reasoning from a reinforcement-learning run. Two recipes also saw imitation level off: NVIDIA reported that its largest fine-tuned model approached DeepSeek-R1, its teacher, but did not exceed it, and Olmo 3’s team found that more fine-tuning on a stronger model’s outputs began to hurt. Llama-Nemotron, Olmo 3

The costs below span four orders of magnitude, because both the model sizes and the methods differ. Only the rough scale of each is comparable.

Work	Model trained	What was done	Compute
s1	Qwen2.5-32B-Instruct	fine-tuning on 1,000 chains of thought	26 minutes on 16 H100s, about 7 GPU hours
Sky-T1	Qwen2.5-32B-Instruct	fine-tuning on 17,000	19 hours on 8 H100s, reported as about $450
OpenThoughts	Qwen2.5-7B-Instruct	generating 1.2 million chains of thought, then fine-tuning	22,000 H100 hours to generate, 25,000 A100 hours to train
DeepScaleR	a distilled 1.5B model	reinforcement learning, about 1,750 steps	about 3,800 A100 hours
DeepSeek-R1	DeepSeek-V3-Base, 671B	R1-Zero and R1 post-training	147,000 H800 hours, $294,000 at$2 an hour

DeepSeek’s figure excludes pretraining the base model and the smaller preliminary runs that came before.

What the starting model has to bring

Reinforcement learning raises the probability of the model’s own responses that scored well. It cannot reward a response the model never writes. If a starting model never checks its work or backs out of a failed approach, reinforcement learning has nothing of that kind to strengthen.

Gandhi and colleagues showed this on Countdown, a puzzle in which a set of numbers has to be combined with arithmetic to hit a target. They gave Qwen2.5-3B and Llama-3.2-3B, two base models of the same size, the same reinforcement learning. Qwen ended near 53% of puzzles solved and Llama near 34% (values read from the paper’s figure). Before any training, Qwen already verified intermediate results and backtracked in its outputs, and Llama rarely did. During training, Qwen’s gains coincided with a rise in both behaviors, and Llama showed neither. Gandhi et al.

The same study then supplied the missing behaviors. Fine-tuning Llama first on 1,000 Countdown solutions written by Claude 3.5 Sonnet that contained backtracking lifted it to about 70 to 77% after the same reinforcement learning. Solutions with wrong final answers but the same behaviors worked as well as correct ones. Priming it with an empty chain of thought, or with placeholder text of the same length, left it near 33 to 40%, so the extra tokens alone did not help. These are 3B models on one puzzle, with one run per condition. Gandhi et al.

The “aha moment” reported for R1-Zero, where the model starts writing “wait” and rechecking, is partly present before training. Liu and colleagues sampled 500 MATH problems 8 times each from several base models and counted the problems where at least one sample reflected on its own work. Reading their figure, Qwen2.5-Math-7B did so on about 19 to 41 problems depending on the sampling temperature, DeepSeek-V3-Base on up to about 8, and Llama-3.1-8B on none. Liu et al.

Outside Qwen, base models often fail under reinforcement learning from scratch. SimpleRL-Zoo ran the same recipe on ten base models. All improved on average, but Mistral-7B’s responses grew into mixed-language gibberish, and weaker models needed easier training problems than Qwen did. Under reinforcement learning, Llama-3.2-3B began its answers with \boxed{} and repeated itself until it hit the length limit. SimpleRL-Zoo, OctoThinker

Mid-training fixed that. Mid-training is the late stage of pretraining in which a model continues next-token training on a smaller, curated mix. Llama mid-trained on 200 billion tokens of math-heavy data, followed by 20 billion that included chains of thought, came level with Qwen2.5-3B under the same reinforcement learning. Olmo 3 built this into its base model: about 8.3 billion of the 100 billion tokens in its mid-training mix were reasoning traces, some written to show behaviors such as backtracking and working backward from the goal. The mid-training digest covers that stage. OctoThinker, Olmo 3

Qwen2.5 improves under any reward

Qwen2.5-Math-7B improves under reinforcement learning even when the reward carries no information. Shao and colleagues rewarded its answers at random, each with probability one half. Its accuracy on MATH-500, taking the most likely token at each step, rose by 21.4 points, against 29.1 points with the correct reward. The same random reward lowered Llama-3.1-8B-Instruct by 11.5 points and OLMo2-7B by 8.3. Shao et al.

One explanation is that Qwen2.5 has seen these benchmarks. Given the first 60% of each MATH-500 problem, Qwen2.5-Math-7B reproduced the remaining 40% word for word on 54.6% of them; Llama-3.1-8B did so on 3.8%. Part of the measured gain is also format. A chat template is the set of markers a chat model expects around each turn, and Qwen2.5 base models score higher without one, so early training partly recovers accuracy the template took away. Wu et al.

The explanation is disputed. Shao and colleagues attribute the gain to a side effect of the update rule. Chen and colleagues argue that side effect is too small to matter, and report random-reward gains in a distilled Llama model too, though that model’s training data may also have included the benchmark. Exploration vs exploitation

Either way, a random reward is a cheap test of a model and a benchmark. On Olmo 3’s base model, whose training data had been filtered against the evaluation sets, random rewards produced no gain on any benchmark. Olmo 3

Base or instruct

No published study runs the same long reasoning training from a base model and from its instruct version and compares them. Two results bear on it. On Mistral-Small-24B, a first stage of ordinary instruction fine-tuning, on the OpenHermes-2.5 dataset, lowered what reinforcement learning later reached: the six-benchmark average after reinforcement learning was 49.6 with no fine-tuning, 38.9 after 200 fine-tuning steps and 34.4 after 1,000. On Olmo 3 7B, reinforcement learning from the DPO checkpoint beat reinforcement learning from the fine-tuned one, 74.1 against 71.9 on a benchmark average, in one run each. SimpleRL-Zoo, Olmo 3

The prompt format matters when starting from a base model. Qwen2.5-Math-7B, before any training, always taking the most likely token and limited to 3,000 tokens, averaged 38.2 across five math benchmarks with no template and 0.0 with the template DeepSeek used for R1-Zero. Olmo 3’s team found simple plain-text prompts worked far better than chat-style templates on its base model. Liu et al., Olmo 3

A practical check before a run follows from this, though it is an inference from these studies rather than a tested protocol. Sample the starting model many times on the training problems, in the prompt format training will use. Look for problems it solves some of the time, for any checking or backtracking in its reasoning, and for answers in a form the verifier can read. If all three are rare, fine-tuning on chains of thought first is the reported fix.

Fine-tuning on a teacher’s reasoning

How many examples

How many examples distillation needs depends mostly on the student. At 32B, a thousand can be enough. The s1 authors picked 1,000 problems from a pool of 59,000, chosen to be hard, varied and well-formed, with chains of thought from Gemini 2.0 Flash Thinking. Fine-tuning Qwen2.5-32B-Instruct on them took it from 26.7% to 50.0% on AIME 2024. Training on all 59,000 scored 53.3%, one problem more. Training on 1,000 chosen at random scored 36.7%. LIMO reported 63.3% on the same benchmark with 800 examples ranked by a scoring rule. Both scored AIME with one sample or a few per problem. s1, LIMO

At 7B, the same small sets leave the model far below what more data reaches. The OpenThoughts team fine-tuned Qwen2.5-7B-Instruct on the s1 and LIMO sets under one recipe (the s1 set in its later version, with DeepSeek-R1’s chains of thought), and each brought it from about 8% on AIME 2025 to about 14 to 15% (values read from the paper’s figure). The team’s own 1.2 million examples brought it to 53.3%, against 38.0% for DeepSeek’s distilled 7B, which used 800,000. The Bespoke-Stratos team’s 17,000 examples raised the same model from 10.0% to 20.0% on AIME 2024. OpenThoughts, Bespoke-Stratos

The thousand-example results are therefore a finding about strong 32B models, and nearly all of them about Qwen2.5-32B. None has been replicated on a base model from another family. They are also reported on AIME 2024 and MATH-500, where the smaller Qwen2.5 models show signs of memorization.

Below about 3B, long chains of thought can make a student worse. With 7,500 math problems, Qwen2.5-1.5B fine-tuned on long chains of thought from QwQ-32B-Preview averaged 27.0 across five benchmarks, and on short solutions from Qwen2.5-32B-Instruct 34.2. At 32B the long chains won, 73.0 to 59.3. With more data the small models recover, but slowly: in one study Qwen2.5-0.5B stayed below its starting score even after 220,000 examples. Li et al., Luo et al.

What makes the examples good

OpenThoughts ran the largest controlled study of distillation data: over a thousand experiments, each fine-tuning Qwen2.5-7B-Instruct on 31,600 examples with one part of the pipeline changed. Where the questions came from mattered most. The best and worst of 27 sources of coding questions differed by 17.2 points on average, and drawing from the top one or two sources beat mixing sixteen. OpenThoughts

The rest of the pipeline mattered less. Keeping the hard questions helped, whether a language model rated the difficulty or the length of a language model’s answer stood in for it. Collecting up to 16 teacher responses per question was an effective way to grow the dataset. Removing responses with wrong answers did not help at this size. OpenThoughts

The best teacher was not the highest-scoring model. QwQ-32B produced better students than DeepSeek-R1 across math, code and science, although R1 scored slightly higher on average in the authors’ own evaluation, 65.3 against 64.2. The paper does not say why. OpenThoughts

Chains of thought should be kept whole. When s1 was trained with sequences cut at 4,096 tokens, 74% of its examples lost their endings, often including the final answer. That model thought longer, averaging 20,721 thinking tokens on AIME 2024, and scored 30.0%. Trained on complete examples it averaged 6,984 tokens and scored 50.0%. The authors’ explanation, which they did not test separately, is that a model trained mostly on examples that never finish gets few chances to learn how to finish. s1

What fine-tuning breaks

In one controlled study, distillation on math alone damaged other abilities. Huan and colleagues trained Qwen3-14B-Base on 47,000 math problems two ways. Fine-tuned on Qwen3-32B’s chains of thought, its IFEval score fell from 69.2 to 42.3. Trained with reinforcement learning on the same problems, it scored 70.0. This is one base model and one run. Huan et al.

The recipes that avoided this mixed other data in or repaired it afterwards. About a quarter of DeepSeek-R1’s 800,000 fine-tuning examples were ordinary chat, writing and factual questions. NVIDIA’s Llama-Nemotron models lost instruction following during reasoning fine-tuning and ran a separate reinforcement-learning stage on IFEval-style prompts to restore it. DeepSeek-R1, Llama-Nemotron

Fine-tuning settings from published runs

Run	Examples	Epochs	Learning rate	Other
s1, 32B	1,000	5	1 × 10⁻⁵, cosine decay	batch 16; sequences long enough that nothing is cut
DeepSeek-R1 distilled models, 1.5B to 70B	800,000	2 to 3	1 × 10⁻⁴ at 1.5B down to 2 × 10⁻⁵ at 70B, cosine decay to a tenth	batch 64; 32,768-token context
OpenThoughts, 7B, over 31,600 examples	up to 1.2 million	5	8 × 10⁻⁵	batch 512; sequences packed together
Olmo 3 Think, 7B	2.27 million prompts	2	5 × 10⁻⁵	32,768-token context; sequences packed together

NVIDIA reported that long chains of thought needed higher learning rates than ordinary fine-tuning, and that several epochs helped, especially for small models; AceReason’s accuracy kept rising until about the fifth epoch. Llama-Nemotron, AceReason-Nemotron 1.1

The reinforcement-learning stage

What one update does

The algorithm most recipes start from is GRPO, group relative policy optimization, introduced with DeepSeekMath. For each problem, the model being trained samples a group of responses. The verifier scores each one, and each response’s score is compared with the group’s average. DeepSeekMath

Take a group of 8 responses to one problem, 3 of them correct. Each correct response gets a reward of 1 and each wrong one 0. The group’s mean reward is 0.375, and its standard deviation, dividing by the group size of 8, is 0.484. GRPO’s advantage for a response is its reward minus the mean, divided by the standard deviation. That gives +1.29 for each correct response and −0.77 for each wrong one. Every token in a response receives that response’s advantage. The update raises the probability of the tokens in correct responses and lowers it for the tokens in wrong ones, in proportion to the advantage.

What happens when every response in the group is correct? Every reward equals the mean, so every numerator is 0; the standard deviation is 0 too, and implementations add a small constant to it or set the advantage to 0. Either way the problem contributes nothing to the update. The same holds when every response is wrong. The model spent the compute to generate eight responses and learned nothing from them. When one response in eight is correct, it gets an advantage of +2.65 and each wrong one −0.38, so a rare success is weighted heavily.

How GRPO weights each response in a group of eight

Each column is one sampled response to the same problem, scored 1 if correct (✓) and 0 if not (×). The bar is its advantage: the reward minus the group's mean, divided by the group's standard deviation. Every token in the response is pushed up or down in proportion to it.

3 of 8 correct

mean 0.375, standard deviation 0.484

✓✓✓×××××

right +1.29, wrong −0.77

1 of 8 correct

mean 0.125, standard deviation 0.331

✓×××××××

right +2.65, wrong −0.38

8 of 8 correct

mean 1.000, standard deviation 0.000

✓✓✓✓✓✓✓✓

every response 0

This is why the choice of training problems matters so much. In Skywork’s pool of math problems, sampled 16 times each from a distilled 7B model, 21.4% were never solved and 32.4% always were. Only the remaining 46.2% could produce any learning signal. Skywork-OR1

The group average replaces the value model in PPO, proximal policy optimization, the older method. PPO trains a second network, as large as the model itself, to predict the expected reward, which costs memory and is hard to train when the only reward arrives at the end of a response. The choice between them matters less than it first appeared. On a 16B mixture-of-experts model, DeepSeek found PPO nearly matched GRPO once one of its settings was tuned, and Open-Reasoner-Zero used PPO throughout. DeepSeek-R1, Open-Reasoner-Zero

The GRPO objective

For a question, sample G responses from the current model. The objective averages over responses and over the tokens of each response:

$$\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \Big\{ \min\!\big[\rho_{i,t}\hat A_i,\ \mathrm{clip}(\rho_{i,t},1-\varepsilon,1+\varepsilon)\,\hat A_i\big] - \beta\, D_{\mathrm{KL}} \Big\}$$

Here o_i is the i-th response and |o_i| its length in tokens. The ratio ρ_i,t is the probability the model now assigns to token t of that response, divided by the probability the model assigned when it sampled the response. Â_i is the group-normalized advantage from the worked example. The clip term stops a token contributing gradient once its ratio has moved outside the range from 1 − ε to 1 + ε in the direction its advantage favors; it does not stop the ratio from moving further through other updates. The last term penalizes divergence from a reference model, usually the starting model, weighted by β.

The 1/|o_i| factor averages each response over its own length. That weights each token of a long wrong response less than each token of a short one, which is discussed below. DeepSeekMath, Liu et al.

What gets rewarded

Most recipes reward reasoning with a program rather than a learned model. DeepSeek chose not to use learned reward models for reasoning at all, citing reward hacking: the model being trained learns to exploit the scorer instead of solving the problem. In R1’s final stage, where learned preference rewards were needed for general helpfulness, DeepSeek used them only for the last 400 of 1,700 steps. Over longer training the reward score kept rising while performance on a coding benchmark fell. Kimi was an exception: its math reward came from a learned model that compared the response with the reference answer, which it spot-checked at 98.5% accuracy. DeepSeek-R1, Kimi k1.5

A program can be exploited too. Hugging Face’s Math-Verify checker treats the reference and the model’s answer asymmetrically so that, for a problem asking to solve an inequality, a model cannot score by restating the inequality. Math-Verify

Most recipes cleaned the problems so the checker could be trusted. DAPO’s authors rewrote math problems so every answer was an integer. Kimi removed multiple-choice, true-or-false and proof questions, and any question a model could answer correctly within 8 guesses without reasoning. Skywork found problems that were broken or incomplete and still passed its difficulty filter, because a model could guess their answers, and removed one to two thousand with a language-model check. DAPO, Kimi k1.5, Skywork-OR1

For answers a program cannot check, such as free-form science answers, a second language model can compare the response with a reference answer. The two kinds of checker disagree often on such answers. In the General-Reasoner study, of the answers a Gemini model judged correct, a rule-based checker accepted only 22.2%. The studies of these judges mostly evaluate the trained model with the same kind of judge they trained against, which makes their gains hard to take at face value. General-Reasoner, Su et al.

Which problems to train on

The worked example says to train on problems the model solves some of the time. Recipes do this in two places. Before training, they drop problems the starting model solves too often: Olmo 3 removed any it solved more than 62.5% of the time, and Skywork also dropped ones it never solved. During training, DAPO’s dynamic sampling draws extra problems and discards any group whose responses are all right or all wrong, until the batch is full. Olmo 3, Skywork-OR1, DAPO

Published recipes used between about 4,000 and 105,000 problems. DAPO used 17,000, DeepScaleR about 40,000, Polaris 53,000, and Olmo 3 105,000 across math, code, instruction following and chat. Qwen3-235B used 3,995 for 170 steps, starting from a model that already scored 70.1% on AIME 2024, and reached 85.1%. DAPO, Polaris, Olmo 3, Qwen3

The settings the recipes changed

DAPO’s authors started from GRPO on Qwen2.5-32B base and added five changes one at a time. AIME 2024 accuracy, averaged over 32 samples per problem, went from 30 to 36 by excluding truncated responses from the loss, to 38 by raising the upper clipping limit, to 41 by a soft penalty on responses approaching the length limit, to 42 by averaging the loss over all tokens in each group rather than over each response first, and to 50 with dynamic sampling. Each row adds a change to the one above, one run each, so the table does not show what each change does alone. DAPO

The upper clipping limit matters because of how clipping works. Within one round of updates on a batch of samples, a token stops contributing gradient once its probability ratio passes 1 + ε in the favored direction. For a token the model gives probability 0.01, the standard ε of 0.2 cuts it off at 0.012 in that round, so a rarely chosen token can grow only slowly. DAPO raised the upper limit to 0.28 to let such tokens grow faster. The aim was to keep the entropy of the model’s next-token choices, a measure of how spread out they are, from falling too quickly. Skywork found 0.28 made entropy climb sharply in its setup, and settled on 0.25 to 0.265. The value is a tuned setting, not a constant to copy. DAPO, Skywork-OR1

Response length is not proof of learning. Averaging each response’s loss over its own length means a long wrong response is penalized less per token than a short one, which is what DAPO’s token-level averaging removes. Liu and colleagues trained a 1.5B model with GRPO and with a version that drops that averaging, and also the division by the standard deviation, under a 3,000-token limit. Under GRPO, wrong answers grew to about 2,000 tokens; under the modified version they stayed near 1,000, at similar accuracy. This was a small model with a short limit, and it does not show how much of R1-Zero’s growth in length was this effect. Part of that growth coincided with DeepSeek raising R1-Zero’s maximum response length from 32,768 to 65,536 tokens late in training. Liu et al., DeepSeek-R1

Recipes split on the penalty for drifting from the starting model. DAPO, Magistral, Skywork and Olmo 3 removed it; Skywork found it held back training in a later stage. DeepSeek-R1 and NVIDIA’s ProRL kept a penalty and periodically replaced the reference model with the current one, every 400 steps in DeepSeek’s case. No study compares the two on the same starting model. DAPO, Magistral, Skywork-OR1, Olmo 3, DeepSeek-R1, ProRL

The largest controlled study of these settings, by Khatri and colleagues, fitted curves of accuracy against compute for an 8B model, with a margin of error of about ±0.02 on the ceiling each curve approaches. A few choices moved the ceiling beyond that margin. One was the form of the loss: two newer variants, CISPO and GSPO, which handle the probability ratio differently from GRPO’s clipping, reached a higher ceiling than DAPO’s. Another was computing the model’s final layer in 32-bit precision, which narrows the gap between the probabilities the generation software computes and those the training software computes. The others were a larger batch, a longer response limit, a larger model, and, for DAPO’s loss, the upper clipping limit. How the loss is averaged, how advantages are normalized, which problems are sampled again and how stale the samples were changed how fast the model got there, not where it ended. Khatri et al.

Reinforcement-learning settings from published runs

Run	Problems × responses per step	Learning rate	Maximum response	Drift penalty	Length of run
DeepSeek-R1-Zero, 671B	32 × 16	3 × 10⁻⁶	32,768 tokens, then 65,536	0.001, reference replaced every 400 steps	10,400 steps
DAPO, 32B	512 × 16	1 × 10⁻⁶	20,480 tokens	none	about 5,300 updates, 16 per batch of samples
ProRL, 1.5B	256 × 16	2 × 10⁻⁶	about 8,000 tokens, 16,000 at the end	yes, with periodic resets	over 2,000 steps
Olmo 3 Think, 7B	64 × 8	1 × 10⁻⁶	32,768 tokens	none	1,400 steps

DeepSeek-R1-Zero and Olmo 3 sampled at temperature 1.0 during training, and ProRL at 1.2. DAPO’s clipping range was 0.2 below and 0.28 above; Olmo 3’s 0.2 and 0.272; ProRL’s 0.2 and 0.4.

What to watch during a run

Watch accuracy on held-out problems, not the training reward. DAPO’s authors found the training reward had little correlation with validation accuracy. DAPO

Entropy usually falls fast. Cui and colleagues found that across eleven base models, 73% of the fall in entropy and 76% of the gain in accuracy happened in the first 200 of 2,400 steps. Once entropy is near zero the model samples nearly the same response every time, groups rarely mix right and wrong answers, and learning stalls. Skywork found two things sped this collapse. One was taking several update steps on each batch of samples, so that later steps learned from responses an older version of the model wrote. The other was sampling at temperature 0.6 during training rather than 1.0. Entropy is a warning sign rather than a target: in the 8B scaling study, two runs with the same entropy curve ended at different accuracies. Cui et al., Skywork-OR1, Khatri et al.

Watch how many responses hit the length limit. In the 8B study, runs where 10 to 15% of responses were cut off typically became unstable. Khatri et al.

How long to let responses run

Several recipes train with a short limit on response length and raise it in stages. DeepScaleR started a distilled 1.5B model at 8,000 tokens. Its wrong responses averaged three times the length of its right ones, 20,346 tokens against 6,395, so a short limit mostly cut off responses that were wrong anyway, and short responses are cheaper to generate. It then moved to 16,000 and 24,000 tokens, and raised AIME 2024 accuracy from 28.8% to 43.1% in about 3,800 A100 hours. DeepScaleR

The Polaris team found the opposite for Qwen3-4B. Training it with a limit of 24,000 tokens damaged it in a way later stages did not undo, while training at 40,000 tokens from the start improved it steadily. They advise starting at the maximum length the model’s developers recommend. The two findings may each hold for their own starting model. Polaris

Where the compute goes

Generating responses dominates the cost. In Skywork’s runs, generation took 72 to 78% of the wall-clock time. Olmo 3 spent about 5 times as much compute generating as training for its 32B model and 14 times as much for its 7B. Skywork-OR1, Olmo 3

Generation also waits on the longest response in each batch, so several systems generate and train at the same time, updating the generator’s weights while it runs. AReaL reported training up to 2.77 times faster this way at the same accuracy. Learning from responses an older copy of the model wrote needs a correction to the update. With it, AReaL’s samples up to 8 updates stale scored within about a point of fully synchronous training; without it, accuracy was lower at every staleness they tested, in one run each. AReaL

Sharpening or new ability

Reinforcement learning with verifiable rewards mostly makes a model more likely to produce answers it could already produce. Yue and colleagues measured pass@k, the chance that at least one of k samples is correct. As training on Qwen2.5-7B went on, its pass@1 on a held-out math set rose from 10.2% to 28.3% while its pass@256 fell from 69.1% to 63.9%. In Qwen3’s comparison, reinforcement learning left pass@64 on AIME 2024 unchanged at 90.0%, while on-policy distillation raised it to 93.3%, a difference of one problem. The mode-collapse digest covers this narrowing. Yue et al., Qwen3

Controlled studies find new abilities under two conditions. The training problems sit at the edge of what the model can do, solved sometimes but not reliably. And the model already has the smaller skills the problems combine. In one synthetic study, an 8B model that had learned individual string functions learned under reinforcement learning to compose two of them, and generalized to compositions of three; fine-tuning on the same compositions did not. This evidence comes from synthetic tasks and models of 8B and smaller. Zhang et al., Yuan et al.

Controlling how long it thinks

Reasoning models spend tokens on questions that do not need them. Asked what 2 plus 3 is, QwQ-32B-Preview wrote 901 tokens and GPT-4o wrote 7. In the same study’s correct responses, the first solution the model wrote was already right more than 92% of the time; the rest of the response rechecked it. Overthinking

Within one question, shorter chains are more often right. Across 20 samples per question, DeepSeek-R1-0528’s shortest chain averaged 83.3% on three math competitions and its longest 58.9%. That compares chains after the fact: a chain may be long because the model was already lost, so the result does not show that cutting a chain short would improve it. Hassid et al.

There are three ways to limit thinking, and they differ in how much training they need.

The first cuts thinking off at inference. When Qwen3’s thinking reaches a budget, the serving code inserts a fixed sentence saying time is up, closes the thinking tags, and lets the model answer from what it has. Qwen’s report attributes this ability to the stage that merged thinking and non-thinking modes; it was not trained directly. Accuracy falls as the budget shrinks, from about 86% on AIME 2024 at 32,000 tokens to about 42% at 1,000. Qwen3

The second asks for a length in the prompt. Fine-tuning on examples labeled with their length did not make this work: the s1 authors trained a model that way, and it kept within the requested count 40% of the time. Only cutting it off enforced the limit. s1

The third puts length into the reward. L1 puts a target length in the prompt and subtracts a penalty for missing it from the reward. OpenAI’s open-weight gpt-oss-120b accepts low, medium and high effort settings, and scored 50.4%, 80.0% and 92.5% on AIME 2025 without tools at the three levels, though its model card does not say how the levels were trained. Kimi’s reward, without setting any budget, favored the shorter correct responses within each group. L1, gpt-oss, Kimi k1.5

One model or two

Qwen3 trained a single model to think or not on request. In the two stages that merged the modes and added general training, Qwen3-32B’s thinking-mode score fell from 83.8 to 81.4 on AIME 2024 and from 68.4 to 65.7 on LiveCodeBench, while IFEval rose 12 points. The report cannot say whether the merging or the general training caused the fall. Qwen3

In July 2025, Qwen’s team split its next release into separate instruct and thinking models “so we can get the best quality possible”. Qwen3.5, in February 2026, returned to one model that thinks by default. Anthropic, Google and DeepSeek now ship single models with effort levels. No one has published a controlled comparison of one merged model against two separate ones trained from the same base and data. The Register, Qwen3.5

Checking that it worked

Evaluation

AIME, the benchmark every report leads with, has 30 problems a year, so one problem is 3.3 points. Hochlehnert and colleagues evaluated 1.5B and 7B models with 20 random seeds and found a standard deviation of 5 to 15 points in pass@1 from the seed alone. Changing the temperature moved scores by up to 15 points, and different hardware by up to 8 with seeds fixed. Under one standard evaluation, several published gains from reinforcement learning on a distilled 1.5B model fell within that noise. They recommend averaging at least 30 runs on benchmarks this small; DeepSeek averaged 64 samples per problem. Hochlehnert et al.

The evaluation settings matter as much as the sample count. Reasoning models are evaluated with sampling rather than always taking the most likely token, which DeepSeek found produced more repetition. The temperature for evaluation is whatever each model’s card recommends: DeepSeek-R1’s is 0.6, Qwen3.5’s 1.0 for general tasks. The response limit has to be long enough, 32,000 tokens or more, that answers are not cut off, and the model’s own chat template has to be used. DeepSeek-R1, Qwen3.5, Hochlehnert et al.

Older benchmarks are contaminated. MathArena normalized each AIME year’s difficulty by how human contestants did, and found most models scored 10 to 20% higher on AIME 2024 than their AIME 2025 results predicted. Competitions held after a model’s training data was collected are the better test, along with the random-reward control described above. MathArena

Abilities the training did not measure

Reinforcement learning on math makes a model much less likely to say it does not know, because the reward never pays for that. On math problems with a missing piece of information, Qwen2.5-7B-Instruct declined to answer 30% of the time before training and 8% after. Adding unanswerable problems as 10% of the training set, with a reward for saying “I don’t know”, raised that to 85%. Accuracy fell on some benchmarks, by 7 points on AMC 2023. Song et al.

Instruction following can suffer. On Qwen2.5-7B, fine-tuning on chains of thought and reinforcement learning each lowered how often the model obeyed formatting constraints attached to math problems, and responses with longer chains of thought obeyed them less often. Qwen3’s general-purpose final stage, and Llama-Nemotron’s dedicated stage, were how those recipes won it back. Fu et al., Qwen3, Llama-Nemotron

The chain of thought does not reliably say what drove the answer. Researchers at Anthropic inserted hints into multiple-choice questions and looked at the cases where a model changed its answer to the hinted one. Claude 3.7 Sonnet mentioned the hint in 25% of those cases and DeepSeek-R1 in 39%. When reinforcement learning taught a model to exploit a flaw in the reward, it mentioned the exploit in fewer than 2% of examples in five of six environments. Reasoning models don’t always say what they think

Training directly against what the chain of thought says can teach a model to hide it. OpenAI found that a monitor reading an agent’s chain of thought caught 95% of its reward hacking, against 60% for a monitor that saw only its actions. When the monitor’s verdict was added to the reward, the agent kept hacking and the monitor’s catch rate fell to near zero. OpenAI and others now leave the raw chain of thought unoptimized and show users a summary. A length penalty is also pressure on the chain of thought, which a joint position paper from several labs flags as a risk to monitoring. Baker et al., Korbak et al.

What the evidence supports

The published pipelines agree on a shape: an optional small fine-tuning stage on chains of thought, reinforcement learning on problems a program can check, a mixed fine-tuning stage and a short general stage. In DeepSeek-R1’s checkpoints, the reinforcement-learning stage added more to reasoning than the stages after it, and the later stages most of the gain in instruction following and chat.

Where a stronger reasoning model exists, copying it has reached higher scores than reinforcement learning in two comparisons on a fixed starting model, and about the same in a third, with results that varied by benchmark. Reinforcement learning after distillation added points in each study that tried it. Only one comparison measured cost, and there distillation used a tenth of the compute.

Reinforcement learning mostly amplifies behaviors the starting model already shows. That is well supported on a puzzle task at 3B and consistent with the math results. Results on Qwen2.5 overstate what a given recipe does, because that family improves even under random rewards on benchmarks it appears to have memorized.

Most recipe settings are tuned values. The clipping limit, whether to keep a penalty on drift, and whether to start with a short length limit each went different ways in different setups. In the one large controlled study, the form of the loss, numerical precision, batch size, response limit, model size and, for DAPO’s loss, the clipping limit set where training ended; most other choices set how fast it got there.

Read as a default, and this is an interpretation rather than a tested plan: with a stronger teacher available, fine-tune on its chains of thought, keep general data in the mix, and then run reinforcement learning on problems the model solves some of the time. Without one, check that the starting model already shows the behaviors to amplify before running reinforcement learning from it.

Several questions remain open. No study compares starting from a base model against starting from its instruct version for this kind of training, or a cold start against none at matched compute. None compares one merged thinking and non-thinking model against two separate ones. None has shown the thousand-example distillation results outside Qwen2.5. And whether reinforcement learning adds new abilities in large models on natural tasks, rather than concentrating ones the model had, has been shown only on synthetic tasks and small models.

Further reading

• DeepSeek-R1: the January 2026 revision carries the Nature supplementary material, including costs, settings and the reward-hacking figure. The Nature version is Guo et al..
• Olmo 3: a recipe released with its training data for every stage, with per-stage numbers and ablations.
• OpenThoughts: the controlled study of distillation data.
• DAPO and Skywork-OR1: the most detailed open accounts of the reinforcement-learning stage.
• Gandhi et al.: what the starting model must already do.
• Hochlehnert et al.: how to evaluate on small benchmarks.
• On this site: mode collapse in post-training, mid-training, and mechanistic interpretability of post-training, which covers what reinforcement learning and distillation change inside a reasoning model.