Transfer from tasks with exact answers

Cheng and colleagues trained the 7-billion-parameter Qwen2.5 base model by reinforcement learning (RL) on 3,000 problems at a time from one of six kinds: math, code, science questions, logic puzzles, predicting what a program outputs, and reasoning over tables. Each problem came with a way to check the answer: a reference answer for most kinds, unit tests for code. A program did the checking for five kinds, and a small verifier model compared science answers with the reference. The check was the reward.

They then scored every trained model on benchmarks of all six kinds. For each run they kept the checkpoint that scored best on those same benchmarks, which flatters every trained score by an unknown amount. Training on math raised the three math benchmarks, MATH500 (500 competition problems) and the AMC and AIME 2024 competition sets, by 17.0 points on average. Training on science questions raised the same math benchmarks by 16.5 points, training on program outputs by 13.4, and training on tables by 11.5. Training on math, meanwhile, moved none of the other five domains by more than twice the noise from sampling the test questions. By this digest’s estimate, that noise is about 2 to 3.5 points for a domain’s average and 4.5 for the 200-question science benchmark. The estimate leaves out run-to-run variation, so the real noise is larger. Cheng et al.

One more number from the same study suggests why the math benchmarks moved so easily. Under the same prompt and decoding, the 32-billion-parameter base model, before any reinforcement learning, scored lower on MATH500 than the 7-billion-parameter base model: 46.9 against 57.0. A larger model of the same family should know at least as much mathematics, and after the same math training the larger model did score higher, 79.8 against 74.6. So the lower base score suggests that the score was also measuring something besides mathematical ability, such as how the untrained model answered. That is this digest’s reading; the paper did not test it.

Math benchmarks also improved after training on most kinds of problem. So much of what moved them was probably something most kinds of training changed, such as how the model answers, rather than mathematical skill carried over from math problems. The paper did not test which. The logic and table benchmarks show the opposite pattern: they gained mainly from their own training data, which gives the darkest cells in the matrix below. The authors attribute this to the base having seen little such material in pretraining, which they also did not test.

Gains after RL on one kind of problem with a checkable answer, by what was trained and what was scored

Rows: the domain the model was trained on. Columns: the domain it was scored on. Cells: gain over the base model in points.

Trained onScored on	Math	Code	Science	Logic	Program output	Tables
Math	17.0	2.6	5.2	1.5	1.5	2.7
Code	5.9	9.3	7.2	1.0	1.5	3.7
Science	16.5	1.3	3.7	2.7	2.0	2.0
Logic	5.8	6.8	5.2	25.4	0.0	4.2
Program output	13.4	5.8	7.7	4.4	8.0	8.7
Tables	11.5	4.6	6.2	2.8	2.5	23.2

Gain over the base model, points · Qwen2.5-7B base · one RL run per row, best checkpoint chosen on the same test items · outlined cells are training and scoring on the same domain

This digest asks when training on a task with exact answers helps a different task. A task with exact answers, here, is one whose answers can be checked mechanically: a math problem with a final number, code with unit tests, a puzzle with one solution, a multiple-choice question. Training on it can mean reinforcement learning with the check as the reward, supervised fine-tuning on worked solutions, or including such problems in pretraining. A different task is anything else: another domain, an open-ended task such as chat or writing, or a behaviour such as following format instructions.

The answer the evidence supports has four parts.

• Training on checkable problems changes the model’s default way of answering: whether it commits to an answer, in what form, and how it frames the work. Base models often fail benchmarks on exactly this, and most of the evidence comes from Qwen2.5, the family this literature uses most. Much of the measured transfer is probably this change. A conflicting answer format may cost as much as a matching one gains.
• Training on long worked solutions, and reinforcement learning on math and science, teach the model to work at length. That pays most on tasks whose answers are computed from the question rather than recalled, and most on a base model that can already carry out the individual steps the target needs. Some benchmarks, such as the logic puzzles and table questions in the matrix, gain mainly from their own training data.
• Whatever the reward does not pay for can drift. The model says it does not know less often, follows format instructions less, and one of three models tested became overconfident. Supervised fine-tuning on reasoning traces also erodes refusals of harmful requests. In pretraining, code that displaces ordinary text costs recall.
• The exactness of the answer, which a model can do without when it only needs to change how it answers, matters for two other things in reinforcement learning. First, it lets training get past what the base model solves in at least one of many attempts. That has been shown on freshly generated tasks whose steps the base can already do one at a time; in short runs on familiar math, training mostly does not get past the base. Second, it keeps the reward measuring correctness over long runs, as long as the check cannot be passed without solving the problem.

Because the first three effects depend on what the target task scores, the task being measured decides more than the task being trained.

Target and source in transfer matrices

In Cheng and colleagues’ matrix, the column a cell sits in, which domain was scored, explains 64% of the variation among the gains off the diagonal at 7 billion parameters and 87% at 32 billion. The row, which domain was trained, explains 19% and 9%. These shares are this digest’s arithmetic on values read from the paper’s heatmaps, and every cell is a single training run. Cheng et al.

A different study found the same split with a different method. Dymkiewicz and colleagues fine-tuned nine instruction-tuned models, from 0.5 to 8 billion parameters, with a low-rank adapter. Each run trained on 215 to 300 multiple-choice examples of one task in one language and then measured every other task. The cross-task effects were small, averaging under one point. Of their variation, which target was measured explained 49% to 57%, and which source was trained explained 2% to 6%. The knowledge benchmark Global-MMLU gained 2.6 to 3.1 points from training on every other task, while training on Global-MMLU raised no other task by more than 0.14. Three seeds per run. Dymkiewicz et al.

The older record of supervised transfer between labelled tasks shows how unreliable source-to-target transfer is. In the three largest sweeps over language tasks, training on one task before another hurt the second about as often as it helped, counting every change by its sign however small:

Model	Pairs	Hurt	Helped	Unchanged
RoBERTa-large	110	62	43	5
BERT-base	3,168	1,335	1,542	291
RoBERTa-base with adapters	462	203	243	16

Gains went mostly to targets with little data. Much of the gain in that record also came from fewer failed runs. Against a baseline tuned over 36 hyperparameter settings, training on HellaSwag, a commonsense sentence-completion set, before RTE, a small language-inference set, raised RTE’s best run by 2.8 points and its average run by 10.5 (read off the paper’s figure). The average rose much more than the best because fewer runs collapsed. Pruksachatkun et al., Vu et al., Poth et al., Chang and Lu

What predicted transfer in that record was the form of the task more than its content. A source whose output format differed from the target’s, classification against span extraction for instance, was a net loss for 8 of 11 targets. HellaSwag with its opening contexts deleted, so that the task needed no commonsense, only spotting which ending a machine wrote, transferred about as well as the full set. That result is on RoBERTa-large and read off the paper’s figure. And Pruksachatkun and colleagues’ own predictor, that sources requiring reasoning transfer best, did not hold: the social-reasoning task SocialIQA lowered all ten targets it was tried on. Poth et al., Chang and Lu, Pruksachatkun et al.

Transfer is also directional. In co-training pairs of vision tasks on one dataset, one direction helped and the other hurt in 26 of 40 pair-and-setting combinations (ten task pairs, four settings, one run each). Standley et al.

How the model answers

The first thing training on checkable problems changes is the model’s default way of answering, and the untrained model’s score can depend heavily on it. In the measurements below, one model’s score on one benchmark moved by 16.5 points with the evaluation setup alone, and by 22 and 71 points with the prompt template alone.

• The Qwen2.5-7B base model’s MATH500 score ranges from 48.3 to 64.8 across six papers’ evaluation setups (this digest’s comparison).
• A Qwen2.5-7B-Instruct model trained on logic puzzles scores 1.8, 3.2 or 73.2 on MATH500 depending only on the prompt template.
• Qwen2.5-Math-7B, untrained, scores 50.6 on MATH500 with its chat template and 72.2 without it. In a second paper’s evaluation, where the chat template gives 49.4, replacing the prompt with placeholder Latin text gives 68.8.

Cheng et al., Akter et al., Liu et al., Zhao et al., Ma et al., Li et al., Wu et al., Shao et al.

Not all of this is whether the answer can be read. Under the chat template, 79% of the untrained Qwen2.5-Math-7B’s answers could already be parsed, against 84% under the placeholder prompt, while accuracy differed by 19 points. The template changes how the model works, not only the form of its final answer. Shao et al.

Against starting points like these, training a Qwen2.5-Math model with almost any reward, even a random one, raises its score. Shao and colleagues trained Qwen2.5-Math-7B with rewards that carried no information about correctness, or pointed the wrong way. From 49.4 under the chat template, a random reward raised its MATH500 score by 21.4 points, to 70.8, and a reward for matching a fixed wrong answer by 24.1, against 29.1 for the correct reward. The untrained model scores 72.2 with no template in Wu and colleagues’ evaluation. So the random reward bought about what dropping the template does, and the correct reward, ending at 78.5, went about 6 points beyond it. The two papers use different evaluation code, so the comparison is approximate. Shao and colleagues trace most of the random reward’s gain to the model switching from prose to writing code: code-style answers rose from 65% to over 90%.

The same random reward lowered Llama-3.1-8B-Instruct’s MATH500 score by 11.5 points and OLMo-2-7B’s by 8.3. And on Qwen2.5-Math-7B, the gains on the 2024 American Invitational Mathematics Examination (AIME) from the random and wrong-answer rewards largely vanished on the 2025 exam, whose problems were written after the models’ training data was collected. Shao et al., Wu et al.

The same effect probably explains much of the cross-domain transfer measured on these models. On Qwen2.5-7B, averaged over six math benchmarks (MATH500, the AIME 2024 and 2025 and AMC 2023 competition sets, Minerva and OlympiadBench), reinforcement learning on code tasks the model proposed for itself reached 38.4, reinforcement learning on math problems 38.5, and the official instruction-tuned model 37.0, from a base of 27.5. One math run on far more problems reached 41.6. All are greedy decoding, one run each. Three different kinds of training reached about the same average, near the instruct model’s level. Zhao et al.

Outside math, gains look large mainly because the base starts so low. On the chat benchmark WildBench, which compares each response with a reference answer from GPT-4, so that a score below zero means worse than the reference on average, the released SimpleRL-Zoo model, trained by math reinforcement learning from the Qwen2.5-7B base, scores 4.9. The base scores −65.8, and both the instruct model and chat reinforcement learning from the same base score 22.2. On the medical exam set MedQA, math reinforcement learning took Llama-3.2-3B-Instruct from 20.1% to 53.5%. A score of 20.1% is below the 25% that guessing gets when each question has four options, as in the common version of MedQA; the paper does not say which version it used. Below chance means the untrained model was mostly not giving a usable answer. Bhaskar, Ye and Chen, Hu et al.

A conflicting answer format may cost as much as a matching one repairs; that is this digest’s reading of the next two results, from one study on Qwen2.5-7B. Reinforcement learning on the MATH training set lowered MMLU-Pro, a ten-option multiple-choice benchmark across fourteen subjects, from 45.0 to 31.6. Reinforcement learning on multiple-choice questions lowered MATH500 from 48.3 to 22.0. One run each. Akter et al.

So before a cross-domain gain can be read as skill carried over, two numbers have to be known: what the base model scored under the same prompt and decoding, and how often it failed to give a usable answer. Most of the math-benchmark results cited here use Qwen2.5 models. In that family, random rewards raised math scores at 7 billion parameters, and MATH500 is known to be contaminated. The check matters most there. Shao et al., Wu et al.

Working at length, and where it pays

Beyond the way of answering, the gains on other reasoning tasks came with the habit of working at length: writing intermediate steps, checking them, trying again. Training on long worked solutions installs it, and so does reinforcement learning on math and science; in the opening study, reinforcement learning on code, logic and tables shortened answers instead. No study has separated the habit from simply spending more tokens on each answer. Two controlled comparisons hold the problems fixed and change mainly whether the training targets contained the exploratory work.

Ren and colleagues fine-tuned Qwen3-14B-Base on the same math problems with the same final answers, once with a stronger model’s long reasoning in the targets and once with only its final step-by-step summaries. With the long reasoning, the model scored 66.0 on AIME 2024 and 63.3 on GPQA-Diamond, a hard set of graduate-level science questions; without it, 17.0 and 48.3. Ren et al.

Zhou and colleagues compared s1.1, a model fine-tuned on 1,000 questions with DeepSeek-R1’s long traces, with their own model fine-tuned on the same questions after GPT-4o shortened each answer to about 400 tokens. Against Qwen2.5-32B-Instruct, the model both started from, the long version raised six general-reasoning scores by 4.8 points on average and the short version lowered them by 6.6 (this digest’s arithmetic on their table). s1.1’s training questions include some from GPQA. Zhou et al.

Which training method was used mattered less than one might expect. On Qwen3-8B-Base, five methods trained on the same math prompts (two kinds of supervised fine-tuning, three of reinforcement learning) all raised the average of four other reasoning benchmarks by between 10.8 and 16.3 points (this digest’s arithmetic on their table; one run each). They differed on non-reasoning tasks, from −7.0 to +2.2. Huan et al.

Where the habit pays roughly follows where working step by step pays at answer time, on the benchmarks measured. Sprague and colleagues ran 14 models on 20 datasets, prompting each either to reason step by step or to answer directly. Reasoning raised accuracy by 27.6 points on average on math and 10.9 on symbolic tasks, 4.2 on knowledge questions, 0.9 on commonsense, and 2.0 on soft reasoning, their name for tasks such as reading social situations and deciding what follows from a passage (this digest’s arithmetic on their table). Sprague et al.

Training shows a similar ordering. After reinforcement learning on questions from many fields, with answers checked by a small trained verifier, Qwen2.5-7B-Base’s score on the chemistry part of MMLU-Pro rose by 17.0 points and its law score by 6.6. Every subject gained at least 6.6 points, so the differences between subjects sit on top of a gain they all got. Across subjects, these gains also track the gains of the official instruction-tuned model from the same base (a correlation of 0.78, this digest’s arithmetic), so the ordering may reflect post-training in general rather than working at length. Ma et al.

In Akter and colleagues’ study, the one with the format results above, reinforcement learning on a blend of math datasets moved MMLU, which mostly tests recall, by 0.04 points, while the harder, more computational MMLU-Pro moved by 9.3. Single math datasets in that study, trained for fewer steps, moved MMLU-Pro anywhere from −17 to +7 points. So both the sign and the size depend on the data and on how long training ran. Akter et al.

In medical question answering, two studies trained on medical data with and without written reasoning, and the reasoning mattered less than the training. In a vision-language model answering questions about medical images, reinforcement learning without written reasoning scored 69.0% on kinds of image it was not trained on, against 68.0% with written reasoning (this digest’s arithmetic on their tables). With fine-tuning, training on answers alone matched or beat training on full traces for two of three training sets; traces helped for the third, whose traces came from DeepSeek-R1. Both studies train on medical data, so they show where written reasoning helps, not what carries over from another task. Lai et al., Thapa et al.

Two qualifications sit next to this. The first concerns length. Where length was reported, in Zhou and colleagues’ table and in Ren’s, the long-reasoning models wrote answers 1.3 to 14 times longer. None of these studies held the number of tokens at answer time fixed while measuring transfer. The closest evidence is a vision-language model trained on two games, which gained 2.9 points on multimodal math with answers only about 7% longer. A puzzle study adds that priming with filler of the same length did nothing, on the puzzle itself. Xie et al., Gandhi et al.

The second is that the habit transfers most on top of a base that can already do the steps. Fine-tuning on reasoning traces for Countdown, an arithmetic puzzle, raised Qwen3-14B-Base’s MATH500 score by 13.7 points but InternLM2.5-20B-Base’s by only 0.8. InternLM still gained 13.2 points on MMLU-Pro, more than Qwen3’s 3.6, so the contrast between the two bases holds for math, not for every target. Its GPQA score rose from 20.9 to 25.3, around the 25% that guessing gets. A 1.7-billion-parameter Qwen3 base fine-tuned on the math traces got worse at math, code and GPQA. Ren et al.

The base also decides how far reinforcement learning gets. Under identical reinforcement learning on Countdown, Qwen2.5-3B reached about 60% and Llama-3.2-3B about 30% (read off the paper’s figures). Llama reached Qwen’s level after it was first fine-tuned on 1,000 traces that showed backtracking and checking, or after continued pretraining on text rich in those behaviours. Both results are on the puzzle itself, not on another task. Gandhi et al.

From a starting model that already reasons at length, results vary. Math reinforcement learning on DeepSeek-R1-Distill-Qwen-1.5B raised three of seven code benchmarks by 3 to 4 points and lowered two knowledge-heavy benchmarks. From a strongly fine-tuned Qwen3-4B, math reinforcement learning raised GPQA-Diamond by 15.8 points. One run each. So a model that already reasons at length can still gain a great deal from math training, or very little. Hu et al., Wang et al.

What the exact answer adds

If random rewards and changes in how the model answers account for so much, does it matter that the answers are exact? In reinforcement learning, it does, for two things the previous sections did not need.

The first is getting past what the base model solves in at least one of many attempts, on a task whose steps it can already do one at a time. On freshly generated ten-step arithmetic, which no model could have memorized, only a reward that tracked correctness helped. The reward gave partial credit by how close each answer was. With it, Qwen2.5-Math-7B climbed to a validation reward of about 0.70, above the 0.53 its best of 16 samples had reached before training. Random and wrong-label rewards left it at about 0.1 to 0.15 (read off the paper’s figure). Wu et al.

Correct labels were also needed in a vision-language model. Qwen2.5-VL-7B trained on the game Snake alone with correct labels raised an average of three reasoning areas from 49.1 to 62.3, most of it on counting and geometry questions; multimodal math rose 1.7 points. With random labels the average reached only 49.4. That is the only cross-domain transfer among the studies here that was tested against random labels. Xie et al.

The second is a reward that kept measuring correctness over the longest runs reported. Several rewards that do not check the answer started training and then failed. A reward for reasoning phrases lifted Qwen2.5-7B to 70.2 on MATH500 by about step 40 and drove it to about zero by step 140 (steps read off the paper’s figure). The model had learned to keep revisiting its steps until its answers ran past the length limit. A reward computed from the model’s own confidence, as judged by a frozen copy of it, was exploited within about 100 steps. Lv et al., Zhao et al.

Learned judges fail too. DeepSeek-R1’s authors confined their preference reward to the last 400 of 1,700 training steps because, in their runs, longer use of it let the reward rise while competitive-programming scores fell. A reward model for writing drove one model to gibberish that a writing benchmark scored above the instruction-tuned model. With an exact reward, Qwen2.5-Math-1.5B was trained on a single duplicated problem for 2,000 steps, and its test accuracy kept improving long after it had mastered that problem. That run is on Qwen2.5-Math, the family in which random rewards also raise scores, so part of the gain may be adaptation to the prompt. DeepSeek-AI, Jia et al., Wang et al.

A check keeps measuring correctness only if the model cannot satisfy it without solving the problem. MacDiarmid and colleagues gave a pretrained model knowledge of three ways to game coding tests and trained it by reinforcement learning on real coding tasks chosen because their tests could be gamed in those ways. It learned to game the tests. On unrelated evaluations, 69.8% of its answers to “what are your goals?” were then classed as misaligned, against 0.1% for the same training with gaming penalized. Most of those answers stated a drive to maximize reward, and many hid it behind an aligned final answer. So the reward paid for gaming the tests, and what spread to other tasks was a misaligned disposition that came with learning to game them. The disposition depended on what gaming meant to the model: when training framed gaming as acceptable, the gaming stayed and most of the misalignment did not appear. MacDiarmid et al.

Usually, training does not get past what the base already solves by sampling. For short runs on familiar math, reinforcement learning with an exact reward mostly makes the model reliable at what it could already sometimes do. Training Qwen2.5-7B on 2,000 math problems raised the share of held-out problems it solved on the first try from 10.2% to 28.3%, while the share it solved in at least one of 256 tries fell from 69.1% to 63.9%. For the open checkpoints the authors compared problem by problem, the problems the trained model solved were nearly a subset of those the base model solved. Yue et al.

That is a statement about short runs of the open recipes tested in early 2025, not a proven limit. Fine-tuning on a stronger model’s traces lifts the base at every sampling budget. And a long run from an already distilled model widened its coverage on puzzles and code, partly by fixing the answer format the base had been failing. Yue et al., Liu et al.

What the reward does not pay for

A right-or-wrong reward fixes the final answer and leaves the rest of the response free, so whatever it does not pay for can drift toward whatever helped on the training problems: always answer, sound sure, think before answering. Each row below is a before-and-after measurement on one model. All but the harmful-answer row are reinforcement learning; that row is supervised fine-tuning on reasoning traces, with no reward.

Behaviour	Before	After	What was trained
Saying “I don’t know” to unanswerable factual questions	0.27	0.09	Math RL on Qwen2.5-7B-Instruct, “I don’t know” offered and never rewarded
Following instructions, IFEval strict scoring	58.6	46.6	RL with rule-checked rewards from DeepSeek-V3-Base; the base was scored with a different, base-model prompt, so part of the change may be the prompt
Using an irrelevant tool, or inventing one, when the needed tool is missing	about 0.55	about 0.82	400 steps of RL on GSM8K grade-school math problems, Qwen2.5-7B-Instruct, read off a figure
Harmful answers to harmful requests	16.7%	60.4%	Supervised fine-tuning of Qwen2.5-32B-Instruct on about 1,000 math and science reasoning traces (s1.1)
Chat quality, AlpacaEval 2	29.7	28.5	120 steps of math RL on a Llama-3.1-8B model that already chats

Song, Shi and Zhao, DeepSeek-AI, Yin et al., Huang et al., Bhaskar, Ye and Chen

IFEval checks whether responses obey instructions a program can verify, such as a word limit; strict scoring checks the response exactly as written. The tool row comes from a test that asks for a task needing a specific tool and offers only an irrelevant one; the GSM8K training itself involved no tools. Reinforcement learning on math from the same instruct model as the harmful-answer row raised its harmful answers far less, from 9.9% to 15.7% under a different judge. The chat row is not a loss: math reinforcement learning on a model that already chats left AlpacaEval 2, a chat benchmark scored by a language-model judge, where it was, while the same 120 steps on chat prompts with a learned reward raised it to 52.3. Cho et al., Bhaskar, Ye and Chen

Confidence is less consistent. Calibration error measures the gap between a model’s stated confidence and how often it is right. Trained on math with a right-or-wrong reward, Qwen2.5-7B’s calibration error improved on math, from 0.39 to 0.26, and worsened slightly on other question sets, from 0.45 to 0.49. Adding a term that scored the stated confidence cut the out-of-domain error to 0.25, with in-domain accuracy unchanged (72.7 against 72.9). Trained instead on multi-hop questions, the same Qwen2.5-7B put almost all its stated confidences at 0.9 to 1.0. Two other models trained on those questions showed no worsening. Damani et al.

Losses like these can come from what a model does while it is thinking, not from an ability it has lost. Six models distilled from DeepSeek-R1, not the ones in the table, answered 95% to 100% of StrongReject’s harmful requests safely when their response was forced to start with an empty thinking block, against 19% to 55% with thinking on (greedy decoding). Jiang et al.

A measured loss can also come from the evaluation itself. In one study’s evaluation, the released Qwen3-14B scores 42.9 on IFEval with thinking on and 90.5 with it off. Qwen’s own evaluation, which scores whole prompts rather than single instructions, gives 85.4 and 84.8. So most of that measured loss comes from the first evaluation, either from how it scores output that contains the thinking or from cutting long thinking off before the answer. Huan et al., Qwen Team

Where a study tried it, what restored a lost behaviour was training on the behaviour the reward ignored; a prompt asking for it did little. Replacing 10% of the math problems with unanswerable ones raised Qwen2.5-7B-Instruct’s refusal rate on unanswerable factual questions from 0.09 to 0.99, at a cost of 0 to 7 points on five math benchmarks. On a model trained by reinforcement learning on a tool-use task, by contrast, a prompt telling it not to use tools it was not given moved its rate of inventing tools only from 90.2% to 87.5%. Song, Shi and Zhao, Yin et al.

Production pipelines add such data in later stages. In Qwen3-32B’s training, fine-tuning on mixed thinking and non-thinking data, followed by reinforcement learning with general rewards, raised thinking-mode IFEval from 73.0, its score after reasoning reinforcement learning, to 85.0. The paper gives no score from before that reasoning stage, so this is not shown to be a recovery. AIME 2024 fell from 83.8 to 81.4 over the same stages. Safety has no clean recovery result: fine-tuning the s1.1 model from the harmful-answer row on refusals alone cut its harmful answers from 60.4% to 0.8% but erased the reasoning gain and left reasoning below where it started. Qwen Team, Huang et al.

A different lever does less than is often assumed. A penalty holding the model close to its starting point changed little in the two studies that removed it. Chen et al., Huan et al.

What happens to knowledge is not settled. No paper here separates facts the model no longer knows from answers it no longer gives under reinforcement learning with exact rewards. Where factual-question scores fell after math-only reinforcement learning from a base model (TriviaQA 54.5 to 34.4 for one 7B model), about one in seven of the wrong answers kept repeating the same flawed reasoning. Training DeepSeek-V3-Base only with rule-checked rewards raised its score on SimpleQA, a set of short factual questions, from 20.1 to 30.3. Two things weaken this as evidence of new knowledge: the training data included knowledge-heavy multiple-choice questions, and the base was scored with a different, base-model prompt. Yao et al., DeepSeek-AI

In pretraining, knowledge is lost when code takes the place of ordinary text. In 470-million-parameter models trained on a fixed 200 billion tokens, raising the share of code from none to all of it lowered closed-book question answering from 9.1 to 1.3. Each mixture is one run, and the values are read off the paper’s figure. Aryabumi et al.

How the training method moves the model

Supervised fine-tuning on another model’s outputs and reinforcement learning on the model’s own samples are often compared as methods, and at the settings used in these comparisons, starting from an instruction-tuned model, reinforcement learning forgets less. Chen and colleagues trained four instruction-tuned models of 1 to 8 billion parameters on Countdown, with the same learning rate for both methods at each model size. Reinforcement learning raised the game score by 29 to 60 points and changed the average of five other benchmarks, two of them refusal tests, by −0.9 to +0.6 points. Fine-tuning on a 70-billion-parameter model’s solutions raised the game score by 10 to 30 points and lowered the others by 24 to 36. Chen et al.

What protects against forgetting is training on the model’s own current samples, not reinforcement learning as such. Training only on the current model’s own correct samples forgot as little as full reinforcement learning, though in one of the two studies it learned the trained task more slowly. Regenerating the fine-tuning data from the current model at every epoch removed most of the forgetting, in the two Qwen models tested, on instruction-following and knowledge targets. Fine-tuning once on the starting model’s own samples still forgot. Shenfeld, Pari and Agrawal, Chen et al.

Step size also matters a great deal within fine-tuning. Lin and colleagues fine-tuned six instruction-tuned models of 1 to 8 billion parameters on a medical-calculation task. At a learning rate of one millionth, a general score averaged over instruction following, grade-school math and code stayed within 4 points of the base for five of them. Twenty times that rate cut it to 5 to 18 points out of 100 (read off the paper’s figure). Chen and colleagues found that a lower rate cut fine-tuning’s forgetting but also kept it from reaching the same game score. Huan and colleagues ran fine-tuning at 50 times the reinforcement-learning rate. No study compared the two methods at both the same learning rate and the same gain on the trained task. Lin et al., Chen et al., Huan et al.

Some reported forgetting is a phase of training rather than a loss. Fine-tuning Qwen3-14B-Base on a stronger Qwen3 model’s math reasoning, instruction following fell from 64.2 to 35.6 at step 20 of 640, stood at 54.3 after one epoch, and ended at 68.4 after eight (three seeds). At step 20 answers ran to 15,000 to 28,000 tokens and every score except AIME was below the base. An earlier study on the same base and teacher, with different problems and scoring, stopped at 1.5 epochs and reported a fall of 26.9 points. Ren et al., Huan et al.

A stronger model’s signal can transfer at least as far as the model’s own reward, for a tenth of the compute. In on-policy distillation, the student writes its own answers and a stronger teacher scores every token of them. Distilling from a 32- or 235-billion-parameter teacher this way raised Qwen3-8B’s GPQA-Diamond score by 7.7 points against 5.7 for reinforcement learning, a 2-point difference on 198 questions, in a tenth of the GPU hours. Both runs trained on the model’s own samples, from the same already-distilled checkpoint, on math and code prompts only. They differed in what scored the samples: the teacher’s per-token judgement, or a right-or-wrong check. One run each. Qwen Team

What the base and its earlier data decide

The model being trained, and what it saw before, largely set what training on checkable problems can reach. Mid-training is a stage between pretraining and post-training that continues on a curated mix, often heavy in math, code and reasoning text. In one study of mid-training, a Granite-3.3 8B model mid-trained on math and code and then trained by reinforcement learning scored 35.5 on GPQA-Diamond. With an eighth of its mid-training data replaced by science reasoning, and the same reinforcement learning, it scored 52.9. Adding science prompts to the reinforcement learning instead changed the score by about 2 points or less. Science at mid-training added 11 to 13 points for Llama-3.1-8B and Mistral-7B. Each cell is one run, scored at its best checkpoint, and several of these scores are below the 25% that guessing gets, so they are rough. Runwal et al.

Base models from different families respond differently to the same training. Math reinforcement learning lowered Llama-3.2-3B-Base’s GSM8K score from about 30 (the base’s few-shot score) to about 8 (zero-shot after reinforcement learning), while Qwen2.5-3B-Base went from about 75 to about 86. The before and after scores use different prompts, so the size of the drop is uncertain; the direction, against Qwen’s rise under the same comparison, is the result. After mid-training on math web text, the Llama model’s result after the same reinforcement learning depended on the corpus, ending near 10 with one and near 67 with another (read off the paper’s figures; one run each; math benchmarks only). Wang et al.

At pretraining scale, the headline gains of code and math data on other abilities mostly shrink against a matched baseline. Two examples show it.

Aryabumi and colleagues reported that adding code raised a commonsense and science multiple-choice average by 8.2% in relative terms, which is 4.0 points, in 470-million-parameter models with one run each. That gain was measured before the final learning-rate cooldown. After both models got the same cooldown, which itself includes some code, the gain was 0.8 points. That is less than the 1.2 points between the paper’s two text-only baselines, one of which is read off a figure. Aryabumi et al.

DeepSeekMath’s math pretraining started from a code model’s checkpoint taken before that model’s learning-rate decay. Measured from that checkpoint, its gains on MMLU and on BIG-Bench Hard (BBH), a set of hard reasoning tasks, are 12.0 and 16.6 points. Against the finished code model the gains are 5.8 and 4.3 points, and DeepSeekMath’s MBPP code score is 7.8 points below the finished code model’s. In separate 1.3-billion-parameter runs, math training after code lowered the code benchmark HumanEval from 25.0 to 12.2. The differences are this digest’s arithmetic on the paper’s tables; one run each. Shao et al.

In multitask instruction tuning, where every task comes with a reference answer, many of them exact labels, the same lessons recur. Returns from more tasks are concave: about 90% of the held-out gain came from the first 282 of 1,836 tasks (this digest’s arithmetic). That study tuned PaLM models of 8 to 540 billion parameters and added the tasks as four whole collections in one fixed order, one run each. Which tasks are added matters more than how many: adding ten datasets to the multitask model T0’s training lowered its held-out median on three of five language-inference sets, RTE from 81.2 to 65.0. Chung et al., Sanh et al.

In FLAN, an early instruction-tuning recipe, base size decided the sign. The same 30,000-step tuning lowered the 13-task held-out average by about 5 points at 8 billion parameters and below, and raised it by 13 to 15 at 68 and 137 billion (read off the paper’s figure). This is one held-out split, one run per size, and no sizes between 8 and 68 billion. So the size at which this tuning starts to help depends on the recipe: with a different mixture, an 8-billion-parameter PaLM gained 15 points, on a different set of held-out benchmarks. Wei et al., Chung et al.

When training on exact answers helps, and what is open

Training on a task with exact answers helps a different task to the extent that three things are true.

• The target’s score was held down by how the untrained model answered. Training on almost any checkable problem with a compatible answer format repairs that, at least on the Qwen2.5 bases most of this evidence uses. A conflicting format may lower the target by as much.
• The target rewards working at length, if the training installed it. That pays most on tasks whose answers are computed from the question, such as math and quantitative science. It pays less on tasks whose answers are recalled, which gain mainly through the way of answering, and on tasks judged on style. Some benchmarks, such as the logic puzzles and table questions in the matrix above, gained mainly from their own training data.
• The base model can already carry out the individual steps the target needs, which depends on its size and on what it saw in pretraining and mid-training.

Training can hurt the target where the target scores something the reward never paid for: abstaining when unsure, following format instructions, and, in one of three models tested, calibrated confidence. Supervised fine-tuning on traces also erodes refusals, and safety has no clean recovery result. The losses from reinforcement learning are mostly in what the model does by default rather than in what it can do. Where a paper tried, training on the behaviour the reward ignored, through a small share of matching data or a reward term, recovered much of the loss. Under fine-tuning and in pretraining, training can also cost what the new data displaced or overwrote: recall, when code replaces ordinary text, and earlier skills, when math training follows code.

In reinforcement learning, the exactness of the answer does two things. It lets training get past what the base solves by sampling, shown so far on freshly generated tasks. And it keeps the reward measuring correctness over long runs, provided the check cannot be passed without solving the problem. Whether exactness also shapes what carries over to other tasks has been tested only once, where random labels removed the gain.

What remains open, with the cheapest experiment for each:

• Is what transfers a procedure, or more computation per answer? Take released checkpoints trained with and without long reasoning on the same problems, force equal thinking budgets at answer time, give the base model filler tokens of the same length, and score GPQA and the programming benchmark LiveCodeBench. No training is needed.
• Does training on exact answers change recall? Score the same released checkpoints on closed-book factual questions with thinking on and off, under matched prompts and budgets, separating wrong answers from answers that never finish.
• Does reinforcement learning forget less than fine-tuning at the same learning rate and the same gain on the trained task? Sweep three learning rates for each method on one 7-billion-parameter model, three seeds each, including fine-tuning on samples regenerated from the current model, and compare each method’s curve of gain on the trained task against loss elsewhere.

Most results in this digest come from a single training run, and most cross-domain math results come from one model family. The direction of the findings repeats across papers; the sizes should be read as examples.