Post-training protein language models

Sep 14, 2026 · 13 min read

Digests are written primarily by LLMs with only readability edits by me. If this one reads like slop, please shoot me an email at max.zhuang@berkeley.edu.

Post-training teaches a protein language model to favor proteins that are useful for a particular purpose. A model trained on natural sequences can already propose plausible proteins. Further training can make it favor variants that stay folded, fluoresce more brightly, or fit a supplied structure. The question is how to turn those goals into something the model can learn.

Reinforcement learning (RL) for text models supplies a useful comparison. A protein generator can propose candidates, receive feedback, and change its future proposals. But protein language models (pLMs) generate in several different ways: some extend a sequence, some fill missing amino acids, and some repeatedly denoise an incomplete sequence. The architecture and generation procedure shape how that feedback becomes a training update.

The field’s recent work explores both sides of that relationship. ProteinDPO uses measured stability to teach preferences. RLXF and ESM3 adapt masked models. DRAKES trains through diffusion steps. Online methods generate new training examples as they improve. Following these choices explains more than reading every paper as another application of RL.

From protein families to measured preferences

The starting idea was already present in UniRep in 2019: train on a broad collection of proteins, then continue training on evolutionarily related sequences. Its evotuning procedure adapted a reusable model to a narrower biological setting. ProGen later demonstrated this approach for generation: after fine-tuning on lysozyme families, it produced enzymes with experimentally measured activity. UniRep, ProGen

Family adaptation gives the model examples of the kind of protein to make. Experimental measurements provide a more specific signal: which variants do the job better. Two sequences can both resemble natural enzymes while differing substantially in stability or activity.

Large measurement datasets made that second signal easier to use. In 2023, Tsuboyama and colleagues released Megascale, with roughly 776,000 curated measurements of folding stability across hundreds of small protein domains. It supplied many related variants whose relative stability could be compared under a shared experimental procedure. Stability became a particularly tractable target for post-training because this dataset combined measurement scale with biological diversity. Megascale

ProteinDPO: learning from the better and worse variants

ProteinDPO, first posted in May 2024, combined these measurements with direct preference optimization, or DPO. The starting model, ESM-IF1, performs inverse folding: given a protein’s backbone, the three-dimensional arrangement of its main chain, it proposes a compatible amino-acid sequence. The design task is then specific: keep the target structure fixed and teach the model which sequences to prefer for it. ESM-IF1, ProteinDPO preprint

Imagine two variants associated with the same backbone, one measured as more stable than the other. Supervised fine-tuning could train the model to imitate the more stable variant. DPO also uses the comparison: it increases the model’s relative preference for the better variant, measured against an unchanged reference model. The reference preserves a connection to the pretrained model’s preferences while the new feedback changes them. DPO

The less successful measurements are used as well, since they help specify which differences matter. Training can use an existing dataset of comparisons, so it does not require generating and testing a new protein for every update.

ProteinDPO’s results show what the choice bought. The model improved stability prediction beyond both the original ESM-IF1 and a version fine-tuned on stabilizing examples. The 2026 journal paper also reported transfer to larger proteins and binding-affinity prediction, despite training on stability changes in small domains. Its experimental follow-up included stabilizing a vaccine antigen. These results suggest that preference training can teach information that transfers beyond the immediate training examples. ProteinDPO journal paper

A large pretrained model may already contain useful information about a property, while a comparatively focused experimental dataset teaches it how to use that information. That possibility raises a question for every new method: how much does comparing successes and failures add over training on the successes alone?

How generation changes the training method

ESM-IF1 makes DPO convenient because it generates a sequence one amino acid at a time. Each choice has a probability conditioned on the structure and the residues already generated. Multiplying those probabilities gives the probability of the sequence. DPO can compare those sequence probabilities; an RL method can instead use rewards to update the decisions made during generation. ESM-IF1, DPO

Autoregressive generation is therefore a convenient fit for familiar post-training methods. Masked and diffusion models require working with their own generation decisions. What distinguishes them is what the model predicts, what context it sees, and how those predictions become a protein.

Model and generation procedure	What the model does	What post-training must account for
Autoregressive (AR)	Chooses the next residue given the generated sequence and any prompt	Probabilities along the sequence provide a direct basis for preference losses and policy updates
Masked language model (MLM)	Fills hidden residues using the visible sequence around them	The choice of positions, masking pattern, and order of edits determines the generation process
Discrete diffusion	Repeatedly denoises a corrupted or masked sequence	Training can act on the denoising steps, using suitable probability estimates or differentiable approximations

These labels combine choices about model structure, pretraining, and sampling; they are not three mutually exclusive network architectures. Masked diffusion, for example, uses a masked prediction task across different noise levels. DPLM, introduced in 2024, learns to reconstruct proteins from heavily masked inputs as well as partially observed ones, making repeated denoising a way to generate whole sequences. DPLM

An existing masked model can therefore be used in two ways: teach it better edits to known proteins, or adapt a generator that already knows how to fill much larger gaps.

RLXF: improving mutations proposed by a masked model

Reinforcement learning from experimental feedback, or RLXF, takes the editing approach. Its 2024 workshop study adapted ESM-2, a masked pLM, to propose variants of the fluorescent protein CreiLOV. Experimental measurements trained a separate fluorescence predictor. The researchers first fine-tuned ESM-2 on successful variants, then used the predictor to score its proposals during RL. RLXF workshop paper

The model’s decisions are replacements at masked positions. A policy is the model’s rule for assigning probabilities to those choices. RLXF uses a proximal policy optimization (PPO) objective to update those probabilities, with a reference-model penalty to discourage large changes. A reward for the resulting variant can therefore teach a masked model which replacements to favor, adapting a mutation generator without converting ESM-2 into a left-to-right model. RLXF methods, PPO

The two studies differ in how far they got. The workshop study increased the fraction of functional designs but did not exceed the parent protein’s brightness. The September 2026 journal study reported improved generation across five protein families and fluorescent variants exceeding the parent. Increasing the chance of retaining function is useful, and improving the starting protein is a further achievement. 2024 results, 2026 study

ESM3: learning preferences over generated proteins

ESM3, released in 2024, supplies the other masked-model example. It generates by progressively filling missing sequence, structure, and function tokens. Its post-training experiment collected preferred and dispreferred generations using predicted structural quality and agreement with a supplied structure prompt. ESM3

Choosing the probability to train on is difficult here. A completed protein may be reached through many different orders of filling masks. ESM3 therefore uses a masking-based approximation to sequence likelihood in its preference objective. It learns from comparisons through an objective adapted to masked prediction, alongside a term that reinforces preferred examples.

The resulting models improved on the computational structure-prompt tasks. The paper’s fluorescent-protein demonstration tested the base model separately; it does not establish that preference training caused that experimental success. ESM3 methods and experiments

RLXF and ESM3 start from masked models and post-train them differently. One optimizes masked mutation proposals with predicted functional rewards; the other learns preferences over generations using a likelihood approximation. A shared masked-model ancestry does not prescribe a single post-training recipe.

Why a masked-model score needs care

A masked model can score a completed sequence by hiding each residue in turn and predicting it from all the others. Combining those predictions gives a pseudo-likelihood: a measure of how well each residue fits its surroundings.

A generator may have seen much less context when it chose those residues. Its actual generation decisions therefore need not have those same probabilities. To learn which sampled decisions deserve higher rewards, one approach is to record their probabilities along the actual generation path. For preference training, another is to derive a suitable approximation, as ESM3 does. A useful sequence score cannot automatically be substituted for either quantity. ESM3 likelihood approximation

Diffusion: training through denoising

In diffusion models, the training update can follow the denoising process itself.

DRAKES, first posted in October 2024 and published at ICLR 2025, explored this for discrete diffusion, including protein inverse folding. Its protein generator starts with a target backbone and denoises a sequence. A stability predictor supplies the objective. During training, DRAKES approximates discrete amino-acid choices with continuous values so it can propagate gradients, meaning information about how changes affect the score, back through the generation steps. A penalty discourages excessive departure from the pretrained generator. DRAKES

The method improved predicted stability while retaining structural agreement in computational evaluations. The method requires access to the reward model’s gradients. PPO-style training can work with a returned score alone; DRAKES needs a differentiable route through the scorer and its approximated sampling process. DRAKES, PPO

That requirement connects model architecture to feedback availability. A laboratory measurement supplies a number. A learned predictor can supply both a number and derivatives. Once measurements have trained that predictor, they can support update rules that direct experimental feedback could not. The predictor’s accuracy now matters for the direction of the update as well as the score it assigns.

Diffusion post-training is therefore its own methodological branch. The generator already performs a sequence of denoising decisions, and those decisions provide places to apply learning. What distinguishes the methods is which parts of that process they can differentiate or assign probabilities to.

Online RL: learning from fresh generations

Preference datasets let a model learn from known comparisons. In online training the changing model generates fresh candidates, those candidates receive scores, and the next update uses that feedback, so the procedure can discover candidates that were absent from the initial preference dataset.

This idea has older roots in protein design. Conditioning by adaptive sampling (CbAS, 2019) adapted generators toward predicted properties while accounting for the predictor’s reliability. Model-based biological sequence RL followed a related approach in 2020. Modern pretrained pLMs give these optimization procedures more capable starting generators. CbAS, Biological sequence RL

An online loop can repeatedly consult the same fixed predictor. Running new laboratory experiments, then updating that predictor, adds a second loop. Keeping those two loops distinct helps explain both the speed of current methods and their limits.

Model updates within an experimental round

The inner cycle reuses one predictor. The outer cycle collects measurements and refits it for further model updates. Experiments can also refresh preference comparisons, as in ProtRL.

ProteinZero: structural agreement, stability, and diversity

ProteinZero, first posted in June 2025, applies online optimization to inverse folding. Its August 2026 revision combines scores for predicted agreement with a target backbone, an estimate of stability, and a term encouraging diverse designs. The stability estimate comes from protein-model likelihoods; it is a computational proxy for the desired property. ProteinZero

One update method it studies is group relative policy optimization (GRPO). Generate several sequences for the same backbone, compare their rewards, and use their relative performance to update the generator. This comparison supplies a baseline without the separate value model commonly used in PPO to estimate expected reward. Protein sequences can supply the outputs being compared; the procedure does not require a written reasoning trace. GRPO

The diversity term keeps the objective from reducing to single-score maximization. A model that repeatedly proposes close relatives of one successful sequence gives an engineer fewer alternatives. ProteinZero penalizes excessive similarity using the model’s numerical representations of proteins. Its computational evaluations examine structural agreement and stability with additional predictors, providing evidence beyond improvement on the training reward. Its generated designs have not yet been tested in the laboratory. ProteinZero

ProtRL: choosing the reward and updating it with experiments

ProtRL works with autoregressive pLMs and external scoring procedures. First posted as DPO_pLM in December 2024, its August 2026 revision covers several optimization methods and experimental follow-up.

In one experiment, rewarding structural similarity alone led the generator to increase sequence length. Adding a length-related term corrected the behavior. The optimizer responded to feedback, but the original score left an unwanted way to improve it.

The same paper also describes a binding campaign that incorporated new experimental measurements into successive rounds of ranked preference optimization. The generator could explore sequence-length changes as well as substitutions. That flexibility is an architectural advantage when length variation helps the task, and something the reward must control when it does not. ProtRL

This also shows why “DPO versus RL” is an incomplete way to organize the field. Preference optimization can run inside a repeated design-and-test process. The update rule, the source of feedback, and whether new candidates are collected are separate decisions.

Model scale: learning more from the same feedback

So far, the choices have concerned how a generator works and how it receives feedback. ProGen3, introduced in April 2025, asks how much the starting model’s size changes what post-training accomplishes.

ProGen3 uses an offline objective combining experimental preference comparisons with imitation of high-scoring examples. Larger aligned models generally performed better on the reported property-prediction tasks. A larger model may therefore learn more useful distinctions from a limited experimental dataset. ProGen3

The generation results are more specific. For the 46-billion-parameter model, the authors tested designs from 32 prompts with a split-GFP fluorescence assay that measures expression. Alignment improved expression for 8 prompts, reduced it for 3, and produced no significant change for 21. Expression can relate to stability, but this assay does not directly measure the energy difference between folded and unfolded proteins. ProGen3

Expression after alignment: 32 prompts

8 improved
3 declined
21 no significant change

One dot = one prompt · ProGen3, 46 billion parameters

The split-GFP assay compared expression of designs from the aligned and pretrained models. Categories follow the paper’s statistical test; “no significant change” does not establish equal expression. Source: ProGen3.

The result gives a concrete reason to study scale and post-training together, and it points to the next experiment: establish where better property prediction translates into a higher experimental success rate. Connecting a scaling result to engineering value requires measuring both.

What these branches add up to

The studies support several ways to improve a protein generator, each suited to different starting conditions.

With measured variants, preference learning can use both successful and unsuccessful examples. ProteinDPO makes this approach concrete, and ProGen3 asks how its benefits change with scale.

With an existing protein to improve, masked models can learn better edits. With a broader generative masked model, preferences can instead act on whole candidates, provided the objective handles their likelihoods appropriately. RLXF and ESM3 illustrate that distinction.

With a differentiable predictor, diffusion models can learn through approximations to their denoising steps. With a scorer that returns only a number, PPO-style methods still offer a route. With new experiments, either a reward model or a preference dataset can be refreshed between rounds.

These choices can also be combined. A generator might first adapt to a family, then learn from measured comparisons, then collect new candidates for further optimization. Each stage adds something observable: a useful property, better adherence to the design constraints, or more successful candidates per experimental batch.

Architecture determines how a model can propose and revise proteins, and the feedback determines which of those proposals it learns to favor. The most informative results show how a particular combination improves the proteins that researchers obtain.