Quick digests
Short explanations of research topics, with links to the papers. Each one takes a topic I wanted to understand, such as mid-training or mode collapse, and works through the main ideas in plain language, citing the papers as it goes. They range a bit in terms of length but they tend to be around 20 minutes of reading time.
Digests are mostly written by LLMs. They've honestly been incredibly helpful at explaining topics to me and it's been something I lean on in order to get introductions to fascinating fields in days rather than months. Of course, I do revise them for readability purposes, but this is the one place in the site that may read like slop. If so, please tell me (max.zhuang@berkeley.edu).
Evaluating mid-training
What it means for one mid-training recipe to be better than another, why a mid-trained checkpoint's own scores mislead in specific ways, and how to compare recipes so that the comparison holds.
Combining unreliable evidence
What rule a trained network applies when it combines several inputs that can each be wrong, why it does not double count dependence that stayed fixed throughout its training, what it takes to learn to discount a copy, and what language models, majority votes and label aggregators do with evidence that is not independent.
Training a latent variable instead of searching for it
When a trained network can stand in for a per-input search over an unobserved quantity, such as a move sequence, a chain of thought, an autoencoder's code or a 3D scene, what it loses, where search still has to come back, and how game players and reasoning models combine the two.
Transfer from tasks with exact answers
When training a language model on problems with checkable answers, such as math, code and puzzles, makes it better or worse at other tasks; what the exact answer contributes; and why the task being measured decides more than the task being trained.
Generalizing beyond the training distribution
What has to be true for a model to be right on inputs its training data never covered, where the information that makes it right can come from, and what controlled experiments say about each source, from task diversity and built-in structure to test-time training and the reason-or-recite debate.
Usable information in a fixed dataset
Given a fixed corpus and every other degree of freedom, what sets the most a model can learn from it, and which methods close the gap, from priors and averaging over runs to keeping what was seen once, rewriting, and computation at answer time.
How cofolding models are trained
How AlphaFold 3 and the open models that reproduce it are made: what goes into the training set, including the predicted structures mixed in with experimental ones, the denoising objective, how the confidence head is trained, the schedule and its cost, what later models changed, and what the evaluations say the training taught.
So you want to turn your model into a reasoning model
How to train a pretrained language model to write a long chain of thought before it answers: when to copy a stronger model and when to use reinforcement learning, what the starting model must already do, how the published recipes set up rewards, data and length, and how to tell whether it worked.
Mechanistic interpretability of post-training
What comparing a model with its own base model has shown about how fine-tuning, preference and safety training, and reinforcement learning change a model inside, where the methods have worked on production models, where they mislead, and where the field is going, including for biological sequence models.
Mid-training
What the stage between pretraining and post-training contains, why the last few percent of a training run matter so much, and how the choices made there shape the post-training that follows.
Mode collapse in post-training
Why post-training can throw away useful variety, how to tell collapse apart from a model that got better, and what the evidence means for language models and for bioAI.
Protein language models and the wider protein AI field
A detailed guide to the major protein model families, their architectures, and what each contributes, from ESM and AlphaFold to Chai, Boltz, and generative design.
Structure-based drug design and its clinical record
What knowing a protein's structure lets a drug developer decide, what it leaves unresolved, and how much of the recent computational wave has been tested in patients.
Generative models and how they're built
How latent diffusion, DiT, Stable Diffusion 3, MaskGIT, and other models combine representations, learning signals, and repeated computation.
Post-training protein language models
How experimental preferences, masked-model editing, diffusion, and online reinforcement learning are changing protein generation.
Looped transformers and AlphaFold recycling
What a model gains by running the same layers again, and what AlphaFold feeds back between predictions.