Generative models and how they're built

Sep 14, 2026 · 15 min read

Digests are written primarily by LLMs with only readability edits by me. If this one reads like slop, please shoot me an email at max.zhuang@berkeley.edu.

An image generator can contain several kinds of model, each doing a different job. DiT, the Diffusion Transformer introduced in 2022, uses an autoencoder to represent images as smaller arrays, a diffusion process to generate new arrays, and a transformer to make the predictions that process needs. DiT

The combination is easier to understand by following an image through it. Compression, prediction, and generation each present a different problem. Researchers can improve one part while keeping the others, and sometimes discover that a familiar component works well in a very different setting.

Latent diffusion: generate a smaller representation

Repeatedly processing a large image is expensive. Latent diffusion reduces that cost by doing the repeated work in a smaller representation. An encoder turns an image into an array of learned features; a decoder turns the array back into pixels. Together, these form an autoencoder. Its training asks the reconstruction to preserve useful image information. Rombach et al.

The smaller array is called a latent representation. Its values are learned features, so it needn’t look like a miniature photograph. DiT’s pretrained autoencoder, for example, represents a 256-by-256 color image as a 32-by-32 grid with four feature values at each position. DiT

Now train a separate generator on those arrays. To produce a new image, generate an array and pass it through the decoder. The encoder is needed to prepare training examples; generating a new image doesn’t require an input photograph.

Image and latent dimensions

64× fewer spatial positions for repeated generation. Areas are drawn to scale; each image position has 3 color channels and each latent position has 4 learned features. The animation follows encoding and decoding; the latent pattern illustrates learned features, not a smaller photograph. Source: DiT.

This division lets the autoencoder concentrate on representing images while the generator learns which representations to produce. It also introduces a tradeoff: more compression makes generation cheaper, but the decoder has less information from which to reconstruct the image. The latent-diffusion paper studies that tradeoff explicitly. Latent diffusion

What the VAE contributes

DiT uses a variational autoencoder, or VAE, for its image representation. In a basic VAE, the encoder predicts a distribution over latent values for each input. A sampled value goes through the decoder, which predicts a distribution over reconstructed examples. Training balances reconstruction against a penalty for departing from a chosen prior, a distribution we can sample without having an image. A Gaussian, or bell-shaped, distribution is a common choice. Kingma and Welling

A VAE can therefore generate on its own: sample a latent value from the prior, then sample an example from the decoder’s distribution. In latent diffusion, a more expressive generator learns the distribution of encoded images instead. The autoencoder supplies a useful space in which to do that work. The original latent-diffusion paper explored both VAE-style regularization and an alternative that uses a learned collection of discrete codes. VAE tutorial, Latent diffusion

So “uses a VAE” needn’t mean the VAE generates the final image by itself. It may supply one component of a larger system.

What makes an autoencoder “variational”?

The encoder approximates the distribution of latent values that could explain an observed example. This approximation defines an evidence lower bound, or ELBO, on the example’s log likelihood. Joint training of the encoder and decoder maximizes that bound. Its usual form produces the reconstruction and prior-matching terms described above.

For independent Gaussian latent coordinates, the encoder predicts a mean and standard deviation for each coordinate. A sample is the predicted mean plus the predicted standard deviation times a draw of standard Gaussian noise. Separating the randomness this way lets derivatives pass through the learned mean and scale: the reparameterization trick.

Image-generation autoencoders can also use perceptual and adversarial losses. A VAE component in such a system needn’t be trained with only the textbook objective. VAE tutorial, Latent-diffusion autoencoder

Diffusion and DiT: learn denoising, choose a network

The autoencoder has made the image smaller. We still need to learn how to generate its representation. Diffusion supplies a prediction task with a training target we can construct ourselves.

Take a real example, either pixels or an encoded image, and mix it with Gaussian noise. Choose how much noise to add. Show the noisy result and its noise level to a network, and ask it to predict the noise. Because we sampled the noise, we know the answer. The loss, the numerical error minimized during training, compares the prediction with that answer. Ho et al.

Training one denoising prediction doesn’t require generating an entire image. We can directly construct a noisy example at a randomly selected level and train on that one prediction.

Generation assembles those predictions into a longer process:

Training:
real example + known noise → network → compare with known noise

Generation:
fresh noise → update → update → ... → generated example

At each generation step, a sampler uses the network’s prediction to update the current array. The sampler specifies the update size and whether to add fresh randomness. The same network is called again at the next noise level. The standard training procedure supplies examples of these local prediction tasks without backpropagating through the whole generation chain. Ho et al.

DiT changes the computation inside each prediction

Early image-diffusion systems commonly used a U-Net. It processes features at successively coarser resolutions, then restores finer resolution while incorporating features from earlier stages. This combines broad context with local detail. The architecture originated in biomedical image segmentation; its organization also proved useful for denoising. U-Net, Diffusion models

DiT replaces that network with a transformer. It divides the noisy latent array into patches and treats each patch as a token, one element in the network’s input sequence. Attention lets these tokens combine information from other positions. The output supplies diffusion predictions for the sampler. DiT, Transformer

This is a transformer working on an entire noisy representation. It doesn’t need to generate the image one token at a time. The architecture specifies the computations used to make a prediction; it doesn’t, by itself, specify what that prediction should be.

DiT’s experiments also showed why counting parameters can miss an important change. Smaller patches give the same network more tokens to process, increasing computation without increasing its parameter count. Both this change and larger networks improved the paper’s automated comparison of generated and real image collections. DiT experiments

Where mixtures of experts fit

A sparse mixture of experts, or MoE, changes which learned parameters participate in a prediction. A router chooses a subset of subnetworks for each token. Switch Transformers, for example, send each token to one expert in an expert layer.

The experts are learned subnetworks; they needn’t correspond to subjects such as chemistry or grammar. Routing can increase total model capacity while limiting the computation used for each token. It leaves the generation rule to other choices. Switch Transformers

Stable Diffusion 3: learn how samples should move

Better generation can come from a better prediction network or from changing the task the network learns.

The research behind Stable Diffusion 3, published in 2024, combines a transformer with rectified flow, part of the closely related work on flow matching. These methods train a network to predict a velocity: how each coordinate of the current sample should change as generation advances. Stable Diffusion 3 research

For a simple flow-matching example, pair a noise array with a real image representation. Define a straight interpolation between them: noise at time zero, the real example at time one, and an equal mixture halfway through. Here, “time” indexes the generation process.

At a randomly selected time, give the network the mixture and ask for the velocity along that interpolation. For this particular path, the target is the real example minus the starting noise. As in diffusion training, we can construct the input and target directly. We don’t have to generate an image to obtain a training example. Flow matching

To generate, start with fresh noise and follow the predicted velocity. A numerical solver approximates the continuous motion with finite steps. This motion is described by an ordinary differential equation, or ODE.

A straight interpolation sounds as though it should permit one perfect generation step, but straight training interpolations do not guarantee straight generated trajectories. The network sees the current mixture and time, without being handed its destination. Squared-error training averages the velocities compatible with that input; following the learned field can curve. Rectified flow’s reflow procedure creates new endpoint pairs from a learned flow and trains again to straighten its trajectories. Rectified flow

Stable Diffusion 3 tested these choices empirically. Its authors compared 61 training formulations and found that which timesteps received more training weight mattered. Their selected rectified-flow formulation emphasized intermediate timesteps and performed well with fewer sampling steps. Choosing a “flow” formulation didn’t by itself determine the result.

They also changed the network: text and image tokens use separate weights but exchange information through attention. The paper reports improvements in prompt following and typography. This change is to the architecture, separate from the change in training formulation. Esser et al.

The case illustrates how an advance can combine several interventions: a representation that preserves image detail, a target that is easier to learn at useful noise levels, and a network that handles the prompt better.

How flow matching overlaps with diffusion

Continuous Gaussian diffusion also has a deterministic sampling description: the probability-flow ODE. With matching initial distributions, exact predictions, and exact integration, it follows the same distribution of samples at each time as the corresponding stochastic process. The individual trajectories can differ.

This construction uses the score: the derivative of log probability density with respect to the sample’s coordinates. It describes the direction and rate of the steepest local increase in log density. It is not a rating of image quality. For Gaussian corruption, score prediction and noise prediction are closely related. Song et al.

Flow matching can learn velocity fields for diffusion-style paths as well as other paths. A deterministic sampler therefore does not tell you whether a model was trained by flow matching. The training target, probability path, and sampler all matter. Flow matching

PixelRNN and MaskGIT: generate an image as a sequence

The image generators above repeatedly update an array of continuous values. An image can also be generated as an ordered sequence of discrete choices.

PixelRNN, introduced in 2016, predicts pixel values in an order across an image. Each prediction depends on values already supplied. This is autoregression, or AR: the same dependency used by a language model predicting the next token. PixelRNN

For an illustrative sentence, pretend each word is a token:

the → the enzyme → the enzyme works → ...

Training rewards the probability assigned to the observed next token. Generation samples a token from the predicted distribution, appends it, and repeats. Multiplying the conditional probabilities gives the sequence probability, including the stopping decision for a variable-length sequence.

The distinction between training and generation matters here too. During training, the correct preceding tokens are available. A transformer can predict at many positions in parallel while a causal attention mask prevents it from seeing later tokens. Supplying the correct preceding tokens is called teacher forcing. Ordinary generation remains sequential because those later tokens haven’t been chosen yet. Transformer

PixelRNN used recurrence across image positions to make its predictions. A transformer can perform the same autoregressive task using attention. The generation dependency survives the architectural change.

MaskGIT changes which tokens get filled first

MaskGIT, published in 2022, also represents an image as tokens, using an image encoder and a learned collection of discrete codes. But it trains a transformer to recover randomly hidden tokens from the visible ones.

At generation time, it begins with masked positions and predicts replacements for them together. It retains confident predictions, masks less confident ones again, and repeats with progressively fewer masks. The authors demonstrate image generation in eight rounds. MaskGIT

This changes the organization of generation. Several parts of an image can be proposed in the same round, and committed tokens provide context for the next round. The method’s advantage depends on more than predicting hidden tokens: the masking schedule and the rule for retaining predictions turn that ability into a generator.

LLaDA applies masked diffusion to language

LLaDA, introduced in 2025, develops a related alternative for text. Its forward corruption process replaces tokens with masks, and training learns to recover tokens across different masking levels. A transformer can attend to the full visible context. Its generation procedure fills masked tokens over repeated rounds. LLaDA

This is discrete diffusion: corruption changes token identities instead of adding continuous Gaussian noise. A masked objective alone leaves choices about the corruption process, loss weighting, and sampler unspecified. LLaDA supplies those choices and studies language generation with them.

The paper also reports a useful behavioral difference: relatively balanced performance when completing familiar Chinese poem lines forward or backward, where the autoregressive comparison models had a much larger gap. That is one benchmark, but it shows a concrete consequence of a different prediction task. LLaDA experiments

The connection extends to protein models. Masked generators can fill amino acids, while autoregressive generators append them. Those generation choices also affect how models learn from feedback, as explored in the protein post-training digest.

GANs and normalizing flows: change the learning signal

The methods so far learn from a target we can construct: the next token, a hidden token, sampled noise, or a velocity. Other approaches make different parts of generation directly trainable.

StyleGAN learns from a discriminator

A generative adversarial network, or GAN, trains a generator alongside a discriminator that learns to distinguish generated examples from real ones. The generator updates its parameters to make examples the discriminator classifies as real. It doesn’t need each random input to reconstruct a particular training image.

After training, the generator can produce an image without the discriminator, commonly in one forward pass. The cost is a training problem involving two changing networks. It can also suffer mode collapse: producing convincing individual examples while covering too little of the variety in the data. GANs

StyleGAN, introduced in 2018, shows how much an architectural change can add within that framework. It maps a random input into an intermediate representation that controls image synthesis at different resolutions. In its face experiments, changing controls at coarse resolutions changes features such as pose, while finer controls affect details and colors. StyleGAN

That intermediate representation is also called latent. The word identifies unobserved values; it doesn’t imply a VAE encoder, a particular prior-matching loss, or a diffusion process. StyleGAN learns its representation through adversarial training.

Real NVP makes likelihood computable

Real NVP takes another route. It transforms samples from a simple distribution through a series of invertible mappings. These mappings are designed so the model can calculate how each transformation stretches or compresses volume. That makes the resulting probability density computable, allowing training to maximize the likelihood assigned to real examples. Real NVP

This is a normalizing flow. Its architecture is constrained to make probability calculations tractable. Here the architecture and training objective are particularly closely connected.

A continuous normalizing flow instead describes its transformation with an ODE. Flow matching is one way to train such a transformation using velocity targets, without solving the full ODE for each ordinary training example. “Normalizing flow” and “flow matching” therefore describe related choices, with different roles. Flow matching

Looped transformers: repeat the internal computation

A diffusion sampler calls the same network several times, and a looped transformer also runs the same block repeatedly. The two loops differ in what they update.

A diffusion sampler updates a noisy image representation. A looped transformer updates its hidden states, the internal numerical representations used to compute a prediction. Reusing a block adds computation without requiring a new set of parameters for each pass. Universal Transformers

Geiping and colleagues’ 2025 recurrent language model performs several internal passes before a readout converts the resulting states into next-token probabilities. It can then sample a token and begin predicting the next one. The internal loop increases work per prediction while the outer generation procedure remains autoregressive. Geiping et al.

Where the training target enters

In standard diffusion training, we construct a corrupted example at a chosen noise level. That level has a defined meaning, and the corruption process supplies the target for a prediction there.

For a looped transformer trained at its final readout, the intermediate states come from the model’s own earlier passes. We specify what the final output should predict without prescribing what its internal state must represent after pass three.

Training setup	State presented to the repeated computation	Where the target comes from
Standard diffusion	A real example corrupted to a sampled noise level	Known noise, or another target derived from the clean example and corruption
Recurrence with a final-readout loss	Hidden states produced by earlier passes	The desired output after the last internal pass

Here, final means the last internal pass. Language-model training can still supply a next-token target at every sequence position. Diffusion training, Recurrent-depth training

The location of the loss also differs from how far its derivatives are calculated. A gradient describes how the loss changes with the model’s parameters; backpropagation computes it. Fully unrolling a loop lets the final error train earlier passes through their effects on later states. Implementations can shorten that calculation to reduce memory use: Geiping and colleagues propagate gradients through at most the last eight recurrent iterations in their main setup. Geiping et al.

So diffusion’s “intermediate supervision” refers to targets at sampled noise levels. It doesn’t mean every hidden layer receives its own target. Both kinds of network use backpropagation, and both can be extended with additional training losses. The comparison concerns the particular training tasks, rather than an inherent limit of either architecture.

The looped-transformer digest develops this distinction further, including AlphaFold recycling.

How these choices fit together

DiT’s combination now has a concrete explanation. The autoencoder determines the representation in which generation happens. The diffusion objective teaches useful predictions from noisy examples. The sampler assembles those predictions into a new sample. The transformer performs the computation inside each prediction.

The other cases change different parts of that system. Flow matching changes the prediction target and the path between noise and data. MaskGIT changes how discrete choices are proposed and retained. StyleGAN obtains feedback from a learned discriminator. Real NVP constrains the mapping so likelihood is tractable. Recurrence adds work inside a prediction.

These distinctions matter when interpreting a result. Better images after changing the representation tell us something different from better images after changing the network. Faster sampling may come from a different solver, a different learned path, or fewer sequential decisions. A model with more parameters may use only some of them for each token; a model with shared parameters may reuse them many times.

To understand a new model, follow one training example and one generated sample. Identify what the model sees, what it predicts, and how its predictions become an output. Once those procedures are clear, the model’s labels describe parts of an understandable system.