Denoising Diffusion Probabilistic Models
Jonathan Ho, Ajay Jain, Pieter Abbeel
Diffusion models generate high-quality images by learning to reverse a noise-adding process via denoising score matching.
How can we train diffusion probabilistic models to achieve high-quality image synthesis that rivals GANs and likelihood-based models?
Generative models like GANs and VAEs often struggle with training stability or sample quality, making it difficult to synthesize high-fidelity images reliably. The authors propose a diffusion model that learns to reverse a fixed Markov chain that gradually destroys data with Gaussian noise. By parameterizing the model to predict the added noise rather than the mean, the training objective simplifies to a weighted denoising task that mirrors Langevin dynamics. This approach achieves state-of-the-art image quality, reaching an FID score of 3.17 on the CIFAR-10 dataset.
Paper Primer
The core mechanism hinges on a simplified training objective: instead of optimizing the full variational bound, the model learns to predict the specific noise $\epsilon$ added to an image at a given timestep $t$. This is like a sculptor learning to remove exactly the right amount of stone to reveal the original form, where the "stone" is the Gaussian noise added during the forward diffusion process.
Diffusion models achieve state-of-the-art sample quality on unconditional image generation.
Comparison against established generative models on the CIFAR-10 dataset. An FID score of 3.17, outperforming previous diffusion-based and many GAN-based approaches.
The model also functions as a progressive lossy compressor. Because the sampling process reconstructs images from noise in discrete steps, it naturally allocates bits to large-scale features first and fine details later, mirroring autoregressive decoding.
Why use this approach over standard generative models like GANs?
Diffusion models offer a more stable training objective and avoid the adversarial dynamics that often make GANs difficult to converge, while still producing high-quality, diverse samples.
What is the primary limitation of this method?
While sample quality is high, the models do not achieve competitive log-likelihoods compared to other likelihood-based models, as much of the codelength is spent describing imperceptible image details.
Researchers can now treat diffusion as a robust, stable alternative to GANs for high-fidelity synthesis, with the added benefit of a built-in progressive compression framework.
Introduction
Diffusion models reverse a gradual noise process to generate high‑quality images.
Diffusion probabilistic models are a class of latent variable models motivated by nonequilibrium thermodynamics. We refer to the specific instantiation studied here as DDPM (Denoising Diffusion Probabilistic Models). These models are parameterized Markov chains trained with variational inference to match the data distribution after a finite number of steps.
The forward diffusion process injects small Gaussian noise at each step, gradually destroying the original signal. The reverse process is learned to invert this chain, effectively denoising step‑by‑step. Until now, no work had shown that such models could generate samples of comparable quality to GANs or other state‑of‑the‑art generators.
Training adopts a weighted variational bound that reveals an equivalence with denoising score matching and annealed Langevin dynamics across multiple noise levels. This parameterization also yields a progressive lossy‑decompression scheme, which can be seen as a generalization of autoregressive decoding. These insights constitute the paper’s primary methodological contributions.
Empirically, the model reaches an Inception score of 9.46 and an FID of 3.17 on the unconditional CIFAR‑10 benchmark. On 256×256 LSUN it attains sample quality comparable to ProgressiveGAN. Although log‑likelihoods are not competitive with the best likelihood‑based models, they surpass estimates reported for energy‑based approaches, and most of the codelength is spent on imperceptible details—a phenomenon we interpret as lossy compression.
A diffusion probabilistic model defines a stochastic forward process that slowly corrupts data with Gaussian noise and a learned reverse process that denoises step‑by‑step to generate new samples.
Diffusion models provide a viable alternative to GANs for high‑quality image synthesis.
Diffusion Model Foundations
Defines the forward and reverse diffusion processes that underlie the model.
Training a diffusion model naively requires evaluating the full noisy trajectory, which is prohibitively expensive. The remedy is to treat the noise injection as a tractable Markov chain (the forward process) and to learn its inverse (the reverse process). This section formalises both chains and the key distributions that make them computable.
The forward process gradually corrupts a clean datum by adding a small amount of Gaussian noise at each step, so that after many steps the sample is indistinguishable from pure Gaussian noise.
Step 1: $\alpha_1=0.9$, sample noise $\epsilon_1\sim\mathcal{N}(0,1)$ (e.g. $\epsilon_1=0.5$). $x_1=\sqrt{0.9}\cdot1.0+\sqrt{0.1}\cdot0.5\approx0.95$.
Step 2: $\alpha_2=0.8$, sample $\epsilon_2=-0.3$. $x_2=\sqrt{0.8}\cdot0.95+\sqrt{0.2}\cdot(-0.3)\approx0.78$.
Step 3: $\alpha_3=0.7$, sample $\epsilon_3=1.2$. $x_3=\sqrt{0.7}\cdot0.78+\sqrt{0.3}\cdot1.2\approx1.02$.
Result: after three steps the signal has been perturbed from $1.0$ to roughly $1.0$, illustrating that each step adds only a modest amount of noise.
The forward chain preserves most of the original signal when $\beta_t$ are small, which is why the reverse model can reliably recover $x_0$ from a noisy $x_T$.
How does this forward diffusion differ from a generic Markov noise process?
In a generic Markov process the transition distribution can be arbitrary, but here every transition is a *scaled* Gaussian with a *prescribed* variance schedule. The scaling $\sqrt{1-\beta_t}$ guarantees that the signal component decays smoothly, while the fixed Gaussian form makes the whole chain analytically tractable.
The reverse process learns to undo the forward corruption: starting from pure noise it predicts a slightly less noisy predecessor at each step until a clean sample emerges.
Is the reverse process simply a denoising auto‑encoder?
No. A denoising auto‑encoder maps a single noisy input to a clean output, whereas the reverse diffusion chain produces a *sequence* of intermediate distributions, each conditioned on the previous noisy state and the timestep. This temporal conditioning is essential for tractable likelihood estimation.
Rewriting the variational bound yields a KL‑decomposed objective (Eq. 5) that compares the learned reverse kernels to the analytically known forward posteriors. Since every KL term is between two Gaussians, it admits a closed‑form expression, eliminating high‑variance Monte‑Carlo estimates.
**Figure 2.** The directed graphical model considered in this work.
Theoretical Connections
Explains the noise‑prediction trick that turns diffusion training into denoising score matching.
The design space of diffusion models is huge: one must pick a variance schedule $\beta$ₜ, a reverse‑process covariance $\Sigma$$\theta$, and a mean parameterization. Without a principled guide, these choices become a tedious trial‑and‑error exercise.
Instead of forcing the model to reconstruct the clean data x₀, we ask it to predict the random noise $\epsilon$ that was added at step t; this turns the training loss into a simple denoising objective that matches the score of the data distribution.
Compute $x_1 = \sqrt{0.9}\cdot1.0 + \sqrt{0.1}\cdot0.5 \approx 0.95 + 0.158 = 1.108$.
Compute $x_2 = \sqrt{0.72}\cdot1.0 + \sqrt{0.28}\cdot0.5 \approx 0.849 + 0.265 = 1.114$.
The network predicts $\epsilon_\theta(x_2,2)\approx0.48$ (close to the true $0.5$).
Apply the parameterization: $\mu_\theta(x_2,2)=\frac{1}{\sqrt{0.72}}\bigl(1.114 - \sqrt{1-0.72}\cdot0.48\bigr) \approx 1.314 - 0.256 = 1.058$.
Using Eq. (10), the posterior mean $\tilde\mu_2$ equals $1.058$, confirming that the noise‑prediction yields the correct reverse mean.
Predicting the noise lets the model recover the exact reverse‑process mean with a single $L_2$ regression, avoiding the more complex posterior‑mean expression.
Why does predicting the noise $ε$ simplify training compared to predicting the clean data $x₀$?
Because $ε$ follows a standard normal distribution regardless of the data, the regression target is isotropic and has constant variance. Predicting $x₀$ would require the model to undo the scaling by $\sqrt{\barα_t}$ and the additive noise, which varies with $t$ and makes the loss landscape harder to optimize. The noise‑prediction loss is therefore a uniform $L_2$ objective that directly matches the score of the data distribution.
Predicting $ε$ is easier than reconstructing $x₀$ because the model only needs to infer the disturbance that was added, not the original signal itself — like estimating how much rain fell on a wet street rather than trying to reconstruct the exact dry pavement.
Sample a clean datum $x_0 \sim q(x_0)$.
Uniformly pick a timestep $t \in \{1,\dots,T\}$.
Draw noise $\epsilon \sim \mathcal N(0,I)$ and form the noisy sample $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$.
Feed $(x_t,t)$ to the network and obtain $\epsilon_\theta(x_t,t)$.
Take a gradient step on the loss $\frac{1}{2\sigma_t^2}\|\,\epsilon - \epsilon_\theta(x_t,t)\|^2$.
Draw $x_T \sim \mathcal N(0,I)$.
For $t = T, T-1, \dots, 1$:
If $t>1$, sample $z \sim \mathcal N(0,I)$; otherwise set $z=0$.
Compute the reverse mean using the predicted noise: $x_{t-1}= \frac{1}{\sqrt{\alpha_t}}\bigl(x_t - \sqrt{1-\bar\alpha_t}\,\epsilon_\theta(x_t,t)\bigr) + \sigma_t z$.
Return $x_0$ as the generated sample.
The Simplified Training Objective
We reshape the diffusion loss to predict noise directly, enabling efficient training and sampling.
Diffusion models assume continuous latents, yet raw images are integer pixels 0–255. Treating them as discrete forces the reverse process to start from a standard‑normal prior while still producing exact integer outputs.
Training the full variational bound is costly and requires careful weighting of each timestep. By discarding those weights we obtain a single‑term loss that predicts the added noise directly, which is both simpler to implement and empirically stronger.
Instead of weighting every diffusion step, we ask the network to predict the exact noise added at a randomly chosen timestep and measure the squared error.
Compute the noisy image $\mathbf{x}_2 = \sqrt{0.64}\,\mathbf{x}_0 + \sqrt{1-0.64}\,\boldsymbol{\epsilon}$ (the exact $\mathbf{x}_0$ is irrelevant for the loss).
Form the error vector $\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta = (0.05,0.05,-0.05,0.05)$.
Square each component: $(0.0025,0.0025,0.0025,0.0025)$.
Sum them to obtain $\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta\|_2^{2}=0.01$.
Average over many such samples; the expectation is approximated by this single value in the toy run.
The loss directly measures how well the network recovers the exact noise, regardless of the underlying image content.
How does this simplified loss differ from the original variational bound that weights each timestep?
The original bound multiplies the reconstruction error at timestep $t$ by a coefficient that grows as $t$ approaches $T$, emphasizing early (low‑noise) steps. Our loss drops those coefficients entirely, treating every $t$ equally; consequently the model spends more capacity on denoising heavily corrupted images, which are the bottleneck for high‑quality generation.
**Algorithm 1 Training** 1: **repeat** 2: $\mathbf{x}_0 \sim q(\mathbf{x}_0)$ 3: $t \sim \text{Uniform}(\{1, \dots, T\})$ 4: $\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ 5: Take gradient descent step on $\nabla_\theta \|\epsilon - \epsilon_\theta(\sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\epsilon, t)\|^2$ 6: **until converged**
**Table.** Comparison of models on LSUN datasets.
Empirical Results
We report sample quality, ablations, and progressive coding results for our diffusion model.
Diffusion models learn to reverse a gradual noise‑addition process; by training to predict the added noise directly they generate high‑quality images with far fewer network evaluations ($T=1000$).
Our unconditional DDPM attains an FID of 3.17 on CIFAR‑10, surpassing most prior unconditional generative models.
Table 1 reports 3.17 FID (training‑set evaluation), better than the next‑best unconditional entry.
**Figure 1.** Generated samples on CelebA-HQ 256 × 256 (left) and unconditional CIFAR10 (right)
**Table 1.** CIFAR10 results. NLL measured in bits/dim.
**Figure 3.** LSUN Church samples. FID=7.89
**Figure 4.** LSUN Bedroom samples. FID=4.90
**Figure 5.** Unconditional CIFAR10 test set rate-distortion vs. time. Distortion is measured in root mean squared error on a $[0, 255]$ scale. See Table 4 for details.
**Figure 6.** Unconditional CIFAR10 progressive generation ($\hat{x}_0$ over time, from left to right). Extended samples and sample quality metrics over time in the appendix (Figs. 10 and 14).
**Figure 7.** When conditioned on the same latent, CelebA-HQ 256 × 256 samples share high-level attributes. Bottom-right quadrants are $x_t$, and other quadrants are samples from $p_\theta(x_0|x_t)$.
**Figure 8.** Interpolations of CelebA-HQ 256x256 images with 500 timesteps of diffusion.
**Figure 9.** Coarse-to-fine interpolations that vary the number of diffusion steps prior to latent mixing.
DDPM achieves competitive FID scores on CIFAR10 and LSUN.
Experimental Details and Architecture
Experimental details and ablation insights for the diffusion model.
We now detail the concrete design choices that differentiate our diffusion model from the baseline PixelCNN++ U‑Net and report the ablations that justify each choice.
The model stacks convolutional encoder‑decoder blocks with skip connections, allowing high‑resolution details to be reconstructed from coarse latent features.
How does this U‑Net differ from the standard PixelCNN++ decoder?
PixelCNN++ uses a RefineNet decoder with dilated convolutions, while our design replaces it with a Wide ResNet‑style encoder‑decoder pair and adds explicit self‑attention at $16\times16$, which gives a richer global context without sacrificing locality.
Normalization is performed over groups of channels rather than across the batch dimension, making the layer independent of batch size.
Why not keep the original weight normalization?
Weight normalization introduces per‑weight scaling parameters that are expensive to compute on TPUs and can lead to slower convergence; group normalization eliminates those extra parameters and yields comparable or better sample quality.
At the intermediate spatial resolution the model inserts a multi‑head attention block, letting each patch attend to every other patch.
Why place attention only at $16\times16$ and not at higher resolutions?
At $16\times16$ the spatial map is small enough for quadratic attention cost to be affordable, yet large enough to capture meaningful global structure; adding attention at higher resolutions would explode compute without proportional quality gains.
Each residual block receives a deterministic vector that encodes the current diffusion step $t$, allowing the network to adapt its behavior as noise increases.
Why not inject the step embedding only at the network’s input?
Embedding only at the input forces the early layers to learn a universal denoising function, which cannot adapt to the rapidly changing noise scale; distributing the embedding throughout the network lets each stage specialize for its local noise level.
The variance added at each diffusion step is controlled by a linearly increasing schedule, keeping the overall noise growth smooth.
Why not use a constant $\beta$ across all steps?
A constant schedule injects the same amount of noise each step, which either under‑noises early steps or over‑noises later steps, breaking the Markov chain’s smoothness and degrading sample quality.
Randomly zeroing activations with probability $0.1$ regularizes the model, preventing over‑fitting on the small $32\times32$ dataset.
Why is dropout omitted for the larger $256^2$ datasets?
Those datasets provide enough diversity and the larger models already have sufficient capacity; adding dropout offered no measurable benefit and slightly slowed convergence.
During training each image is randomly mirrored left‑right, augmenting the data distribution without changing semantics.
Why keep flips for LSUN Bedroom?
Bedroom layouts have a canonical left‑right orientation (e.g., windows vs. doors); flipping would create physically implausible scenes and hurt realism.
During training we maintain a shadow copy of the model parameters that updates slowly (decay $0.9999$), and we use this smoothed model for sampling.
Why not use a smaller decay (e.g., $0.99$) for faster adaptation?
A smaller decay reacts too quickly to recent noisy updates, re‑introducing variance that the EMA is meant to suppress, and the ablation showed higher FID and less stable samples.
**Figure 10:** Unconditional CIFAR10 progressive sampling quality over time
**Figure 11.** CelebA-HQ 256 × 256 generated samples
**Figure.** (a) Pixel space nearest neighbors (b) Inception feature space nearest neighbors
**Figure 13.** Unconditional CIFAR10 generated samples
**Figure 14.** Unconditional CIFAR10 progressive generation
**Figure 15.** Unconditional CIFAR10 nearest neighbors. Generated samples are in the leftmost column and training set nearest neighbors are in the remaining columns.
**Figure 16.** LSUN Church generated samples. FID=7.89
**Figure 17.** LSUN Bedroom generated samples, large model. FID=4.90
**Figure 18.** LSUN Bedroom generated samples, small model. FID=6.36
**Figure 19.** LSUN Cat generated samples. FID=19.75
Conclusion and Discussion
We discuss broader impacts, extra experimental tables, and limitations of our compression proof‑of‑concept.
We have demonstrated high‑quality image samples generated by diffusion models and highlighted their conceptual links to variational inference for Markov chains, denoising score matching, annealed Langevin dynamics (and, by extension, energy‑based models), autoregressive models, and progressive lossy compression.
While diffusion models advance generative capabilities, they inherit the broader societal risks of deep generative models: easy creation of realistic fake media, reinforcement of dataset biases, and the potential erosion of detection methods as synthesis improves.
Our progressive compression experiment (Section 4.3) is a proof‑of‑concept; it relies on Algorithms 3 and 4, which in turn require a minimal random coding step that is not tractable for high‑dimensional data.
Mathematical Derivations
Derives the reduced‑variance variational bound used for training diffusion models.