Discrete Diffusion Models: A Unified Framework from Tokenization to Generation

Ye Yuan, Weien Li, Rui Song, Zeyu Li, Haochen Liu, Xiangyu Kong, Zixuan Dong, Linfeng Du, Zipeng Sun, Weixu Zhang, Jiaxin Huang, Changjiang Han, Yonghan Yang, Zichen Zhao, Xiuyuan Hu, Haolun Wu, Yankai Chen, Fengran Mo, Jikun Kang, Bowei He, Philip S. Yu, Xue Liu

A unified framework for discrete diffusion models that treats tokenization as the primary design axis.

How can we unify the diverse landscape of discrete diffusion models into a single framework centered on tokenization?

Autoregressive models generate sequences token-by-token, creating a serial bottleneck that prevents global revision and makes long-range planning difficult. This paper introduces a unified framework for discrete diffusion that treats tokenization—the mapping of raw data to discrete symbols—as the central design axis. By decomposing models into a shared four-part structure (corruption, denoiser, objective, sampler), the authors show how disparate approaches like masking, transition-matrix, and score-based models are simply different instantiations of the same state-space construction. The framework provides a cross-domain design reference for text, multimodal, and scientific applications, surfacing shared trade-offs in scaling, efficiency, and structural validity.

Paper Primer

Discrete diffusion models replace the sequential, left-to-right commitment of autoregressive generation with iterative refinement. By starting from a corrupted state and denoising all positions in parallel, these models enable bidirectional context and global planning, which are critical for tasks like infilling and constrained editing.

The core mechanism hinges on the interaction between the corruption operator and the discrete state space. The authors define a four-component decomposition: the corruption operator (how data is masked or substituted), the denoiser parameterization (how it predicts clean tokens), the training objective, and the sampling schedule.

Tokenization is a first-class design axis that dictates the topology of perturbations and the difficulty of the denoising task.

The authors demonstrate that the choice of vocabulary (semantic, quantized, or natural) determines whether the corruption process should be uniform, absorbing, or structured to reflect domain-specific distances. This unified lens allows for the first systematic comparison of discrete diffusion across text, multimodal, and scientific domains (e.g., proteins, molecules).

Why is discrete diffusion not just "continuous diffusion applied to embeddings"?

Applying continuous Gaussian diffusion to embeddings and rounding back to tokens introduces a geometry mismatch, as the continuous score function does not respect the discrete structure of the vocabulary, leading to representation collapse and poor sample quality.

What is the primary advantage of this framework for a practitioner?

It provides a cross-domain checklist for selecting corruption operators and training objectives, allowing designers to move beyond trial-and-error by matching the corruption process to the intrinsic structure of their specific token space.

Discrete diffusion is no longer a collection of isolated techniques; it is a structured design space where the choice of tokenization determines the success of the entire generative pipeline.

Introduction to Discrete Diffusion

We argue that tokenization drives the effectiveness of discrete diffusion models.

Autoregressive (AR) models have become the de‑facto standard for generating discrete sequences by factorizing the joint distribution into left‑to‑right conditionals, which yields a clean maximum‑likelihood objective and stable training. However, AR generation is inherently sequential (O(L) steps) and irrevocable, so each token must be emitted before the next can be considered and cannot be revised later. Diffusion models sidestep these limits by starting from a fully corrupted sequence and iteratively denoising all positions in parallel, and we argue that the way the discrete state space is constructed—i.e., the tokenization—determines how effective this process can be.

Discrete diffusion treats a token sequence as a categorical Markov chain that is first corrupted and then repeatedly denoised until a clean sample emerges.

AR modeling generates a sequence token by token, each step conditioning on all previously generated tokens.

**Figure 1.** Overview of the structure of this survey on Discrete Diffusion Models.

Parallel, globally‑aware denoising lets diffusion generate faster and more flexibly than the inherently sequential AR approach.

Related Work

We map existing diffusion surveys, highlight their common focus, and expose the tokenization‑centric gap this work fills.

Most recent diffusion surveys organize literature by model families or application domains, leaving tokenization and discrete state‑space design as peripheral topics.

**Table 1.** Summary of related survey papers and their relevance to this paper.

**Table.** Comparison of existing surveys on diffusion models.

These surveys (Li et al., 2023a; Čeović et al., 2023; Zou et al., 2023; Yi et al., 2024; Li et al., 2025g; Tseng et al., 2025; Yu et al., 2025) organize work around non‑autoregressive generation, continuous vs. discrete formulations, and the rise of diffusion language models. They excel at cataloguing model families and recent large‑scale language model adaptations.

Works such as Lin et al., 2026; Liu et al., 2025c; Zhang et al., 2025c; Wang et al., 2026d concentrate on system‑level concerns: caching, parallel decoding, deployment bottlenecks, and hardware‑aware optimizations for diffusion models.

This camp covers surveys on graphs (Liu et al., 2023), molecules (Wang et al., 2025g), biomolecules (Yang et al., 2023b; Norton & Bhattacharya, 2025), and recommender systems (Lin et al., 2024a). They illustrate how diffusion adapts to structured scientific and industrial data.

Broad overviews (Yang et al., 2023b; Shou et al., 2026; Zou et al., 2023; etc.) catalog diffusion methods across image, video, and text modalities, emphasizing probabilistic foundations, algorithmic improvements, and application landscapes.

Foundations and Notation

Notation and foundations for discrete diffusion, contrasting AR and iterative refinement.

AR models are serial and immutable, while iterative‑refinement and diffusion approaches enable parallel, revisable generation; the field agrees that discrete diffusion is the only framework that respects token‑level structure, yet it still lacks a principled tokenization design.

Two research strands underpin discrete diffusion: the probabilistic diffusion foundation and the earlier non‑autoregressive, order‑agnostic generation methods that introduced the iterative‑refinement view.

Autoregressive (AR) generation follows this factorization, emitting tokens one‑by‑one; the serial dependency makes generation $O(L)$ and prevents any post‑hoc correction.

Iterative‑refinement methods replace the strict left‑to‑right order with a fixed number of global update steps, allowing all positions to be revised jointly.

Applying the continuous framework to discrete sequences by embedding tokens first leads to a geometry mismatch: the continuous score does not respect the categorical simplex, causing rounding errors.

AR models decompose the joint distribution of a token sequence via the chain rule and generate tokens left‑to‑right. Each conditional is modeled by a causal transformer that sees only previously emitted tokens. Training uses teacher‑forcing cross‑entropy; inference proceeds sequentially, requiring one forward pass per token.

Iterative‑refinement methods break the sequential bottleneck by updating all positions in parallel across a small number of refinement steps $T\ll L$. Mask‑Predict exemplifies this approach: the model repeatedly masks low‑confidence tokens, predicts them in parallel, and re‑masks until convergence.

Continuous diffusion adds isotropic Gaussian noise to a latent vector over many timesteps, then learns a reverse denoiser (often a score network) to recover the data. The forward process is analytically tractable, but applying it to discrete sequences requires embedding tokens first, which creates a mismatch between the continuous score and the categorical nature of the data.

Discrete diffusion defines a forward corruption as a categorical Markov chain with transition matrices $Q_t$ (or absorbing‑state masking). The reverse process directly predicts categorical distributions over the vocabulary, preserving the discrete nature of the data. Modern large‑scale models predominantly use absorbing‑state masking because it yields a simple, tractable prior.

The image provides a table of symbols and their corresponding meanings used in the context of discrete diffusion models. The table lists variables such as $K$ (vocabulary size), $\mathcal{V}$ (vocabulary), $L$ (sequence length), $\boldsymbol{x}$ (discrete sequence), $\boldsymbol{e}_k$ (one-hot vector), $T$ (diffusion steps), $\boldsymbol{Q}_t$ (transition matrix), $\bar{\boldsymbol{Q}}_t$ (cumulative transition matrix), $q(x_t | x_0)$ (forward marginal), $p_\theta(x_{t-1} | \boldsymbol{x}_t)$ (reverse transition), $\boldsymbol{\pi}_\theta(\boldsymbol{x}, t)$ (predicted categorical distribution), $\boldsymbol{c}$ (conditioning signal), $\beta_t$ (noise rate), $\alpha_t$ (cumulative survival probability), $\boldsymbol{R}_t$ (rate matrix), $s_\theta$ (score network), $\Delta^{K-1}$ (probability simplex), and $\text{m}$ (mask token).

Tokenization as a Design Axis

Tokenization is a primary design axis that shapes diffusion topology, difficulty, controllability, and cost.

Across domains, tokenization determines the geometry of the discrete state space, which in turn governs how corruption is defined, how hard denoising is, how easily constraints can be enforced, and how much computation is required.

Tokenization maps raw data—text, pixels, waveforms, or molecular graphs—into a finite set of discrete symbols that the diffusion model predicts; the granularity and vocabulary choice dictate the shape of the state space and thus every subsequent design decision.

The survey groups existing tokenization strategies into three recurring families, each embodying a distinct trade‑off across the four axes identified above.

Subword segmentation algorithms such as BPE, WordPiece, Unigram, and byte‑level tokenizers partition raw character streams into a fixed‑size vocabulary of variable‑length pieces. Because the vocabulary is built purely on frequency compression, token indices carry no intrinsic metric; adjacent indices may be semantically unrelated. Consequently, most text‑based discrete diffusion models adopt an absorbing‑state (masking) corruption that sidesteps the need for a notion of distance between tokens.

Continuous signals are discretized via vector‑quantization codecs (VQ‑VAE, VQ‑GAN) into indices of a learned codebook. The codebook embeddings endow the token space with a geometry: indices whose vectors are close in $\ell_2$ distance correspond to perceptually similar patches, frames, or spectral segments. This structure enables corruption processes that preferentially substitute nearby codes, offering a more informative denoising signal than uniform masking.

Scientific data often come already discretized: proteins use a 20‑amino‑acid alphabet, nucleic acids a 4‑letter alphabet, and molecular graphs have categorical atom‑type and bond‑type labels. These alphabets carry domain‑specific similarity matrices (e.g., BLOSUM for amino‑acids) and hard validity constraints (valence rules). Tokenization therefore aligns perfectly with the underlying physics or biology, enabling corruption processes that respect substitution probabilities and structural feasibility.

Tokenization Across Modalities

Quantized tokenization balances fidelity, efficiency, and stability across media diffusion models.

Media diffusion hinges on how continuous signals are discretized: VQ‑VAE offers simplicity, RVQ adds hierarchical fidelity, and SQ trades geometry for stability.

The VQ‑VAE pipeline encodes an image, audio, or video frame into a spatial grid of continuous latents, then snaps each latent to the nearest entry in a learned codebook. The discrete indices become the tokens fed to downstream diffusion models. VQ‑GAN replaces the pixel‑wise L2 reconstruction loss with a perceptual loss (VGG feature distance) and a patch‑level adversarial loss, sharpening outputs and encouraging more uniform codebook usage.

RVQ refines VQ by repeatedly quantizing the residual error of the previous stage, yielding a hierarchy of codebook indices that together reconstruct the signal with higher precision. Kim et al. predict the sum of all hierarchy embeddings in a single diffusion step, decoupling inference depth from sequence length.

SQ quantizes each dimension of a continuous latent vector independently onto a fixed, axis‑aligned grid. No learned codebook is required, eliminating collapse and cascade instabilities inherent to VQ and RVQ. However, the grid cannot adapt to the latent space’s intrinsic geometry.

**Figure 3.** Token families in discrete diffusion. (a) Semantic text and code tokens are frequency-defined categorical symbols with no intrinsic metric, so corruption is commonly represented as absorbing masking. (b) Quantized media tokens inherit a learned codebook geometry, enabling structured corruption toward nearby codes. (c) Natural scientific alphabets, such as nucleotide or amino-acid tokens, can use external similarity matrices and validity constraints to guide transition kernels.

Discrete Structures in Science

Token families vary in structure, corruption, and design constraints, exposing a granularity gap.

Across scientific domains, tokenization choices cluster into four families, each with distinct corruption strategies and design trade‑offs.

Tokens derived from natural language or programming languages form the “semantic” family. They are typically processed with standard byte‑pair or unigram vocabularies and evaluated with language‑model perplexity or code‑completion accuracy. The survey notes that masking‑based absorbing‑state diffusion is the dominant corruption method for these tokens, preserving syntactic validity while allowing joint reconstruction of distant dependencies.

Discrete representations of continuous signals are obtained by vector‑quantization or learned codebooks. The survey reports that token sequences encode quantized patches (e.g., VQ‑GAN tokens) and are trained with the same masking diffusion used for text, but the corruption must respect perceptual similarity to avoid implausible pixel‑level noise.

Biological sequences use small, fixed vocabularies (20 amino acids or 4 nucleotides). The survey highlights that long‑range co‑evolutionary couplings demand diffusion models that can jointly denoise distant residues. Absorbing‑state masking is the prevalent forward corruption because it never introduces biologically impossible symbols, and domain‑specific priors such as BLOSUM matrices can be injected as structured transition kernels.

Discrete representations of chemical graphs or combinatorial layouts map atoms/bonds or layout primitives to categorical node and edge types. The survey distinguishes string‑based tokenizations (SMILES, SELFIES, SAFE) from direct graph tokenizations (DiGress). Graph‑based diffusion must respect valence and connectivity constraints; masking or structured edge‑node corruption is used to avoid invalid intermediate states, and projection mechanisms are often added at inference.

The table categorizes token families based on their examples, metrics, typical corruption methods, and key design concerns.

Evaluating Tokenizers

Tokenization quality drives diffusion performance, yet diffusion‑specific diagnostics remain scarce.

Across surveyed studies, researchers agree that tokenizers must be evaluated both for static reconstruction quality and for how they interact with the diffusion process; the conspicuous gap is the lack of lightweight, diffusion‑facing diagnostics.

Measures the compression‑granularity trade‑off of a tokenizer. Larger vocabularies reduce sequence length but enlarge the categorical state space the diffusion model must traverse.

Quantifies information loss introduced by quantization or codec‑tokenization before diffusion training, using perceptual and distributional metrics.

Assesses whether a codebook’s capacity is being exploited, preventing collapse to a few active codes.

Evaluates whether geometric proximity in the embedding space aligns with semantic similarity of reconstructed outputs.

Measures how well the learned code‑space respects an external domain‑specific similarity prior (e.g., BLOSUM for proteins).

Compares reconstruction quality (rFID) with generation quality (FID) to expose token spaces that are hard for diffusion models despite good reconstructions.

Tracks the per‑timestep denoising loss L(t) to reveal how a tokenizer and corruption schedule distribute difficulty across diffusion steps.

Measures how robust a tokenizer is to changes in the diffusion noise schedule.

The field disagrees primarily on diffusion‑facing diagnostics: whether a tokenizer should be evaluated with model‑free proxies (e.g., neighborhood consistency) or only after end‑to‑end diffusion training (e.g., reconstruction‑generation gap).

Markov Chain Formulations

Transition matrices define how discrete tokens corrupt and how reverse denoising is computed.

The D3PM framework treats each token as a state in a discrete‑time Markov chain whose forward dynamics are governed by a transition matrix $Q_t$. By chaining these matrices we obtain the cumulative corruption matrix $\bar{Q}_t$, which directly yields the marginal noisy distribution.

The dominant practical trick is to predict the clean token $\hat{x}_0$ from $x_t$ and then plug that prediction into the closed‑form posterior above. This “predict‑$x_0$‑then‑posterior” approach sidesteps learning the reverse distribution directly and underlies most recent discrete diffusion models.

Across the surveyed families, the transition matrix $Q_t$ is the sole design lever; masking‑based matrices dominate because they turn denoising into a fill‑in‑the‑blank problem that bidirectional Transformers solve efficiently, while uniform or similarity‑biased matrices trade simplicity for harder inference.

Each token is independently replaced by a randomly drawn vocabulary token with probability $\beta_t$. The transition matrix is a convex blend of the identity and an all‑ones matrix, yielding a uniform stationary distribution. The closed‑form cumulative matrix makes analysis straightforward, but denoising is intrinsically hard because the model must pick the true token out of $K\!-\!1$ equally likely alternatives.

Tokens are replaced by a special mask token $m$ with probability $\beta_t$ and remain masked thereafter. The transition matrix operates on an extended vocabulary $V_m$ and drives the sequence toward an all‑mask stationary distribution. Denoising reduces to a fill‑in‑the‑blank task, which bidirectional Transformers excel at, and this design has shown the best empirical scaling across language and image benchmarks.

Corruption probabilities are modulated by token‑embedding similarity via a Gaussian kernel. Tokens with nearby embeddings are more likely to substitute each other, creating a coarse‑to‑fine corruption trajectory. While conceptually appealing, the survey notes that tuning the bandwidth $\sigma_t$ is delicate, leading to training instability and lower performance on generic text; however, in domains with meaningful similarity metrics (e.g., protein sequences) it can improve fidelity.

Developed in parallel with D3PM, this framework defines the forward process as a linear interpolation toward a uniform categorical distribution. The reverse process is derived via a discrete analogue of score matching, linking the model to score‑based diffusion. It matches D3PM in formulation but differs in how the reverse model is parameterized. The survey cites comparable performance on small‑scale tasks but limited scalability relative to masking‑based designs.

**Figure 2.** Discrete forward corruption and learned reverse denoising. Absorbing-state masking removes token identity, uniform substitution replaces tokens with categorical alternatives, and the reverse model denoises using full-sequence bidirectional context.

Absorbing-State Diffusion

Absorbing‑state masking dominates because it is simple, stable, and fits bidirectional Transformers.

Across diffusion studies, absorbing‑state (masking) corruption is preferred for its single‑parameter simplicity, stable classification loss, and seamless integration with bidirectional Transformers; the remaining camps trade this elegance for richer noise signals.

This camp treats corruption as a binary mask that replaces a token with a special “missing” symbol m. The forward process is governed by a per‑step masking rate $\beta$ₜ, yielding a closed‑form survival probability $\alpha$ₜ = ∏ₛ₌₁ᵗ(1 − $\beta$ₛ). Training reduces to a re‑weighted masked‑language‑modeling loss: the model predicts the original token at masked positions given the unmasked context. Because each step is a pure classification problem over the vocabulary, the loss landscape is smooth and the continuous‑time ELBO collapses to an integral of cross‑entropy terms, enabling stable large‑scale training.

Here corruption replaces a token with a uniformly random vocabulary entry rather than a special mask. The forward transition probability is a mixture: with probability $\alpha$ₜ the token stays unchanged, otherwise it becomes a random token. This forces the denoising model to solve a classification‑under‑label‑noise problem, which becomes harder as vocabulary size grows.

Hybrid schemes allocate a portion of the per‑step corruption probability to masking and the remainder to substitution. This yields intermediate states that are neither fully clean nor fully masked, encouraging the model to recover from plausible but incorrect tokens—a scenario that arises when the model’s own predictions are fed back during generation.

Masking treats a token as “absorbed” into a special missing state m; the model repeatedly predicts the original token for those masked positions using the surrounding context.

How does absorbing‑state masking differ from substitution noise, which also corrupts tokens?

Masking replaces a token with a dedicated “missing” symbol m, so the model knows the position is corrupted and must infer the original token. Substitution swaps the token for a random vocabulary entry, leaving the model uncertain whether the observed token is correct or noise; this adds a classification‑under‑label‑noise problem and weakens the training signal, especially for large vocabularies.

Continuous-Time Formulations

Continuous‑time diffusion reframes denoising as a CTMC, enabling flexible inference and flow‑matching links.

Continuous‑time diffusion reframes discrete denoising as a CTMC, giving a unified objective, flexible inference schedules, and a bridge to flow‑matching, while incurring matrix‑exponential costs.

Introduces a systematic CTMC formulation by defining explicit rate matrices for uniform, absorbing, and general transition processes. Training objectives are derived directly in continuous time, yielding an ELBO expressed as an integral that is invariant to the exact noise schedule beyond its endpoints. Because the model is trained continuously, inference can use any discretization of the time interval, enabling adaptive or accelerated sampling without retraining.

Develops the CTMC perspective independently, emphasizing the theoretical equivalence between discrete‑time chains and continuous‑time processes via piecewise‑constant rate matrices. The work highlights the unified objective derivation and notes the practical challenge of computing matrix exponentials for large state spaces.

Connects continuous‑time discrete diffusion to discrete flow matching and optimal‑transport formulations. By interpreting the CTMC as a flow, the authors import tools from transport theory and propose rate matrices with analytically tractable exponentials, enabling accelerated sampling and a clear link to physical jump processes such as the Ehrenfest model.

Reverse Transition Characterization

Score‑based reverse transitions tighten likelihood bounds and improve training stability.

Ratio‑based reverse transition parameterizations yield tighter ELBO bounds and more stable optimization than direct x₀ prediction, especially under uniform corruption.

This camp predicts the clean token distribution $\hat p(x_0\mid x_t)$ at each diffusion step and then derives the reverse transition probabilities via Bayes rule. The approach treats the denoising problem as a multi‑class classification task, producing an absolute probability over the vocabulary.

This camp learns a score function $s_\theta(x_t,t)$ that directly estimates the ratio of marginal state probabilities $q_t(k)/q_t(j)$. The ratio replaces explicit prediction of $x_0$ and enables exact computation of reverse transition probabilities via a change‑of‑variables identity for CTMCs.

This camp outputs unnormalized logits for each state, which are exponentiated and normalised to obtain reverse transition probabilities. It sits between the absolute‑distribution and ratio approaches, offering a flexible scaling of scores before softmax.

A Unifying Framework

We map discrete diffusion models onto a four‑component taxonomy and expose their common structure.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers