Hierarchical Continuous Diffusion Language Models
Hui Ren, Zihan Li, Chang Liu, Huidong Liu, Alexander Schwing
HC-DLM couples discrete token generation with a continuous latent trajectory to resolve cross-token dependencies in parallel decoding.
How can we overcome the token-independence bottleneck in discrete diffusion language models to enable better global constraint satisfaction?
Parallel decoding in discrete diffusion models suffers from a token-independence bottleneck: because each position is sampled independently, the model fails to capture the tight statistical dependencies required for complex reasoning tasks like Sudoku or mathematical planning. Hierarchical Continuous Diffusion Language Models (HC-DLM) solve this by introducing a two-level reverse process where a continuous latent trajectory acts as the persistent state, and tokens are read out at every step to serve as a conditioning scaffold for the next latent update. On structured reasoning and language modeling benchmarks, this hierarchical coupling consistently outperforms both discrete and continuous diffusion baselines at matched model sizes.
Paper Primer
HC-DLM organizes generation as a hierarchical chain: a continuous latent $x_t$ is denoised while simultaneously producing a discrete token scaffold $k_t$ that constrains the latent's evolution. This structure forces the model to resolve continuous-space ambiguity using the discrete token estimates, effectively routing cross-token dependencies through the shared latent state.
HC-DLM improves accuracy on structured reasoning tasks compared to matched-scale discrete and hybrid diffusion baselines.
On the "Hard" split of Sudoku puzzles, HC-DLM achieves 72.41% accuracy, outperforming the hybrid CCDD baseline (70.73%) and significantly exceeding purely discrete masked diffusion models. A clear performance lead in out-of-distribution reasoning, where global constraint satisfaction is most critical.
The hierarchical coupling is essential for performance; removing either the continuous latent or the discrete token feedback significantly degrades accuracy.
Ablation studies on Sudoku show that removing the discrete conditioning (Latent DM) drops accuracy from 94.21% to 50.46% on the Easy split and from 72.41% to 24.74% on the Hard split. The discrete scaffold is a primary driver of the model's ability to satisfy global constraints.
Why does this approach outperform standard discrete diffusion models?
Standard discrete diffusion models sample tokens independently at each step, failing to capture dependencies. HC-DLM uses a shared continuous latent to tie these tokens together, allowing the model to refine the entire sequence globally rather than position-by-position.
What is the primary scope of this framework?
HC-DLM is designed for tasks requiring global constraint satisfaction and bidirectional reasoning, such as logical puzzles and mathematical planning, as well as general unconditional language modeling.
The Token-Independence Bottleneck
We expose why existing diffusion language models fail to capture global token dependencies.
Discrete diffusion language models generate many tokens in parallel, but each token is sampled independently from its marginal distribution. This independence severs the statistical ties that many tasks—such as puzzles or planning—require, creating a bottleneck that prevents the model from satisfying global constraints.
When a diffusion step unmasks several positions at once, the model treats each position as if the others do not exist, so the joint distribution collapses to a product of independent marginals.
Continuous diffusion models avoid the marginal product by sharing a single continuous latent across all tokens, yet their denoiser never sees the discrete token configuration during the diffusion trajectory. Consequently, the latent is not anchored to a valid token arrangement until the final decoding step, leaving the same global‑dependency gap.
HC‑DLM resolves both issues by making the continuous latent the sole persistent state while reading tokens from it at every reverse step. The read‑out tokens become part of the generative model’s forward kernel, providing a scaffold that conditions the next latent update and thereby re‑introducing cross‑token dependence throughout the denoising process.
Discrete diffusion struggles with global constraints because parallel token sampling breaks the necessary cross‑token dependencies.
Discrete Diffusion Foundations
Foundations of discrete and continuous diffusion models and their hierarchical extension.
Discrete diffusion models treat text as a sequence of tokens that are gradually corrupted, while continuous diffusion models operate on real‑valued latent vectors that are denoised back to data.
Think of a sentence being slowly turned into a cloud of MASK symbols, one token at a time, and then reconstructed from that cloud.
During generation the model predicts each token independently given the noisy sequence, so cross‑token correlations are lost. This factorised form $p_{\theta}(k_{0}\mid k_{t})=\prod_{i}p_{\theta}(k_{i}^{0}\mid k_{t})$ is the source of the token‑independence bottleneck.
Imagine a point in space being blurred by Gaussian noise and then gradually sharpened by a learned velocity field that tells the point how to move back to its original location.
Run a continuous diffusion in a latent space while simultaneously tracking a discrete token chain; at each latent update the current latent state emits a provisional token sequence that is fed back as a scaffold for the next update.
The HC-DLM Architecture
HC‑DLM couples a continuous diffusion trajectory with a token scaffold to inject cross‑token context at every denoising step.
The token‑independence bottleneck forces a reverse process that must carry cross‑token information before any token is final, while latent diffusion alone would drift without anchoring to the token sequence.
The model threads a continuous latent trajectory through the denoising steps, using the current token scaffold to steer it, so each reverse step refines both the latent field and the token sequence together.
Step 1: Compute token distribution $p_{\theta}(k_T\mid x_T)$; suppose it yields $k_T=[\text{“the”},\text{
Step 2: Condition the continuous denoiser on $k_T$ and produce $x_{T-1}=p_{\phi}(x_{T-1}\mid x_T,k_T)$, which shifts the latent vectors toward the embedding of “the”.
Step 3: Re‑evaluate $p_{\theta}(k_{T-1}\mid x_{T-1})$, now obtaining $k_{T-1}=[\text{"the"},\text{"cat"}]$ as the second token becomes confident.
Step 4: Apply $p_{\phi}$ again to get $x_{T-2}$, now fully aligned with the embeddings of “the cat”.
The hierarchy lets the discrete scaffold progressively constrain the continuous latent, preventing the diffusion from wandering arbitrarily and enabling revisable token predictions at each step.
How does this hierarchy differ from a standard diffusion model that treats latent and token diffusion independently?
In a standard model the latent and token chains evolve in parallel without influencing each other, so the token scaffold never guides the latent trajectory. Here each reverse step explicitly conditions the latent denoiser on the current token state, allowing cross‑token context to flow into the continuous space and to be revised at every step.
**Figure 1.** HC-DLM coupled diffusion process. The forward direction independently corrupts the continuous latent trajectory $x_0 arrow x_T$ and the discrete token trajectory $k_0 arrow k_T$ via $q(x_t | x_{t-1})$ and $q(k_t | k_{t-1})$. The reverse direction couples the two channels: $p_\phi(x_{t-1} | x_t, k_t)$ advances the continuous state conditioned on the current token state, while $p_\theta(k_t | x_t)$ reads token distributions from the latent state, iterating until the clean states $(x_0, k_0)$ are reached.
Training Objective Derivation
Derives the ELBO for HC‑DLM and explains its components.
The two‑level reverse chain requires a training objective that respects the hierarchy while staying tractable for variational learning. Because the continuous and discrete corruptions are independent, a clean lower‑bound can be derived.
The ELBO gives a computable surrogate for the log‑likelihood of the observed token sequence, breaking the objective into reconstruction, prior‑matching, and hierarchy‑specific regularizers.
Reconstruction: assume $\log p_{\theta}(k_{0}\mid x_{0}) = -0.5$ (higher when the latent explains the token).
Prior matching: KL\big($q_{\psi}$($x_{0}$\mid $k_{0}$)\;\|\;$p_{\theta}$($x_{0}$)\big) = 0.5\) for this Gaussian pair.
Boundary continuous denoising: with a single diffusion step the term evaluates to $0$ (no further noise).
Encoder entropy: entropy of $\mathcal{N}([1,0],I)$ equals $\tfrac{1}{2}\log\big((2\pi e)^{2}\big) \approx 2.53$.
Boundary discrete token prediction: assume $\log p_{\phi}(x_{0}\mid x_{1},k_{1}) = -0.2$.
ELBO sum: $-0.5 - 0.5 + 0 + 2.53 - 0.2 = 1.33$.
Even in this minimal example the ELBO splits into a reconstruction reward, a KL penalty, and extra hierarchy‑specific terms that shape the latent dynamics.
How does this ELBO differ from the standard VAE ELBO?
The standard VAE ELBO contains only a reconstruction term and a KL divergence between posterior and prior. Here the ELBO adds three hierarchy‑specific pieces: a boundary continuous denoising term that aligns the final diffusion step, an encoder‑entropy term that encourages a spread‑out posterior, and a boundary discrete token prediction term that ties the discrete token predictor to the latent hierarchy.
The HC-DLM Loss Function
Defines the loss that drives HC‑DLM to learn coordinated token and latent dynamics.
The model must learn to reconstruct clean tokens while simultaneously shaping a continuous latent trajectory. Naïvely training only on token reconstruction leaves the latent hierarchy unused, and training only on the latent yields token‑wise independence. A joint objective that ties both together is therefore essential.
We combine three families of losses—reconstruction, entropy, and continuity—so that the model learns a coherent token‑latent pair at every denoising step.
Compute the reconstruction log‑likelihood: assume log p_$\theta$(x₀ | k₀) = ‑0.8.
Compute the encoder entropy: assume log q_$\psi$(x₀ | k₀) = ‑0.3, so the entropy contribution is 0.5 × (‑0.3) = ‑0.15.
Compute the continuous denoising KL at t = 1: assume KL = 0.4, weighted by $\lambda_{c}$ont × $\omega$₁ = 2 × 1 = 2, giving 0.8.
Sum the three contributions: J = 1 × (‑0.8) + (‑0.15) + 0.8 = ‑0.15.
This tiny example shows how the entropy term can offset a negative reconstruction loss, while the continuous denoising term can dominate the total objective when $\lambda_{c}$ont is large.
Monte‑Carlo training loop for HC‑DLM (single sample per iteration).
Training Procedure
Describes the step‑by‑step training loop that aligns continuous latents and discrete tokens.
The training loop interleaves three networks—encoder $q_{\psi}$, denoiser $d_{\phi}$, and token predictor $p_{\theta}$—so that each diffusion step updates both continuous latents and discrete tokens.
Each diffusion step is treated as a mini‑training episode: a timestep is drawn, the clean latent and token are corrupted, a joint loss is evaluated, and all three networks are updated together.
How does this loop differ from a standard diffusion model that only denoises continuous latents?
Standard diffusion updates a single network that predicts a clean latent from a noisy one. HC‑DLM updates three coupled networks: the encoder ties discrete tokens to the latent, the denoiser restores the latent, and the token predictor forces the latent to contain enough information to recover the original tokens. This joint update creates cross‑token context during training.
Step 1: Sample $t=2$. Encode $k_0$ → $x_0$ (e.g., $x_0 = (0.5, -0.3)$).
Step 2: Add continuous noise: $x_2 \sim \mathcal{N}(x_0, \alpha_2 I)$ → $x_2 = (0.8, -0.1)$.
Step 3: Add discrete noise: $k_2$ flips the second token with probability $0.2$ → $k_2 = (A,A)$.
Step 4: Compute loss $J$ (reconstruction error between $x_2$ and $x_0$, entropy of $q_{\psi}$, continuity between $x_2$ and $\hat{x}_0$).
Step 5: Back‑propagate and update $\theta$, $\phi$, $\psi$.
The toy illustrates how a single timestep simultaneously corrupts both modalities, forcing the three networks to cooperate to recover the original pair.
Training loop for HC‑DLM.
**Figure 2.** **HC-DLM training pipeline.** The encoder $q_{\psi}(x_0 | k_0)$ maps clean tokens $k_0$ to a continuous latent $x_0$, while independent forward kernels produce the noisy pair $(x_t, k_t)$. The token-conditioned denoiser $d_{\phi}(x_t, k_t, t/T)$ produces the clean-latent estimate $\hat{x}_0$ (loss $\mathcal{J}_{\text{cont}}$), the token predictor $p_{\theta}(k_0 | x_0)$ decodes $x_0$ to $\hat{k}_0$ (loss $\mathcal{J}_{\text{recon}}$), and the encoder is regularized by $\mathcal{J}_{\text{ent}}$.
After the model has learned to align latents and tokens, generation proceeds by reversing the diffusion process.
Sampling (reverse diffusion) for HC‑DLM.
Inference and Sampling
How the model reverses diffusion to generate coherent token sequences.
The inference stage runs the coupled reverse chain from Eq. (3), reusing the loss terms $J_{\text{recon}}$, $J_{\text{cont}}$, and $J_{\text{ent}}$ defined during training, but now focuses on generating tokens rather than updating parameters.
Generating a sequence is like unwrapping a gift layer by layer: each step peels back a bit of noise to reveal a cleaner latent, then reads the hidden token before re‑masking for the next peel.
t=4: Denoiser receives $x_4$ and produces $\hat{x}_0^{(4)}$; token predictor reads $\hat{k}_0^{(4)}$ from $\hat{x}_0^{(4)}$.
Re‑noise $\hat{k}_0^{(4)}$ with the forward kernel to obtain $k_3$; confidence exceeds the adaptive threshold, so only low‑confidence positions are re‑masked.
t=3: Denoiser takes $x_3$ (the noisy version of $\hat{x}_0^{(4)}$) and outputs $\hat{x}_0^{(3)}$; token predictor yields $\hat{k}_0^{(3)}$.
Re‑noise $\hat{k}_0^{(3)}$ to get $k_2$, again applying adaptive masking.
t=2 and t=1 repeat the same pattern, progressively sharpening the latent and token estimates.
After $t=1$, the final token $\hat{k}_0$ is returned as the generated sentence.
Even with only four diffusion steps the algorithm recovers a high‑quality token sequence because each step simultaneously refines the latent and the token, leveraging the hierarchical latent to propagate global context.
**Figure.** (a) Trajectory decoding. (b) Step-count robustness.
Experimental Setup
HC-DLM achieves state‑of‑the‑art Sudoku accuracy with modest parameters.
HC‑DLM outperforms the strongest Sudoku baseline by +2.0 % on the easy split.
Best baseline (ARM with ordering) reaches 92.65 % accuracy.
We evaluate HC‑DLM on three complementary tasks: structured reasoning (Sudoku), mathematical planning (Countdown), and general language generation (LM1B). For each task we compare against autoregressive, discrete diffusion, and continuous or hybrid diffusion baselines using comparable parameter budgets.
Sudoku baselines include the autoregressive model (ARM) with and without its learned variable‑ordering heuristic, and masked diffusion (MDM) variants using vanilla, top‑probability, and top‑probability‑margin sampling rules. Countdown baselines comprise GPT‑2 Scratch, Stream‑of‑Search, LLaMA, VDM, D3PM, RDM, and MDM decoding variants, plus the hybrid discrete–continuous CCDD reproduced at a matched 6 M‑parameter scale.
LM1B baselines consist of an autoregressive Transformer, discrete diffusion models (MDM, SEDD, Duo), and continuous diffusion models (Plaid, LangFlow). All baselines share the same sampling‑time generative parameter count, excluding token embeddings, to ensure a fair comparison.
Sudoku Reasoning Results
HC-DLM reaches 72.41% accuracy on hard Sudoku puzzles, outperforming prior diffusion and autoregressive baselines.
HC‑DLM leads on the out‑of‑distribution hard Sudoku split.
Table 1 shows HC‑DLM at 72.41% versus 70.73% for the next‑best hybrid baseline and far lower scores for diffusion‑only and larger autoregressive models.
**Table 1.** Sudoku puzzle-solving accuracy (%↑) on easy and hard splits. Best result per column in bold.
Mathematical and Language Results
HC-DLM sets new accuracy records on Countdown and achieves top perplexity on LM1B.
HC‑DLM outperforms the hybrid CCDD baseline on the CD5 subtask by 12.17 percentage points.
HC‑DLM achieves 37.52 % CD5 accuracy versus 25.35 % for CCDD at the same 6 M‑parameter scale.
For natural‑language modeling we evaluate HC‑DLM on the One Billion Word benchmark (LM1B) using 128‑token sequences and a 118 M‑parameter denoiser. Table 3 shows HC‑DLM attains the lowest generative perplexity among diffusion models, surpassing all discrete diffusion baselines and both continuous‑only decoders that generate in a single step. Entropy remains stable as NFE is reduced (see Fig. 5), and wall‑clock sampling time scales favorably with batch size (Fig. 4), confirming efficient parallel generation.
Ablation and Sensitivity Analysis
HC‑DLM addresses the token‑independence bottleneck with a continuous latent hierarchy.
HC‑DLM addresses the token‑independence bottleneck by using a continuous latent hierarchy that carries global context throughout denoising.
HC‑DLM achieves a generation perplexity of 75.5 on LM1B, outperforming the best diffusion baseline by 1.8 PPL.
Diffusion baselines report perplexities of 77.3 (Plaid) and higher.
HC‑DLM trains in 48 hours, roughly six times faster than the LangFlow baseline.
LangFlow requires 292 hours and Plaid 375 hours under comparable settings.
Guidance scale $w\!=\!2.75$ yields the lowest perplexity (~70) on LM1B.
Perplexity drops sharply from $w\!=\!1$ to $w\!=\!3.25$ then rises slightly toward $w\!=\!5$.
Related Work
We derive the ELBO for Proposition 1 by applying Jensen’s inequality and telescoping KL terms.
We prove Proposition 1 by expanding the ELBO step by step, starting from the marginal likelihood of $k_0$.
Apply Jensen’s inequality to the marginal likelihood of $k_0$.
Substitute the model factorizations into the bound.
Telescoping on continuous terms simplifies the sum.
Rewrite the remaining terms as KL divergences, completing the ELBO.
Comparison with Hybrid Models
We compare HC‑DLM to three hybrid diffusion models and discuss its training overhead.
This section expands the comparison of Section 5 by examining three representative hybrid diffusion models—VMD, CADD, and CCDD—and then highlights where HC‑DLM diverges.
Table 10 shows that only HC‑DLM allows the decoded token to be revised later, because its token predictor reads the continuously refined latent at every denoising step.
Across the Easy and Hard benchmarks, HC‑DLM attains the highest scores (94.21 % and 72.41 %), confirming that the continuous latent hierarchy improves both accuracy and robustness.
HC‑DLM’s joint optimization of an encoder, a continuous denoiser, and a token predictor makes each training step more expensive than a purely discrete masked diffusion baseline.
At inference time the encoder is discarded, so sampling remains faster than the baseline at equal step counts, but scaling the model to larger pretrained backbones will be limited by compute resources typical of academic settings.
Mathematical Proofs: Discrete Part
Derives the KL‑based decomposition of the training objective and proves Proposition 1.
This appendix works through the KL‑based terms that appear in the training objective, showing how they decompose into tractable learning signals.
Collecting the reconstruction term, the encoder‑entropy term, the prior‑matching term, and the KL contributions (i)–(iv) yields Proposition 1, which states that the full training objective can be written as a sum of these four components.
Applying the chain‑rule identity (20) with $A=k_t$ and $B=x_{t-1}$ yields the decomposition (19), which underpins the two complementary learning signals used in the HC‑DLM training objective.
Mathematical Proofs: Continuous Part
Derives the practical ELBO parameterization used for HC‑DLM training.
Step 5 rewrites the ELBO in terms of tractable Gaussian posteriors and a single clean‑token predictor, exposing two practical obstacles that the authors resolve.
Obstacle 1 notes that a separate predictor for every diffusion step $t$ would duplicate information already encoded in the known forward kernel, and would be inconsistent with the clean‑endpoint head already supervised by the reconstruction term.
Obstacle 2 observes that differentiating through the mixture in (25) yields a diffuse gradient because many clean tokens map to the same noisy symbol (e.g., the MASK token), so the authors replace the noisy‑token KL with a clean‑token upper bound via the data‑processing inequality.
Implementation Details
Implementation specifics of the encoder, denoiser, token predictor, and training details.
The architecture implements the practical objective of Eq. (6) with three jointly trained transformer modules: an encoder, a token predictor, and a continuous denoiser that interact as shown in Fig. 2.
The encoder $q_{\psi}(x_{0}\mid k_{0})$ is a bidirectional transformer that maps the token sequence $k_{0}$ to a continuous sequence $x_{0}\in\mathbb{R}^{M\times d}$ using a fully factorized Gaussian parameterization, enabling a closed‑form $\log q_{\psi}$ for the entropy regularizer $J_{\text{ent}}$.
The continuous denoiser $d_{\phi}(x_{t},k_{t},t/T)$ adopts an $x$‑prediction parameterization, concatenates the noisy latent $x_{t}$ with token embeddings $k_{t}$, and processes the joint sequence of length $M+L$ with self‑attention, while injecting the normalized timestep $t/T$ via AdaLN.
The token predictor is a lightweight transformer that consumes a clean latent estimate $\hat{x}_{0}(x_{t})$ and emits per‑position logits over the vocabulary $V$, defining $p_{\theta}(k_{0}\mid\cdot)$; during training it reads the encoder sample $x_{0}$, so its cross‑entropy loss does not back‑propagate into the denoiser.
Conditional generation augments each learned component with an additional conditioning embedding $c$, leaving the unconditional forward kernels unchanged, and yields a conditional bound $\log p(k_{0}\mid c)\ge L(k_{0},c)$ with the same four‑term structure as Eq. (4).
For Sudoku, puzzles are drawn from Radcliffe (2020) with a standard split of 1.8 M training and 0.1 M test puzzles, and a hard split of 1.1 M puzzles; each puzzle is serialized as an 81‑token sequence over $\{1,\dots,9\}$ with 0 marking blanks.
Countdown problems are generated with target values 10–100, using four or five input numbers; each example is serialized as a prompt “$n_{1},\dots,n_{k}= \text{target}$” and a comma‑separated chain of sub‑equations, with a 17‑symbol vocabulary.
Training uses three modules of hidden size 256, a single latent token ($M=1$) for Sudoku and Countdown, and DiT blocks (8 for encoder and denoiser, 2 for token predictor) with shared AdaLN and attention parameters following DiT‑Air.
Hyperparameters include Adam with learning rate $1\times10^{-4}$, batch size 256 per GPU on 7 NVIDIA L40s, loss weights $\lambda_{\text{recon}}=1.0$, $\lambda_{\text{ent}}=8\times10^{-3}$, $\lambda_{\text{cont}}=0.5$, dropout 0.1, EMA decay 0.999, and a BERT‑style reconstruction loss weight 0.2.
LM1B experiments scale the flow backbone to a 107.8 M‑parameter DiT‑small (hidden size 768, 14 blocks, 12 heads); sampling uses 1,024 unconditional samples with 128 steps and classifier‑free guidance weight $w=2.75$, reporting mean GPT‑2‑Large perplexity.
Table 7 reports generative perplexity and entropy across different numbers of NFEs, illustrating sample quality trends.
Figure 4 compares wall‑clock sampling time per batch for HC‑DLM versus LangFlow, showing HC‑DLM maintains lower time per batch at larger batch sizes.
Classifier‑free guidance modifies the denoiser output by computing $\hat{x}_{0,u}=d_{\phi}(x_{t},\varnothing_{t},t/T)$ and $\hat{x}_{0,c}=d_{\phi}(x_{t},k_{t},t/T)$, then extrapolating $\hat{x}_{\text{CFG}} = \hat{x}_{0,u} + w(\hat{x}_{0,c}-\hat{x}_{0,u})$, where $w>1$ increases reliance on the token scaffold.
Figure 5 shows that increasing $w$ lowers perplexity while preserving entropy, indicating improved sample quality without sacrificing diversity.
Table 8 presents three unconditional LM1B generation examples, demonstrating coherent text generation.
Table 9 varies the latent sequence length $M$ in $\{1,2,4,8,16\}$, revealing that accuracy improves up to $M=4$ before degrading, suggesting an optimal latent capacity.
Figure 6 reports wall‑clock inference time versus denoising steps, confirming HC‑DLM is consistently faster than MDM at equal step counts.
The uniform corruption kernel suits the hierarchy because it preserves a complete token scaffold for continual revision, unlike an absorbing kernel that freezes commitments early and hampers constraint‑dense tasks.