Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout
Zhuoran Zhao, Shengju Qian, Tongtong Liang, Xianghao Kong, Songchun Zhang, Junchao Huang, Guian Fang, Xin Wang, Pan Hui, Anyi Rao
Mask Forcing improves autoregressive video diffusion by injecting dual-noise masks into self-rollout to prevent mode collapse.
How can we prevent mode collapse in autoregressive video diffusion distillation caused by the mode-seeking nature of the reverse-KL objective?
Autoregressive video diffusion models often suffer from over-saturation and over-smoothing because their training objective—Distribution Matching Distillation (DMD)—tends to collapse onto a narrow subset of teacher modes. Mask Forcing addresses this by injecting cleaner, lower-noise tokens into the rollout input via random spatial and temporal masks. This dual-noise strategy forces the model to explore a broader range of teacher modes while using cleaner tokens as denoising guidance to reduce error accumulation. Experiments show this approach consistently improves visual quality and motion dynamics across multiple autoregressive baselines without requiring real video data or post-training.
Paper Primer
The core mechanism is a dual-noise masking rollout: at each denoising step, the model receives a mix of original noisy tokens and cleaner tokens re-noised to a lower noise level. This is like a student learning to solve a puzzle by occasionally being shown a partially completed version of the image, which both guides their next move and prevents them from getting stuck in a single, incorrect interpretation of the scene.
Mask Forcing significantly enhances visual quality and realism across diverse autoregressive video distillation baselines.
Human evaluation shows 79–83% preference for Mask Forcing over standard baselines like Self Forcing and LongLive. Consistent gains in HPSv3 and VisionReward scores across both chunk-wise and frame-wise generation settings.
The method accelerates training convergence and improves distributional alignment with the teacher model.
Lower Maximum Mean Discrepancy (MMD) scores in CLIP and V-JEPA feature spaces compared to unmasked baselines. Faster convergence in visual quality metrics (HPSv3) throughout the training process.
Why does the model need to be "forced" with masks instead of just training longer?
Standard distillation uses a reverse-KL objective that is inherently mode-seeking, meaning it naturally converges to a limited set of high-probability outputs. Masking forces the model to process a more diverse set of inputs, preventing this collapse without needing external real-world data.
Does this approach require changing the model architecture or inference process?
No. Mask Forcing is a training-time perturbation strategy applied during self-rollout; the model architecture and inference-time denoising schedule remain identical to the base autoregressive model.
Motivation and Context
We expose how reverse‑KL mode‑seeking collapses AR video diffusion distillation and introduce Mask Forcing to regularize rollouts.
Autoregressive video diffusion distillation inherits the efficiency of causal generation but inherits a severe mode‑seeking collapse from the reverse‑KL objective, limiting visual fidelity and diversity.
DMD aligns a causal student model with a bidirectional teacher by minimizing a reverse‑KL divergence between their sample distributions.
How does DMD differ from standard knowledge distillation?
Standard distillation matches logits or features point‑wise, while DMD matches full sample distributions via a reverse‑KL divergence, which directly shapes the student’s generative density rather than just its predictions.
Minimizing reverse‑KL forces the student to place probability mass on the teacher’s highest‑density regions, effectively “seeking” a few modes and ignoring the rest.
Why does reverse‑KL cause mode‑seeking rather than mode‑covering?
Because reverse‑KL penalizes the student for allocating probability where the teacher assigns little, the gradient pushes the student to concentrate on the teacher’s peaks, ignoring the tails that would provide coverage of diverse modes.
**Figure 1.** Mask Forcing improves autoregressive video generation through a Dual-Noise Masking Rollout strategy that alleviates the mode-seeking behavior of the reverse KL objective in DMD, enhancing video visual quality efficiently without incorporating real video data or additional post-training. We show consistent improvement in visual realism across short-video, long-video, and camera-controlled autoregressive generation.
The core trade‑off is between AR efficiency and the mode‑seeking collapse induced by reverse‑KL in DMD.
Autoregressive Video Generation
We expose how an autoregressive video model turns a text prompt into a sequence of denoised video chunks.
Training an autoregressive video model is tricky because errors made early in the generation cascade into later frames, causing severe quality degradation. The section therefore lays out the basic generative formulation and the three common training regimes that address this cascade.
Think of a video as a chain of short clips; the model produces each clip after seeing everything that came before and the text prompt.
Chunk 1: sample a noise $\epsilon_1\sim\mathcal{N}(0,1)$, pick $t=0.5$, compute $x_1^{t}=0.5\cdot0.2+0.5\epsilon_1$, then denoise to recover $\hat{x}_1\approx0.2$ and store it in the KV cache.
Chunk 2: form $x_2^{t}=0.5\cdot0.7+0.5\epsilon_2$, condition the diffusion network on $\hat{x}_1$ from the cache and the prompt $c$, denoise to $\hat{x}_2\approx0.7$.
Chunk 3: similarly compute $x_3^{t}=0.5\cdot0.4+0.5\epsilon_3$, attend to $\{\hat{x}_1,\hat{x}_2\}$, and denoise to $\hat{x}_3\approx0.4$.
The final generated video is the sequence $(\hat{x}_1,\hat{x}_2,\hat{x}_3)$, which matches the ground‑truth pixels.
Even with a tiny toy example, the model must repeatedly query the cache and denoise each noisy chunk, illustrating the core sequential dependency that later training tricks aim to stabilize.
Different paradigms decide how much noise and how much ground‑truth context the model sees while learning to denoise each chunk.
The Mask Forcing Mechanism
Dual-Noise Masking injects low‑noise signals via random masks to diversify rollout trajectories and reduce mode collapse.
During self‑rollout DMD training the reverse‑KL objective drives the student distribution toward a narrow set of high‑density modes, producing over‑saturated and over‑smoothed video frames.
We perturb each denoising step with a mixture of a low‑noise sample and the regular noisy sample, letting a random binary mask decide which tokens come from which source.
How does Dual-Noise Masking differ from simply adding extra Gaussian noise at every step?
Simple extra noise would raise the overall noise level uniformly, but Dual-Noise Masking injects a *structured* lower‑noise signal at selected token positions. The mask preserves clean context for part of the input, guiding the network while still perturbing the trajectory, which a uniform noise boost cannot achieve.
Step 1 (t₁ = 0.4): the student predicts a clean chunk $\hat{x}_0$. We sample a lower timestep t′ = 0.2 (within the window) and compute $x_{t'}$ = 0.8 $\hat{x}_0$ + 0.2 $\epsilon$.
We draw a binary mask M = [1, 0]; token 1 takes the cleaner sample $x_{t'}$, token 2 keeps the original noisy sample $x_{t₁}$ = 0.6 $\hat{x}_0$ + 0.4 $\epsilon$.
Mixing yields $x_{\text{mix}} = [x_{t',1},\,x_{t₁,2}]$. Feeding this to the student produces an updated clean prediction $\hat{x}_0^{(1)}$.
Step 2 (t₂ = 0.8): repeat the process with a new lower timestep t′ = 0.6, a fresh mask M = [0, 1], and obtain the final mixed input for the last denoising update.
The mask lets the network learn to rely on partially clean tokens as anchors, while the random lower‑noise timesteps keep the rollout trajectory diverse.
For each chunk i, initialize the noisy input $x_{t_T,i}$ at the highest noise level $t_T$ = 1.
Iterate over denoising steps j = T … 1:
Predict a clean chunk $\hat{x}_{0,i,j}$ from the current noisy input.
Sample a lower‑noise timestep t′ ∈ [ max(`t_min`, $t_j$ − $\Delta$/$N_t$), $t_j$ ] and compute the cleaner sample $x_{t',i}$.
Sample a binary mask $M_i$ with ratio $\alpha$ and form $x_{\text{mix}} = M_i \odot x_{t',i} + (1-M_i) \odot x_{t_j,i}$.
Feed $x_{\text{mix}}$ and the timestep $t_j$ to the student to obtain the next noisy input for step j‑1.
When the exit step s is reached, store $\hat{x}_{0,i,s}$ in the KV cache for the next chunk.
Dual‑Noise Masking rollout pseudocode (training)
**Figure 2.** (a) Standard self-rollout DMD training suffers from mode collapse due to the mode-seeking reverse-KL objective, resulting in over-saturation and limited realism. (b) Mask Forcing perturbs the student rollout trajectory through dual-noise masking rollout, encouraging broader teacher-mode coverage while providing cleaner context for denoising noisy tokens.
By perturbing the rollout with structured low‑noise signals, Dual‑Noise Masking mitigates mode‑seeking behavior while preserving the original denoising schedule at inference time.
Empirical Results
Mask Forcing raises total scores by up to +6.5 points across all baselines.
Mask Forcing consistently lifts total scores, achieving up to +6.5 points improvement in the chunk‑wise setting.
Table 1 shows the Self Forcing total score rising from 38.50 to 45.03, a +6.53 gain—the largest increase among all baselines.
**Figure 10.** Qualitative comparison of Self Forcing and Mask Forcing (Ours) on Sekai-Game and Sekai-Walking. As the rollout progresses, Self Forcing videos darken and lose fine details. Mask Forcing mitigates this degradation, preserving rich details throughout generation.
**Figure 11.** Additional qualitative comparisons between Self Forcing and Self Forcing (+Ours).
The figure displays six sets of comparative video generation results, each showing a sequence of frames generated by "LongLive" versus "LongLive + Ours". Each set includes a descriptive prompt below the frames: 1. A violinist in a street. 2. A man on a winter night with aurora. 3. A cat mermaid swimming. 4. A strawberry bear walking in a forest. 5. A person on a ship during a storm. 6. A serene lake surrounded by mountains.
**Figure 13.** Additional qualitative comparisons between Causal Forcing and Causal Forcing (+Ours).
**Figure 3.** Qualitative comparison of our method (+Ours) against different baselines. Visual results show that our method generates videos with higher visual quality and greater realism, exhibiting fewer over-saturation artifacts and richer high-frequency details.
**Figure 5.** V-JEPA2 Maximum Mean Discrepancy (VMMD) over training steps on the evaluation set. Incorporating Mask Forcing leads to faster convergence across various baselines.
**Figure 6.** Pairwise human preferences between baseline models with and without Mask Forcing. Blue and red denote preferences for models without and with Mask Forcing, respectively, while hatched regions indicate ties.
**Figure 7.** Pairwise human preference between joint distillation methods (DistillAlign and Causal-rCM) and Mask Forcing.
This figure presents a qualitative comparison of video generation results across three different methods: Causal-rCM, DistillAlign, and Ours. The figure is organized into three distinct blocks, each corresponding to a specific prompt describing a scene. Each block contains three rows of six frames, representing the temporal progression of the generated video for each method. The prompts are provided below each block, detailing the scene, subject, and camera movement.
**Figure 9.** HPSv3 and CMMD over training steps on the evaluation set for LongLive and LongLive (+Ours).
**Figure 4.** Qualitative results of single-prompt long-video setting on LongLive. Incorporating our method (+Ours) enhances the video generation quality with more visual details.
Ablation Studies
We dissect how each Mask Forcing component impacts quality, dynamics, and convergence.
We run a suite of ablations on the 100‑prompt set, using Self Forcing as the baseline, to isolate the contribution of each Mask Forcing design choice.
Mask ratio $\\alpha=0.2$ yields the highest Dynamic score (82) while keeping HPSv3 at 9.84.
Table 3 reports the trade‑off across ratios.
A timestep window $\\Delta=450$ achieves the peak HPSv3 of 9.98 with Dynamic 80.
Table 4 lists performance for each window size.
Mask scheme “per‑frame, per‑step” attains a Dynamic score of 74, the strongest among temporal variants.
Table 5 compares spatial‑temporal mask configurations.
**Table 2.** Comparison of LongLive and LongLive + Ours on VBench metrics.
**Table 3.** Mask ratio $\alpha$ ablation.
**Table 4.** Timestep window $\Delta$ ablation.
**Table 5.** Mask scheme ablation.
Theoretical Justification
We derive the KL decomposition of the masked rollout distribution and its implications for reverse‑KL training.
We first formalize the distributions that appear when Mask Forcing is applied. The teacher’s noisy marginal at noise level $\tau$ is denoted $p_\tau(x)$, while a masking trajectory $V$ (a set of masks and cleaner timesteps) induces a conditional student distribution $q^\theta_\tau(x)$. Averaging over all possible trajectories yields the marginal student distribution $\bar q^\theta_\tau(x)$, which is the object of the reverse‑KL objective.
$D_{\text{KL}$}(\bar q^\theta_\tau \,\|\, p_\tau) = \mathbb{E}_V\big[ $D_{\text{KL}$}($q^V_{\theta,\tau}$ \,\|\, p_\tau) \big] - I_\theta(V ; X_\tau \mid c)
Table 6 reports the empirical KL values for Mask Forcing (our method) versus the baselines Causal‑rCM and DistillAlign on the 100‑prompt benchmark, confirming that the mutual‑information term is indeed reduced by our masking strategy.
Supplementary Experiments
Additional quantitative, qualitative, and algorithmic details for Mask Forcing.
We supplement the main paper with further experiments that probe diversity, long‑duration generation, and an interactive world‑model scenario, and we expose the full training loop for Mask Forcing.
Table 8 reports CLIP and HPSv3 scores for LongLive and LongLive + Mask Forcing across five 6‑second windows (0‑6 s … 24‑30 s). In every window, the Mask‑Forcing variant attains higher scores, confirming that the improvement persists throughout long rollouts.
The mode‑coverage analysis shows that, under limited model capacity, reverse‑KL drives the conditional rollout distribution toward a single high‑density region of the teacher. Randomized masking trajectories generate distinct conditional distributions, yielding a positive mutual information $I_{\theta}(V;X_{\tau}\mid c) > 0$. Equation 21 then guarantees that the marginal mixture of these trajectories has strictly lower reverse‑KL than the average of the individual conditionals, enabling coverage of multiple teacher‑supported modes without forcing each conditional to span the full support.
In the joint‑distillation comparison (Section 6.2), Mask Forcing applied to the Causal Forcing baseline outperforms both DistillAlign and Causal‑rCM on visual quality, motion dynamics, and semantic alignment, as reflected in human‑preference percentages of 83 % over DistillAlign and 77 % over Causal‑rCM.
Additional LongLive experiments (Section 6.3) reveal faster convergence: Mask Forcing reaches higher HPSv3 and lower CMMD in fewer training steps, and maintains superior CLIP and HPSv3 scores across all 6‑second intervals on MovieGen, despite the inevitable decay caused by error accumulation in autoregressive rollouts.
Diversity evaluation (Section 6.4) confirms that Mask Forcing consistently raises CLIP diversity for all three baselines and boosts DINO v3 diversity for Causal Forcing and LongLive, while only slightly reducing DINO diversity for Self Forcing.
The interactive world‑model experiment (Section 6.5) demonstrates that Mask Forcing mitigates the darkening and detail loss observed with Self Forcing during camera‑controlled image‑to‑video generation, preserving richer textures throughout the rollout.
Algorithm 1 – Mask Forcing training loop (no extra forward passes required).