Speculative Forcing for Efficient Autoregressive Video Generation
Hyunwook Choi, Seungryong Kim, Dongyoon Han, Sangdoo Yun
Speculative Forcing accelerates autoregressive video generation by committing converged tokens early, skipping unnecessary denoising steps.
How can we accelerate autoregressive video generation by dynamically stopping the denoising process for tokens that have already converged?
Autoregressive video generators typically force every token through the full denoising schedule, even though many tokens settle into their final appearance long before the last step. Speculative Forcing treats intermediate clean estimates as drafts, using a verifier to identify and freeze converged tokens while continuing to refine only the remaining active ones. This approach maintains the quality of full-step sampling while reducing computation, achieving up to 1.5× higher throughput on long-video and world-model benchmarks.
Paper Primer
The core mechanism hinges on token-wise commitment: each token is evaluated at every step, but once a verifier confirms it has converged, it is "accepted" and removed from the denoising loop. The accepted token's cached keys and values remain available as context for the remaining active tokens, ensuring that the model's autoregressive dependencies are preserved without redundant computation.
The verifier acts like a quality-control gate: it reads the current latent draft and decides whether to "ship" it as the final output or send it back for more refinement. This gate is implemented either as a training-free check (comparing similarity to history or stability between steps) or a learned 3D convolutional head that predicts the remaining convergence gap.
Speculative Forcing maintains full-step generation quality while significantly reducing the average number of denoising steps per token.
On the LongLive benchmark, the method matches the four-step generator's quality using ~1.7 denoising steps per token. Throughput increases by over 1.5× on long-video tasks and 1.35× on world-model rollouts.
Token-wise commitment outperforms frame- or chunk-level adaptive denoising.
Ablation studies show that chunk-wise acceptance fails to distinguish between converged and unsettled tokens within the same region, leading to higher error drift and lower quality compared to the token-wise approach. Token-wise commitment achieves lower drift and higher quality at comparable mean denoising steps.
Why is this approach better than simply skipping steps for the entire model?
Uniform skipping forces all tokens to share the same denoising depth, which ignores the fact that different regions of a video—such as static backgrounds versus newly appearing structures—require different amounts of refinement. Speculative Forcing allows the model to spend compute only where it is actually needed.
Does this method require retraining the underlying video generator?
No. The framework keeps the generator frozen and only trains a shallow convolutional head (for the learned verifier) or uses training-free heuristics, making it compatible with existing distilled autoregressive models.
Researchers can now treat denoising depth as a dynamic, token-level variable rather than a fixed hyperparameter, enabling real-time streaming generation without sacrificing visual fidelity.
Introduction and Motivation
Uniform denoising wastes compute; Speculative Forcing stops early for converged tokens.
Autoregressive video generators apply the same number of denoising steps to every token, even though many tokens already match their final output after only a few steps. This uniform computation wastes cycles and limits streaming efficiency. We observe that tokens reproducing previously seen content tend to settle early, while regions introducing new structures keep changing until the last step.
Applying an identical denoising schedule to all tokens ignores the fact that different spatial‑temporal regions need different amounts of refinement.
Speculative Forcing addresses this waste by treating each intermediate clean estimate as a draft and using a verifier to decide whether the draft is already close enough to the final output. Accepted drafts are frozen as the token’s final value and no longer participate in subsequent denoising, while their $KV$ caches remain available for the remaining tokens. Two verifier variants are provided: a training‑free one based on similarity and stability, and a learned one that predicts the remaining error from the generator’s intermediate features.
**Figure 1.** Compared with full-step sampling (a), Speculative Forcing treats intermediate token estimates as drafts, accepts those that have converged, and refines only the rest (b), achieving comparable quality with fewer denoising steps and higher throughput (c).
The key inefficiency in current AR video generators is the blanket application of uniform denoising to every token.
The Speculative Forcing Mechanism
Speculative Forcing lets tokens stop denoising early by using a verifier to predict convergence.
Different tokens reach their final appearance at different rates, so applying the same number of denoising steps to every token wastes compute. Speculative Forcing exploits this disparity by stopping denoising for tokens that have already converged.
Instead of forcing every token through a fixed schedule, the method treats each intermediate estimate as a draft and asks a verifier whether the draft is already close enough to the final result.
How does Speculative Forcing differ from simply truncating the denoising schedule for all tokens?
Truncating applies the same early stop to every token, discarding useful refinement for tokens that have not yet converged. Speculative Forcing makes token‑wise decisions, preserving refinement for hard‑to‑converge regions while still cutting work for easy ones.
The verifier is a lightweight predictor that estimates how far a token’s current draft is from its final output.
Why not use the raw distance ‖ĥ₀^{(k)}(i)−ĥ₀^{(K)}(i)‖ instead of the normalized relₖ(i) ?
Raw distances vary with overall video intensity; normalizing by $\sigma$ₜ makes the threshold $\epsilon$ scale‑invariant across scenes and prevents a fixed $\epsilon$ from being too strict in low‑energy videos or too lax in high‑energy ones.
Each token has its own convergence step k* (i); once a token’s draft is close enough, it is fixed and no longer participates in later denoising passes.
Step 1 drafts: rel₁ = [0.15, 0.35, 0.08, 0.40]. Tokens 1 and 3 satisfy rel₁ ≤ 0.2 → k* (1)=k* (3)=1.
Step 2 drafts for remaining tokens 2 and 4: rel₂ = [0.12, 0.18]. Both now satisfy the criterion, so k* (2)=k* (4)=2.
All tokens are committed by step 2, so the final chunk uses drafts from steps 1 and 2 only, saving one full denoising pass for each token.
The example shows that early acceptance can cut the total denoising work from 3 steps per token to an average of 1.5 steps, illustrating the speed‑quality trade‑off.
Without learning, the verifier uses two cheap proxies: similarity to already clean content (redundancy) and change between consecutive drafts (stability).
A shallow convolutional head predicts the convergence gap ĥᵣₖ(i) directly from the generator’s intermediate hidden state.
**Figure 2.** **Token-wise convergence during denoising.** We visualize latent differences between each chunk's $k$-step prediction and the baseline four-step output. (a) In video generation, background and object regions that remain similar to history tend to converge early, while other regions continue to change. (b) In world generation, previously observed regions often converge early, whereas newly revealed regions tend to require further denoising.
**Figure 3.** **Speculative Forcing.** Baseline runs every token through the full denoising schedule. Speculative Forcing instead treats each token's intermediate clean estimate as a draft and asks a verifier to predict its remaining gap $rel_k(i)$ to the full-step output. Tokens predicted to have converged are accepted and committed, after which they enter the next step only as keys and values, so the following denoising pass is computed for the rejected tokens alone.
Performance and Quality Evaluation
OursTF delivers a 71% speedup while preserving video quality.
Our training‑free verifier (OursTF) reaches 1.71 FPS on 30‑second videos, a 71% speedup over the 4‑step baseline.
Table 2 shows OursTF 1‑step achieving 1.71 FPS versus 4.00 FPS for the 4‑step LongLive baseline.
LongLive is a baseline autoregressive video generator that applies a fixed number of denoising steps uniformly to every token.
How does LongLive differ from a conventional autoregressive generator?
LongLive applies the same denoising schedule to all tokens, whereas a conventional generator could stop early for tokens that have already converged, saving compute.
Lingbot‑World 2.0 is an autoregressive world‑model that predicts camera motion and scene dynamics jointly.
Why is Lingbot‑World 2.0 a harder test than standard video generation?
Because it requires the generator to keep spatial coherence while the camera moves, exposing any drift or inconsistency in the generated world.
**Figure 4.** Qualitative comparisons on video generation and world modeling. Both our head and training-free verifiers preserve visual quality comparable to the four-step baseline without visibly increasing quality drift over the illustrated rollouts on our benchmark.
Throughput gains of up to 71% are achieved without sacrificing video quality.
Ablation Studies
We isolate how verification tolerance, signals, and granularity impact quality and speed.
We probe three design knobs—verification tolerance, the verification signal, and the granularity of acceptance—to see how each influences the speed‑quality balance of our video generator.
Setting the verification tolerance to $\\epsilon = 0.20$ cuts the mean denoising steps to 2.95 and improves drift while preserving the four‑step baseline quality.
Table 4 shows $\\epsilon=0.20$ yields 2.95 mean steps, 22.4 FPS, quality 82.72, imaging quality 72.64, and drift 2.862, outperforming the uniform 4‑step baseline.
Replacing our learned verifier with the adapted temporal‑difference or channel‑variance signals drops quality by roughly 2 points at comparable mean steps.
Table 5 reports quality 80.97 (temporal difference) and 78.23 (channel variance) versus 82.65 for our learned verifier, with similar mean steps.
Token‑wise commitment yields higher quality (+0.07) and lower drift (2.839) than chunk‑wise acceptance at similar mean steps.
Table 5 shows OursH (token‑wise) achieving quality 82.65 and drift 2.839 versus chunk‑wise acceptance’s quality 82.54 and drift 3.214, with mean steps 2.55 vs 2.61.
**Figure 5.** Qualitative ablations of verification and commitment granularity. Our learned verifier handles distortions that challenge training-free verification. Premature chunk-wise acceptance propagates errors to later chunks, while motion-based verification struggles to distinguish camera-induced changes from distortions.
Additional Results
Additional quantitative results on verifier variants and quality metrics.
Our OursH variant reduces drift to 2.839 and improves imaging quality to 72.65, outperforming all baselines.
Table B shows drift = 2.839 and imaging quality = 72.65 for OursH, lower drift and higher imaging quality than on‑policy training, per‑token MLP, and the w/o‑spatiotemporal‑smoothing variant.
Table B compares four verifier designs. OursH matches the on‑policy and per‑token MLP baselines in throughput (≈24.5 FPS) while achieving the lowest drift and the best imaging quality.
A per‑token MLP, which replaces the convolutional head, attains verification accuracy comparable to OursH, indicating that the backbone’s contextualized token features already encode useful convergence cues.
The latent‑based signals exhibit two phases: redundancy dominates the first denoising step, while stability becomes stronger after the token is refined. Our training‑free verifier exploits this by applying a redundancy threshold early and a stability threshold later, using two separate thresholds.
Spatiotemporal smoothing across neighboring tokens encourages coherent acceptance decisions, which reduces drift (2.839 → 2.882) and raises imaging quality (72.65 → 72.54) compared to the unsmoothed variant.
Table C reports extensive quality metrics for several step‑wise generation strategies on LongLive. Across all metrics, the OursH variants (especially with $\epsilon = 0.28$) consistently rank near the top, confirming the effectiveness of speculative forcing combined with the verifier.
Limitations and Caveats
Limits discussion: extra results, policy details, and noted constraints of the approach.
Speculative Forcing stops denoising early for tokens that have already converged, unlike uniform denoising which applies the same number of steps to all tokens.
We report full quality and semantic dimensions for LongLive in Tables C–F, covering both 5‑second (VBench) and 30‑second (VBench‑Long) videos. Uniform sampling with one to four denoising steps is compared against Speculative Forcing at $\epsilon$ = 0.24 and $\epsilon$ = 0.28. Additionally, we implement the step‑allocation policy of Surprise Forcing, ranking the maximum cosine distance between adjacent latent frames of the first clean estimate across the preceding 20 chunks; chunks below a 0.4 rank threshold use only the first and fourth denoising steps, while the rest follow the full schedule, preserving LongLive’s original memory management.
**Table E.** Quality evaluation on LongLive. Evaluated on VBench-Long.
Additional qualitative results are provided in the supplementary video materials, offering visual confirmation of the quantitative trends.
Implementation Details
Implementation specifics for LongLive and Lingbot‑World 2.0, including schedules, verifier design, and caching.
LongLive uses an attention sink of three latent frames, a local attention window of twelve latent frames, and processes three latent frames per chunk. Short videos contain eighty‑one frames, while long videos contain four‑hundred‑seventy‑seven frames.
Lingbot‑World 2.0 runs the 1.3 B Causal‑Fast model with an attention sink of six latent frames, a local window of eighteen latent frames, and four latent frames per chunk. Each benchmark video has 253 frames, and per‑frame camera trajectories are encoded via Plücker embeddings normalized by the maximum translation magnitude.
Training‑free verifier thresholds are ($\tau_{r}$ed, $\tau_{s}$tab) = (0.497, 0.327) for five‑second LongLive videos, (0.357, 0.235) for thirty‑second LongLive videos, and (0.375, 0.300) for Lingbot‑World 2.0.
Both models share the shifted timestep schedule [1000.0, 937.5, 833.3, 625.0], taken from a 1000‑step flow‑matching schedule with a shift of 5.0. A token accepted at step k follows exactly the first k steps of this schedule, while uniform‑step baselines use the same prefixes for fair comparison.
The learned verifier reads hidden states from the generator’s final transformer block and consists of layer normalization followed by three 3D convolutions (1536→96→48→1). The first two convolutions use 3 × 3 × 3 kernels with GELU activations; the final convolution uses a 1 × 1 × 1 kernel. Its output is average‑pooled over 2 × 2 × 2 token blocks and up‑sampled with nearest‑neighbor interpolation. The head contains 4.11 M parameters and is trained with smooth‑ℓ₁ loss, Adam (lr = 10⁻³), batch size 4.
Within each chunk, tokens accepted early keep their layer‑wise keys and values from the acceptance step, while active tokens continue updating their KV states. After the chunk finishes, the generator performs a clean forward pass at timestep 0, refreshing KV states for all tokens regardless of when they were accepted, so subsequent chunks attend to the fully assembled output.
Evaluation Details
Evaluation details: metrics, verification accuracy, and benchmark construction.
We apply spatiotemporal smoothing to each verifier’s score map before thresholding at every verification step. Spatial smoothing uses a normalized $9\times9$ Gaussian kernel ($\sigma$ = 1.0 token) with reflect padding, applied independently to each latent frame. Temporal smoothing then averages scores over the current and subsequent latent frames within the same chunk, using Gaussian weights ($\sigma$ = 1.0 latent frame).
Only tokens that have not yet been accepted are forwarded through the generator at subsequent denoising steps. We reduce execution overhead through shared indexing and in‑place KV updates, and we merge each chunk’s clean pass with the next chunk’s first denoising pass while preserving timestep conditioning, attention dependencies, and KV update order.
All baselines use the same applicable execution optimizations for fair throughput comparisons.
We follow VBench (Huang et al., 2023) for 5‑second videos and VBench‑Long (Huang et al., 2024) for 30‑second videos, reporting all 16 dimensions. For Lingbot‑World 2.0 we aggregate six quality dimensions using VBench’s official normalization and weights, excluding temporal flickering because intended camera motion can produce frame‑to‑frame pixel changes.
Following Reward Forcing (Lu et al., 2025) we measure Drift as the standard deviation of imaging‑quality scores across non‑overlapping 2‑second segments, averaged over videos. Lower Drift indicates more consistent imaging quality over time.
The table presents a comparison of different verifier signals and learned verifier designs across varying epsilon ($\epsilon$) values and $k$ parameters. The table is divided into two main sections: "Learned verifier (head) design" and "Latent-based verification signals".
We construct a fixed benchmark of 200 scenes for Lingbot‑World 2.0, each consisting of an initial image, a text prompt, and a camera trajectory. All methods use the same conditioning inputs and sampling seeds. Qwen2.5‑7B‑Instruct generates scene descriptions, and Z‑Image‑Turbo generates the initial images at $1376\times768$.
Each video contains 253 frames, with the 252 camera transitions divided into five control segments of lengths 50, 50, 50, 50, and 52. For each segment we uniformly sample an action (forward, backward, strafe left/right, yaw left/right, stop) while avoiding repetition of the preceding segment’s action. Translation speeds are drawn from $U(0.2, 1.0)$, and yaw rates from $U(0.6, 2.0)$ deg/frame.
To quantify denoising computation we report the average number of denoising evaluations per token across all chunks and videos, excluding the clean pass used to refresh the KV cache. Throughput is measured on a single NVIDIA H100 as generated frames divided by the DiT forward time, excluding VAE encoding and decoding.
We evaluate verification accuracy on rollouts with commitment decisions determined by the oracle criterion at $\epsilon = 0.24$. Each chunk is denoised through all four steps to obtain complete convergence labels, while its output is assembled from the estimates selected by oracle commitment and used as history for subsequent chunks. Table A reports AUROC across three labeling tolerances, with the rollout commitment tolerance fixed at $\epsilon = 0.24$.
Supplementary Data
Additional tables detail speed‑up factors, overhead analysis, and application contexts.
The clean forward pass that refreshes the KV cache adds one full pass per chunk. Assuming computation scales linearly with token evaluations, the ideal speed‑up relative to the four‑step generator is $5/(\bar{k}+1)$. Approaches that reuse noisy KV states, such as CausalRCM (Zheng et al., 2026), could avoid this pass and move the ideal speed‑up toward $4/\bar{k}$.
In practice, throughput does not scale proportionally with the mean denoising depth $\bar{k}$. Each chunk continues until its last active token is accepted, incurring forward‑pass overhead at every executed step even when few tokens remain active.
Reducing this overhead would bring the realized speed‑up closer to the ideal implied by the mean denoising depth. Token‑wise commitment may be especially effective when much of the target content is already constrained by available conditions.
In talking‑head generation (Huang et al., 2026), the background can stay largely unchanged while motion concentrates around the face and upper body. Environment‑grounded world models such as Seoul World Model (Seo et al., 2026) and World Observer (Choi et al., 2026) provide warped scene observations and reference images for parts of the target view, leaving uncovered regions to be generated. Dexterous World Models (Kim et al., 2025) render static scenes along the camera trajectory and generate interaction‑induced changes while preserving unaffected content.
Table F presents a semantic evaluation on LongLive (VBench‑Long), comparing eight methods across fourteen metrics including Mean Steps, Throughput (FPS), and various quality dimensions.
The provided image contains a table comparing various methods across several metrics including Mean Steps, Throughput (FPS), Drift, Total, Semantic, Object Class, Multiple Objects, Human Action, Colour, Spatial Relation, Scene, Appear. Style, Temporal Style, and Overall Consist.