Speculative Sampling in Interactive Video Generation
Yubo Huang, Yuxuan Zhang, Jiaming Liu, Sirui Zhao, Enhong Chen
Speculative Sampling decouples joint video generation from output commitment to enable low-latency interaction without shrinking chunks.
How can we maintain the high quality and GPU efficiency of large-chunk video generation while still responding to user interactions in real-time?
Interactive video generation requires immediate responses to new user prompts, but conventional systems force a trade-off: smaller generation chunks allow faster updates but degrade temporal quality and GPU efficiency. Speculative Sampling resolves this by generating large, high-quality bidirectional chunks while caching predictions for incremental delivery. When a new condition arrives, the system replans only the necessary future, reusing the valid prefix and discarding only the affected suffix. This approach removes the waiting-time floor imposed by fixed chunk boundaries, allowing large-chunk models to achieve low-latency responsiveness comparable to fine-grained autoregressive systems.
Paper Primer
The core mechanism is a two-state finite-state machine that coordinates joint generation and incremental consumption. It treats the generation window as a replaceable buffer: the system produces a large chunk, caches the predictions, and commits them to playback incrementally, keeping the uncommitted future revocable until a new condition forces a branch.
Speculative Sampling eliminates the waiting-time lower bound imposed by chunk boundaries.
Latency analysis shows that decoupling joint generation from output commitment shifts the optimization focus from chunk size to per-chunk generation time. In end-to-end serving, the method reduces response latency by up to 2.8 seconds in long-chunk configurations compared to conventional chunk-level commitment.
Large-chunk models remain responsive while significantly improving GPU efficiency.
Across Wan1.3B and Sol-H3 models, long-chunk configurations reduced per-GPU activity to 17–23% (compared to 50% for short-chunk baselines) while maintaining L50 response times within 0.4 seconds of the short-chunk baseline. The method allows large-chunk models to outperform short-chunk baselines in both latency and resource utilization.
Why does this approach outperform the obvious alternative of simply using smaller chunks?
Smaller chunks limit joint temporal modeling and reduce GPU parallel efficiency. Speculative Sampling retains the modeling power of large bidirectional windows while achieving fine-grained delivery, avoiding the quality degradation and compute overhead associated with frame-wise autoregressive generation.
What is the primary scope of this method?
It is designed for interactive streaming systems using latent diffusion models. It is most effective when condition updates are infrequent enough that the cost of replacing a suffix is lower than the overhead of continuous, fine-grained autoregressive sampling.
Researchers and engineers can now prioritize accelerating high-quality joint generation rather than shrinking generation units, effectively decoupling visual quality from interactive responsiveness.
The Latency-Quality Tradeoff
We expose why small chunks hinder GPU use and introduce Speculative Sampling to decouple chunk size from latency.
Interactive video generation currently uses small, fixed-size chunks to keep response latency low, but this restricts GPU parallelism and harms video quality. Preliminary experiments show that larger chunks boost quality and better utilize GPUs with only modest extra sampling time. This motivates Speculative Sampling, which expands the modeling window while keeping low‑latency interaction.
In interactive settings, new conditions must affect playback promptly; a fast model alone cannot guarantee low latency if output can only be committed at chunk boundaries. Fine‑grained autoregressive generation offers frequent updates but limits joint temporal modeling and parallel efficiency. The central question is whether low‑latency interaction must sacrifice a larger bidirectional modeling window.
**Figure 1.** Fine-grained interaction without shrinking generation chunks. Our method preserves large-chunk joint sampling while allowing updates beyond the protected prefix, rather than only at chunk boundaries. Gray/green denote old/new conditions; the dashed line marks playback position at condition arrival, and thick green edges mark updated output. Positions represent video content, not computation time.
Optimizing per‑chunk generation speed, not chunk size, is the path to lower latency.
Speculative Sampling Mechanism
We decouple video generation from output commitment using a finite-state controller and speculative prediction caching.
Interactive video generation requires balancing low latency with high-quality temporal modeling. Traditional systems commit to fixed-size chunks, which forces a trade-off: smaller chunks reduce wait times but limit the model's ability to leverage long-range temporal dependencies.
Instead of waiting for a full chunk to be generated before committing, we generate large chunks speculatively and cache them. A controller allows us to override these predictions mid-chunk if a new condition arrives, providing fine-grained responsiveness without sacrificing the quality of large-window temporal modeling.
The system generates $(L_1, L_2, L_3, L_4)$ and delivers $L_1$ to the player.
Latents $(L_2, L_3, L_4)$ are stored in the prediction cache.
If the condition $c$ remains unchanged, the system enters the Consume state and delivers $L_2$ from the cache without a new DiT sampling call.
If a new condition $c'$ arrives before $L_2$ is delivered, the FSM triggers a new Produce state, discarding the cached $(L_2, L_3, L_4)$ and generating a new chunk starting from $L_1$.
Speculative sampling allows the model to maintain a large temporal window ($K=4$) while providing the responsiveness of a 1-latent delivery ($q=1$).
The FSM is the control logic governing when to commit or override speculative video frames, ensuring the system only performs expensive DiT sampling when necessary.
**Figure 3.** Speculative Sampling serving system and control. (a) Joint generation populates a prediction cache for fine-grained delivery and playback; playback progress guides refill. (b) The finite-state machine coordinates generation in Produce and prediction reuse in Consume, illustrated for $q = 1$. Condition updates trigger feasible replanning without interrupting active generation; playback continues in both states.
To support bidirectional models, we adapt the continuation interface by decoding the retained boundary to pixels and re-encoding it through the VAE's first-frame path. This ensures the latent format remains compatible with the pretrained model's expectations during successive sampling calls.
Empirical Evaluation
Experiments quantify quality‑efficiency tradeoffs and show Speculative Sampling’s latency gains.
Speculative Sampling raises VBench total scores by up to +2.6 points compared with the baseline.
Table 1 shows Wan1.3B total increasing from 79.00 (K=1) to 81.66 (K=3) and up to 82.13 (K=21).
Increasing chunk size generally improves visual quality but also raises per‑frame latency because more latent frames must be generated before the next update.
How does Speculative Sampling keep latency low while using larger chunks?
The method allows mid‑chunk updates to overwrite speculative frames, so the system can react to new conditions without waiting for the full chunk to finish generating.
**Figure 5.** Chunk size exposes a quality–efficiency tradeoff across models. (a) Wan four-step VBench totals at two resolutions. (b) Wan H200 forward latency across USP sizes at 480p. Quality uses separately trained configurations; timings use a shared reference checkpoint and synthetic fixed-length history. LTX quality and cached-forward profiles are reported in Table 2.
Limitations and Future Directions
We cap the gains of Speculative Sampling and expose its remaining constraints.
Interactive video generation traditionally uses small, fixed‑size chunks to keep latency low. Speculative Sampling instead lets us generate larger chunks and still meet interactive timing by allowing mid‑chunk updates to override earlier predictions.
Empirically, a chunk of $K\!=\!7$ runs in 78.6 ms on USP1, only marginally slower than $K\!=\!1$ (77.5 ms), while a 720p chunk of $K\!=\!21$ drops from 6432.8 ms to 899.7 ms. Table 2 (a) and (b) detail these forward timings and visual quality scores across resolutions, confirming that larger chunks can be efficient when parallelism is exploited.
Online serving measurements (Table 3) compare a short baseline, a long baseline, and their Speculative Sampling variants. Across all three models, the long‑setting with Speculative Sampling reduces the 50 % latency (L50) to within 0.26–0.39 s of the short setting while cutting per‑GPU activity by 20–25 %.
Section 4.3 shows that the observed latency reductions follow the wait‑time model $E[R]/T = (C/T)^2/2$, which predicts larger chunks leave more boundary wait to remove. The measured gains (e.g., 0.493 s vs. theoretical 0.438 s for Sol‑H3) match this trend closely.
When updates arrive continuously, most of each sampled chunk is discarded before reuse, wasting DiT computation; additionally, retained key‑value pairs may still influence the regenerated suffix even after the offending segment is dropped.
These two failure modes limit the method’s scalability: frequent condition changes prevent reuse of predictions, and bidirectional within‑chunk attention can leak information from a suffix that is later removed, reducing the effectiveness of revoking predictions.
In conclusion, we cast interactive video generation as a joint prediction‑commitment problem. Our approach reuses completed predictions under unchanged conditions, revokes only the future that can still be replaced, and thereby shifts design effort from shrinking generation units toward accelerating joint generation and scheduling timely delivery.
Latency Model Derivations
We formalize latency components and derive scaling formulas for fixed and residual generation.
This appendix derives the latency model used throughout the paper and presents closed‑form expressions for fixed‑chunk and residual‑generation scenarios.
For uniform condition‑arrival phase modulo $T$, the expected latency satisfies $E[L] \ge C + \tfrac{T}{2}$, and this bound is attainable under ideal scheduling.
The model expresses total response latency $L$ as the sum of three components: the fixed generation time $C$, the average waiting for the next playback boundary ($T/2$ under uniform arrivals), and the residual wait $R$ that occurs when a new condition arrives while the generator is busy.
A.2 LATENT‑BOUNDARY ALIGNMENT considers the case where the prefix is instantly available and generation still takes $C$ seconds; the earliest display time aligns to the nearest multiple of a finer granularity $\delta$, yielding $E[L]=C+\delta/2$ under ideal conditions.
A.3 RESIDUAL GENERATION UNDER AN ISOLATED UPDATE extends the baseline by accounting for the probability $C/T$ of arriving during a busy period and the uniform residual duration, leading to the expectation $E[R]=C^{2}/(2T)$ and an unconditional average latency $E[L]=C+E[R]+E[Q]$.
B PREDICTION‑USE ACCOUNTING quantifies how many DiT sampling calls are required per displayed latent under different usage patterns.
**Figure 2.** Generation speed alone does not determine response latency. In this periodic reference schedule, the highlighted update waits $R$ for generation, takes $C$ to produce deliverable output, and waits $Q$ for playback. Each chunk is prepared for the next playback period; $V_0$ is prebuffered.
**Figure 4.** At fixed $T$, direct generation scales as $C/T$, while residual wait scales as $E[R]/T = (C/T)^2/2$ under the stated periodic model.
Extended FSM Specification
Formal specification of the extended finite‑state controller for speculative video generation.
This section formalizes the extended finite‑state controller introduced in Section 3.2, describing its states, registers, events, and safety guards.
The controller is a compact state machine that decides when to keep producing speculative video frames and when to consume committed frames, using a rich register set to track predictions, conditions, and deadlines.
Launches atomically lock the retained prefix before $j$, invalidate only replaceable suffixes, record a new job $J$ for a full‑$K$‑latent DiT sampling call with the latest condition, and clear the consumed pending version $u$.
Required invariants: no delivered or locked prefix is overwritten; at most one active job occupies the slot; launches occur only when the guard $g$ is true; playback and cache positions stay consistent after each branch.
Implementation Details and Logs
Details of logging, conditioning adapters, and future co‑optimization directions for video generation.
The event log records predicted latents, $VAE$ decoder invalidations, missed deadlines, superseded updates, and extra work. These entries enable precise accounting of runtime estimates and failure‑handling policies.
Generation is the compute needed to produce deliverable video; for latent diffusion models $C$ comprises conditioning preparation, $DiT$ sampling, and the required $VAE$ decoding for the first output group. Overlapping stages are not simply summed because they can execute concurrently.
$R$ may include scheduling, resource, and conditioning‑input readiness waits, while $Q$ may cover transport, buffering, and playback alignment. User end‑to‑end latency also adds any pre‑arrival processing.
Each event log must contain identifiers and timestamps for condition‑ready, prefix‑ready, generation start, KV‑ready, $DiT$ sampling completion, $VAE$ decoding completion, first deliverable output, target switch, actual display, and final outcome. Generation logs additionally record call ID, absolute positions, used and discarded predictions, runtime, GPU count, and any canceled or repeated work.
The controller operates on retained content and playback deadlines, using adapters that expose continuation without relying on full‑prefix KV reuse. Different adapters affect how retained‑prefix KV, boundary‑image continuation, and latent‑input paths are handled.
Retained‑prefix KV reuse reuses the selected prefix’s inference state at correct positions, invalidating direct suffix entries. Alternative cache policies, such as incremental clean prefill, must be distinguished in ablations.
The boundary‑image path retrieves the full‑resolution RGB frame preceding the selected branch and feeds it to the image‑conditioning interface with the latest prompt. This avoids a new temporal causal mask and uses the H3 image‑conditioned path.
The H3 latent‑input path replaces image encoding with a normalized clean video latent while preserving condition packing and noise handling. A multi‑frame path supplies a phase‑aligned group of retained latents as keyframe conditions, removing the corresponding output overlap.
The output adapter maps only the remaining pixels to the new playback interval, preserving source indices for subsequent anchors and never selecting history from discarded predictions. Non‑uniform latent‑to‑frame spans can produce fewer playable groups than sampled latents.
Streaming $VAE$ decoding can publish finalized groups before the entire clip finishes decoding, while $DiT$ sampling remains joint over the chunk. Adapters must handle $VAE$ overlap and state‑dependent suffix replacement before committing pixels.
Future work aims to co‑design $DiT$ sampling and $VAE$ decoding as a single pipeline to reduce the time to the first deliverable output group. Candidate schedules will be evaluated on first‑group latency, later‑group deadlines, total GPU cost, and playback stalls.