Vidu S1: A Real-Time Interactive Video Generation Model
Jintao Zhang, Kai Jiang, Jintao Chen, Xu Wang, Yang Luo, Yuji Wang, Dechuang Chen, Jungang Li, Chengyang Ye, Marco Chen, Hongzhou Zhu, Min Zhao, Yuxuan Jiang, Zhengkun Huang, Chendong Xiang, Kaiwen Zheng, Haoxu Wang, Xiaohang Wang, Qi Jia, Xin Chen, Yimin Chen, Youhe Jiang, Fangcheng Fu, Zhijie Deng, Fan Bao, Jianfei Chen, Jun Zhu
Vidu S1 enables real-time, voice-controlled, infinite-length video generation at 42 FPS on consumer GPUs.
How does Vidu S1 enable real-time, voice-controlled, infinite-length video generation for digital characters?
Most video generation models operate in an offline, one-shot paradigm where users wait minutes for a complete clip, making them unsuitable for live interactions like virtual hosts or gaming. Vidu S1 treats speech as an explicit control signal and uses a sliding-window autoregressive architecture to generate video streams that respond to user input in real time. On consumer hardware, the system produces 540p video at 42 FPS while maintaining identity consistency and temporal stability over indefinite durations.
Paper Primer
The core mechanism hinges on a three-stage training pipeline that adapts a bidirectional teacher model into a causal, streaming generator. By combining Distribution Matching Distillation (DMD) with Phased Consistency Models (PCM), the model compresses the autoregressive denoising process into a few steps, enabling low-latency inference without the quality collapse typical of long-horizon generation.
To maintain stability during streaming, the model uses TwinCache: a stage-aware caching strategy that maintains separate noisy and clean historical states. The noisy cache provides coarse temporal guidance to prevent drift, while the clean cache is used at the final denoising step to restore fine-grained visual details and identity consistency.
Vidu S1 achieves state-of-the-art performance in identity preservation and audio-visual synchronization while meeting real-time requirements.
Quantitative evaluation on the HDTF benchmark shows a CSIM of 0.9192 and Sync-D of 7.847. The model maintains 42 FPS at 540p resolution on RTX 5090 GPUs, exceeding the 30 FPS real-time playback threshold.
Why is this approach more effective for interactive applications than simply speeding up existing offline models?
Offline models denoise entire videos synchronously, which prevents real-time user intervention. Vidu S1 uses a causal autoregressive design that allows users to provide speech instructions at any moment, which the model then uses as an explicit control signal for the next frames.
How does the model avoid the visual drift that typically plagues long-horizon video generation?
The model uses a persistent reference context—derived from the initial frame—as a global anchor, combined with TwinCache to decouple temporal propagation from appearance refinement, effectively suppressing the accumulation of artifacts over time.
Vidu S1 demonstrates that high-fidelity, controllable video generation can move from batch-processed offline tasks to low-latency, interactive streaming, provided the inference stack is co-designed with the model's autoregressive constraints.
Introduction to Vidu S1
We frame the need for real‑time, voice‑controlled video generation and its challenges.
Current video generation systems operate offline: a user submits a prompt, waits minutes, and receives a completed video. This paradigm cannot satisfy the growing demand for immediate, voice‑driven interaction, where content must respond instantly to spoken instructions.
Real‑time, voice‑controlled video generation is needed because users expect instantaneous visual feedback, and the aggregate demand for such interactive content far exceeds that of offline‑generated videos.
**Figure 1.** Overview of Vidu S1. Vidu S1 supports speech-guided future control, efficient infinite-duration inference, and diverse customized characters for real-time interactive video generation.
Vidu S1 makes real‑time voice‑controlled video generation practical on consumer GPUs.
Data Processing Pipeline
We build a staged pipeline that turns raw videos into high‑quality, annotated clips for training.
Figure 2 visualizes the multi‑stage data pipeline that converts raw videos into training clips.
**Figure 2** Data filtering pipeline for raw videos, showing progressive refinement through prefiltering, single-shot clipping, subject filtering, quality and safety checks, diarization, and caption generation. The final selected clips and their captions are embedded to form training data.
The pipeline refines raw footage step by step, discarding low‑quality or unsafe material early so that later stages operate on already‑clean data.
Prefilter: 10 → 8 videos (removed low‑FPS / resolution failures).
Clipping: 8 videos → 12 clips (average 1.5 clips per video).
Subject filter: 12 → 10 clips (removed clips with multiple subjects).
Quality & safety: 10 → 9 clips (dropped blurry or NSFW content).
Diarization: 9 → 8 clips (filtered out the single overlapping segment).
Captioning: the remaining 8 clips receive both clip‑level and chunk‑level captions.
By discarding problematic material early, each subsequent stage works on a smaller, higher‑quality set, dramatically reducing annotation cost and error propagation.
How does this pipeline differ from a conventional one‑time data curation process?
Instead of a single pass that filters everything at once, the progressive pipeline interleaves cheap technical filters with increasingly expensive semantic models, allowing early stages to prune obvious failures and reserve costly computation for the already‑clean subset.
This pipeline supplies the clean, annotated data that enables streaming inference later.
Training Methodology
We train a streaming video‑audio model by first learning a full‑sequence teacher, then adapting it to causal streaming, and finally distilling it with distribution‑matching and PCM regularization.
Training a streaming video‑audio generator must balance two competing demands: high visual fidelity and low latency. Naïve diffusion models require dozens of denoising steps per frame, which makes real‑time generation infeasible.
First we teach a bidirectional model on full video‑audio sequences, then we convert it into a causal model that can run autoregressively, and finally we compress the autoregressive steps with distribution‑matching distillation and PCM regularization.
Step 1: Sample Bernoulli → outcome “teacher forcing”. The prefix for frame 2 is the clean ground‑truth state $x_{1}^{0}$.
Step 2: Compute causal loss for frame 2 using Equation 5 with $\tau_j=0$. The model receives a stable, noise‑free context.
Step 3: Sample Bernoulli → outcome “diffusion forcing”. The prefix becomes a noisy version $x_{1}^{\tau_j}$ with $\tau_j=0.3$.
Step 4: Compute causal loss for frame 2 again, now with a noisy prefix. The gradient encourages robustness to imperfect histories.
Step 5: Average the two loss terms (teacher‑forcing + diffusion‑forcing) to obtain the final loss for this training step.
Hybrid training forces the model to learn both exact conditioning (for synchronization) and noisy conditioning (for robustness), which is essential for stable streaming inference.
How does this hybrid “teacher‑forcing / diffusion‑forcing” schedule differ from the standard teacher‑forcing used in sequence models?
Standard teacher‑forcing always feeds the ground‑truth previous token, eliminating exposure bias but never exposing the model to its own prediction errors. The hybrid schedule occasionally replaces the clean prefix with a noisy version, so the model learns to correct its own mistakes and remains stable when the autoregressive rollout inevitably deviates from the clean trajectory.
By chaining the bidirectional prior, causal adaptation, and DMD‑PCM distillation, the final model can generate high‑quality video‑audio streams with only a few denoising steps per frame, satisfying the real‑time constraint.
Streaming Inference Architecture
Enables constant‑time video‑audio generation by limiting attention to a moving window.
Streaming inference must generate arbitrarily long video‑audio sequences while keeping per‑step latency constant despite limited GPU memory.
Instead of recomputing attention over the entire past, the model keeps a fixed‑size sliding window that moves forward each step, so each step only sees a bounded amount of context.
Step 1: Window = $\{P, H_0, C_1\}$ – attention is computed over these three tokens.
Step 2: Slide forward – $H_0$ is dropped, new historical token $H_1$ (the denoised $C_1$) is added: window = $\{P, H_1, C_2\}$.
Step 3: Repeat sliding for $i=3,\dots$ – the window always stays size 3, keeping memory constant.
The sliding window guarantees $O(1)$ memory growth while still providing the model with a stable global reference via $P$.
How does Streaming Inference differ from standard autoregressive decoding that recomputes attention over all past tokens?
Standard decoding expands the attention matrix each step, causing latency and memory to grow linearly with sequence length. Streaming Inference fixes the window size, so each step reuses cached keys/values and only attends to a constant number of tokens, breaking the linear growth.
The decoder processes tokens in overlapping fixed‑length windows, each window covering the most recent frames while respecting causality.
Why not simply increase the window size indefinitely to capture more context?
Increasing the window size raises both memory consumption and per‑step latency, defeating the goal of real‑time streaming. Moreover, beyond a certain horizon the additional tokens contribute diminishing returns because the model already has a stable global reference via the persistent context.
Quantitative Performance
Vidu S1 delivers real‑time, high‑quality video generation via streaming sliding‑window inference.
Recall that Vidu S1 combines a streaming sliding‑window inference pipeline with a high‑quality training data stream to enable voice‑controlled video generation.
Vidu S1 achieves real‑time 42 FPS at 540p resolution, meeting latency targets for interactive video generation.
Table 1 reports 42 FPS for Vidu S1 on a 960×540 test, the highest among all evaluated models.
**Table 1.** Comparison of various avatar generation models across metrics including Instruction Following, Real-Time capability, Resolution, FPS/Throughput, CSIM, Sync-D, and DOVER.
HeyGen is an audio‑driven avatar generator that prioritizes photorealistic facial detail, delivering the highest CSIM score in the comparison.
Kling Avatar 2.0 balances visual quality with moderate latency, serving as a strong baseline in the evaluation.
To meet the real‑time latency budget, Vidu S1 distributes computation across multiple GPUs using Ulysses‑style context parallelism.
Evaluation and Qualitative Analysis
We assess Vidu S1’s real‑time performance and user‑controlled avatar quality.
We evaluate Vidu S1 on the in‑house Vidu‑StreamBench benchmark and on the public HDTF suite, measuring both perceptual quality and real‑time inference speed.
**Figure 3.** Human preference evaluation of Vidu S1 versus HeyGen, LemonSlice, and Kling-Avatar-2.0 on Vidu-StreamBench.
**Figure 4** Qualitative comparison of instruction following and visual consistency. Red boxes highlight failure cases from the compared method: in Prompt 1, the subject does not raise the head and noticeable background artifacts appear; in Prompt 2, the thumbs-up gesture is not correctly performed. Green boxes highlight the results of Vidu S1: in Prompt 1, the model correctly follows the instruction by raising the head and adopting a thoughtful pose; in Prompt 2, it successfully performs the thumbs-up gesture.
Vidu S1 runs at 42 FPS on an RTX 5090 GPU, surpassing the 30 FPS real‑time threshold.
Measured throughput of 42 FPS while generating 540p video with a 3‑step configuration.
Overall, Vidu S1 delivers the strongest perceptual scores while meeting real‑time constraints, positioning it as a practical solution for interactive avatar applications.
Questions & answers
What is the main contribution of Vidu S1?
Vidu S1 introduces a real-time interactive video generation system that treats speech as an explicit control signal and uses a sliding-window autoregressive architecture, enabling voice-controlled video streams at 540p and 42 FPS on consumer GPUs.
What problem does Vidu S1 address?
Vidu S1 addresses the limitation of offline, one-shot video generation models that require users to wait minutes for a completed clip, making them unsuitable for live interactive applications such as virtual hosts or gaming.
Why is a causal autoregressive design better for interactive applications than speeding up offline models?
Offline models denoise entire videos synchronously, which prevents real-time user intervention. Vidu S1's causal autoregressive design allows users to provide speech instructions at any moment, which the model uses as an explicit control signal for the next frames.
How does Vidu S1's three-stage training pipeline work?
The three-stage pipeline adapts a bidirectional teacher model into a causal, streaming generator by combining Distribution Matching Distillation (DMD) with Phased Consistency Models (PCM), compressing the autoregressive denoising process into a few steps to enable low-latency inference.
What is TwinCache and how does it work?
TwinCache is a stage-aware caching strategy that maintains two separate historical states: a noisy cache that provides coarse temporal guidance to prevent drift, and a clean cache used at the final denoising step to restore fine-grained visual details and identity consistency.
How does Vidu S1 prevent visual drift over long video generation?
The model uses a persistent reference context derived from the initial frame as a global anchor, combined with TwinCache to decouple temporal propagation from appearance refinement, suppressing the accumulation of artifacts over time.
How does Vidu S1's streaming inference differ from standard autoregressive decoding?
Standard autoregressive decoding expands the attention matrix each step, causing latency and memory to grow linearly with sequence length. Vidu S1's streaming inference fixes the window size so each step reuses cached keys/values and attends to a constant number of tokens, breaking that linear growth.
Why does Vidu S1 not simply use a larger sliding window to capture more context?
Increasing the window size raises both memory consumption and per-step latency, defeating the real-time streaming goal. Beyond a certain horizon, additional tokens contribute diminishing returns because the model already has a stable global reference via the persistent context.
What is the hybrid teacher-forcing/diffusion-forcing training schedule?
Unlike standard teacher-forcing, which always feeds the ground-truth previous token, the hybrid schedule occasionally replaces the clean prefix with a noisy version, training the model to correct its own mistakes and remain stable when autoregressive rollout deviates from the clean trajectory.
How does Vidu S1's data processing pipeline work?
The pipeline uses a multi-stage progressive approach that interleaves cheap technical filters with increasingly expensive semantic models, allowing early stages to prune obvious failures and reserving costly computation for the already-clean subset, rather than applying a single-pass curation.
What benchmarks and datasets are used to evaluate Vidu S1?
Vidu S1 is evaluated on the in-house Vidu-StreamBench benchmark and the public HDTF suite, measuring both perceptual quality and real-time inference speed.
What are the key quantitative performance results of Vidu S1?
Vidu S1 produces 540p video at 42 FPS on consumer hardware and delivers the strongest perceptual scores on Vidu-StreamBench and HDTF while meeting real-time constraints. The paper does not report additional specific numeric metric values beyond these figures.
How does Vidu S1 handle the computational demands of real-time inference?
To meet the real-time latency budget, Vidu S1 distributes computation across multiple GPUs using Ulysses-style context parallelism.
What are the limitations of Vidu S1 as acknowledged in the paper?
The paper does not explicitly enumerate limitations, though it notes that increasing the sliding window size trades off memory and latency, and that the approach is co-designed with specific autoregressive constraints that may not generalize to other architectures.
How does Vidu S1 differ from prior offline video generation approaches?
Prior offline models generate complete video clips in a single batch pass with no real-time interactivity, whereas Vidu S1 uses a causal sliding-window autoregressive design with speech as an explicit control signal, enabling continuous streaming and mid-generation user intervention.
What practical applications does Vidu S1 target?
The paper positions Vidu S1 as a practical solution for interactive avatar applications, including virtual hosts and gaming scenarios that require immediate, voice-driven video responses.
Where and when was Vidu S1 published?
Vidu S1 is available as an arXiv preprint at arxiv.org/abs/2607.03118. The paper does not specify a conference or journal venue, and the paper does not list author names in the provided text.
Key terms
- Vidu S1
- A real-time interactive video generation model that uses speech as a control signal and a sliding-window autoregressive architecture to stream video at 42 FPS on consumer hardware.
- sliding-window autoregressive architecture
- A generation design where the model attends only to a fixed-size window of recent tokens at each step, keeping memory and latency constant regardless of total sequence length.
- Distribution Matching Distillation (DMD)
- A training technique that compresses a multi-step diffusion model into fewer denoising steps by matching the output distribution of the teacher model, reducing inference cost.
- Phased Consistency Models (PCM)
- A consistency-based distillation method that further reduces the number of denoising steps required during inference by enforcing consistency across phases of the generation process.
- TwinCache
- A stage-aware caching strategy in Vidu S1 that maintains separate noisy and clean historical state caches to provide temporal guidance and preserve visual detail during streaming generation.
- causal autoregressive design
- A model architecture where each output token depends only on previously generated tokens, enabling sequential, real-time generation rather than processing the entire sequence at once.
- teacher-forcing
- A training technique for sequence models where the ground-truth previous token is always fed as input, which speeds up training but can cause instability when the model generates its own outputs at inference time.
- diffusion-forcing
- A training variant that occasionally replaces clean context tokens with noisy versions, teaching the model to recover from its own prediction errors during autoregressive rollout.
- persistent reference context
- A fixed representation derived from the initial video frame that serves as a global anchor throughout streaming generation to maintain identity and appearance consistency.
- Ulysses-style context parallelism
- A distributed inference strategy that partitions the attention context across multiple GPUs to reduce per-device memory and latency during long-sequence generation.
- Vidu-StreamBench
- An in-house benchmark developed for evaluating Vidu S1 on perceptual quality and real-time inference speed in streaming video generation scenarios.
- HDTF
- A public high-definition talking-face video dataset used as an external benchmark to evaluate perceptual quality in Vidu S1's experiments.
- exposure bias
- The discrepancy in sequence models between training (where ground-truth tokens are fed as input) and inference (where the model's own predictions are fed), which can cause error accumulation during generation.
- bidirectional teacher model
- A video generation model that can attend to both past and future context during training, used as the high-quality starting point before being adapted into a causal streaming generator.