Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation
Jintao Zhang, Kai Jiang, Jintao Chen, Xu Wang, Deyuan Liu, Jungang Li, Dechuang Chen, Ming Lin, Jingjiang Zhou, Haopeng Jin, Qi Jia, Xiaohang Wang, Yaole Wang, Zhanqiang Zhang, Ran Li, Zhengkun Huang, Shuyue Xiong, Yuji Wang, Zikun Dai, Hui He, Yang Luo, Mang Ning, Weiqi Feng, Chengyang Ye, Xinyue Lin, Min Zhao, Hongzhou Zhu, Hengkai Tan, Zeyuan Wang, Chendong Xiang, Kaiwen Zheng, Zhijie Deng, Fan Bao, Jianfei Chen, Jun Zhu
Vidu S2 enables real-time, interactive 720p video generation and streaming video editing with spatial support.
How can we achieve real-time, interactive, and editable video generation for digital avatars and spatial video?
Most video generation models operate offline, forcing users to wait minutes for a single clip and preventing any active interaction during the process. Vidu S2 introduces a streaming architecture that uses Self-Replay Forcing to align autoregressive rollouts with teacher distributions, enabling real-time, instruction-following video generation and editing. The system supports 720p output at up to 42 FPS and can convert monocular streams into synchronized spatial video for immersive VR experiences.
Paper Primer
The core challenge in real-time video generation is preventing the "drift" or visual collapse that occurs when a model must generate infinite-length streams segment-by-segment. Vidu S2-Avatar solves this with Self-Replay Forcing: after an initial autoregressive rollout, the model re-noises the entire trajectory and performs a second, gradient-enabled causal pass that allows the loss from later segments to supervise preceding ones without backpropagating through the original, detached rollout.
For real-time editing, Vidu S2-Editing uses frame-aligned attention: every target frame reads only the source frame at the same time step, ensuring the edited output preserves the exact motion and timing of the input while applying style, clothing, or background changes.
Vidu S2-Avatar achieves 720p real-time generation at 25–42 FPS.
The system utilizes a lightweight Refiner that operates in latent space to recover fine-grained details, allowing the backbone to maintain high throughput at lower resolutions. This represents a resolution increase from the 540p limit of the previous Vidu S1 model while maintaining real-time performance on consumer GPUs.
Why does the model need a separate "Refiner" instead of just generating at 720p directly?
Generating high-resolution latents directly in the backbone is computationally expensive and would drop the frame rate below real-time requirements. The Refiner uses asymmetric noise levels to recover spatial details from a high-resolution cache while the backbone focuses on coarse motion propagation.
How does the system handle user requests to change objects or backgrounds mid-stream?
A vision-language model (VLM) agent monitors the stream and updates the generation prompts dynamically. It explicitly preserves existing character states (like holding an object) unless the user instruction specifically requests a change, ensuring continuity across updates.
Vidu S2 shifts the paradigm of video generation from static, offline batch processing to a continuous, agentic streaming service capable of real-time spatial interaction.
Introduction and System Overview
We frame the need for real‑time interactive video generation and introduce Vidu S2’s unified framework.
Current video generation models operate offline, requiring minutes of latency before any output appears, which prevents interactive experiences such as live streaming, gaming, or VR where users expect immediate visual feedback.
If each user’s average demand for interactive content is $\alpha$ (e.g., 0.5) and for offline content is $\beta$ (also ≈0.5), the total interactive demand scales as $\alpha \times N$, while offline demand scales as $\beta \times N / m$, where $ m $ (≈100) is the average view count per generated video. Thus interactive demand far exceeds offline demand.
Interactive visual content must be generated on the fly, with the ability to change references and instructions at any moment, so that the user’s experience remains continuous and responsive.
**Figure.** Overview of Vidu S2-Avatar and Vidu S2-Editing capabilities. The top panel illustrates the real-time generation of a spatial video avatar, while the bottom panel demonstrates various real-time editing tasks including style, clothing, subject, and background modifications.
Vidu S2 delivers a unified real‑time generation and editing framework for spatial video avatars.
Avatar Data Pipeline
We refine the avatar data pipeline to ensure high‑quality, temporally consistent video for real‑time generation.
Training an avatar model demands video clips that are both visually clean and temporally precise; noisy cuts or unstable backgrounds quickly degrade real‑time generation quality.
Collect diverse sources (talking‑heads, solo‑dance, 2D/3D animation) to broaden expression coverage.
Apply two‑stage clip detection: coarse shot detection followed by Vision‑Language Model verification.
Filter clips using the six‑dimensional taxonomy plus a high‑clarity selection operator.
Stabilize backgrounds of motion‑rich clips via geometric correction and cropping.
Generate temporally ordered dense captions that encode event boundaries and causal flow.
Embed the processed clips for downstream avatar training.
First flag candidate edit points, then let a Vision‑Language Model (VLM) confirm true cuts, keeping false‑rejection below 2 % while discarding spurious detections.
Extract 0.5 s windows around each candidate (e.g., frames 20‑30 for 2.1 s).
VLM scores continuity: 0.92, 0.31, 0.88 (higher means more continuous).
Retain only the middle candidate (score 0.31) as a true cut; discard the others.
The VLM acts as a semantic filter, turning raw motion cues into reliable edit decisions.
How does this two‑stage approach differ from simply lowering the shot‑detection threshold?
Lowering the threshold would flood the pipeline with many false cuts, overwhelming downstream processing. The VLM verification step explicitly checks visual continuity, so only cuts that truly separate distinct scenes survive, preserving a low false‑rejection rate.
We evaluate resolution, frame rate, codec, pixel format, bit depth, and bitrate together, then aggregate expert‑model scores into a weighted quality metric to keep only videos that meet a strict clarity threshold.
Hard filter keeps all four (all meet ≥ 720p and ≥ 30 fps).
Expert models assign texture scores: A 0.78, B 0.85, C 0.62, D 0.90.
Weighted quality scores (after combining artifact and edge metrics): A 0.81, B 0.88, C 0.55, D 0.92.
Apply clarity cutoff of 0.80 → select B and D; discard A and C.
The weighted score captures subtle compression artifacts that raw bitrate alone would miss.
Why not rely solely on a single resolution threshold?
Resolution alone ignores compression artifacts and color depth; a 1080p video with heavy compression can look worse than a clean 720p clip. The hybrid score ensures that texture fidelity and artifact severity are also considered.
For videos with modest camera motion, we estimate the background’s geometric transform and warp frames to a common static view, preserving foreground motion while eliminating background drift.
Mask the dancer (foreground) leaving background pixels.
Match background features and compute a translation vector ($\Delta$x = +3 px).
Warp each frame by –$\Delta$x to cancel the shift, aligning backgrounds.
Crop to the common intersection region, preserving the dancer’s motion.
Even a tiny translation can cause noticeable drift over long sequences; correcting it early stabilizes the entire clip.
What happens if a clip contains rapid, complex camera moves?
Such clips are excluded before stabilization because the homography model cannot capture large parallax; retaining them would introduce background jitter that harms the model’s ability to learn consistent scene structure.
Captions are written as a chronological list of events with explicit start‑end times, making temporal dependencies explicit for the model.
**Figure 2.** Overview of the data preparation pipelines for Vidu S2-Avatar and Vidu S2-Editing. Left: diverse video sources are processed through clipping, filtering, speech processing, temporal dense captioning, and embedding, with enhanced operators for cut-point detection, high-clarity selection, and background stabilization. Right: filtered Avatar data are further curated for four editing tasks; style-transfer pairs are generated through reconstruction-based V2V training and reference-guided synthesis, followed by post-filtering and comparative selection.
Audio-Visual Diffusion Transformer
The method introduces causal blockwise attention and self‑replay forcing to enable streaming avatar video generation.
Streaming avatar video generation suffers from error accumulation because each segment depends on the previous noisy output. The paper resolves this by converting the bidirectional diffusion transformer into a causal, block‑wise architecture and by training with on‑policy self‑replay. These changes give the model an initial streaming capability while keeping long‑range consistency.
It is a joint audio‑visual diffusion transformer that takes a reference image and a sequence of textual conditions to generate synchronized video‑audio segments.
How does Vidu S2‑Avatar differ from a standard diffusion model that conditions on a single prompt?
Standard diffusion models treat the whole video as one sequence with one global prompt, which forces the model to infer all actions from a single text. Vidu S2‑Avatar instead supplies a separate caption $c_i$ for each segment, letting the model follow fine‑grained instructions while still sharing the reference image $r$ for identity.
It jointly diffuses video and audio latent streams, allowing the two modalities to influence each other during generation.
Instead of letting every segment attend to the entire future, each segment can only look at the reference image, its own condition, and the already‑generated history, which makes streaming possible.
Segment 1 receives clean history (empty) and predicts $\hat{x}_1$ using only $r$ and $c_1$.
Segment 2 receives $x_{<2}^{\tau_2}$ = noisy version of $\hat{x}_1$ (e.g., added Gaussian noise $\sigma=0.1$) and predicts $\hat{x}_2$.
Segment 3 receives $x_{<3}^{\tau_3}$ = noisy versions of $\hat{x}_1$ and $\hat{x}_2$, then predicts $\hat{x}_3$.
The causal mask forces each segment to rely on progressively noisier history, teaching the model to correct its own errors during streaming.
Why can’t we simply keep the original bidirectional attention and run it segment by segment?
Bidirectional attention requires future segments to be present, which is impossible at inference time. The causal blockwise mask removes that requirement while still allowing each segment to attend to all past information, enabling true autoregressive generation.
SRF lets the model learn from its own streaming rollout by replaying the generated trajectory with added diffusion noise, while keeping gradients flowing across block boundaries.
Detach the rollout: gradients stop at $x_1$, $x_2$.
Add Gaussian noise $\epsilon_1$, $\epsilon_2$ to obtain $\tilde{x}_1 = x_1 + \epsilon_1$, $\tilde{x}_2 = x_2 + \epsilon_2$.
Run the causal network on $(\tilde{x}_1,\tilde{x}_2)$ while keeping the KV caches from the detached rollout fixed.
Compute $L_{\text{DMD}}$ between the network output and the clean targets, and add the perceptual loss $L_{\text{perc}}$.
Detaching the rollout prevents the memory cost of back‑propagating through the entire generated sequence, yet the replay pass still teaches the model to correct its own noisy predictions.
Why not simply train with teacher forcing on clean history all the time?
Pure teacher forcing never exposes the model to its own prediction errors, so at inference the accumulated noise would degrade quality. SRF injects realistic noise into the history, forcing the model to learn to recover from its own imperfect outputs.
The Refiner upsamples low‑resolution latent video frames and adds a small amount of noise to produce high‑resolution outputs, using a low‑noise high‑resolution cache for fine details.
Apply $U$: $4\times4 \to 8\times8$ latent.
Sample noise $\epsilon_i$ (e.g., standard normal) and compute $(1-t_r)=0.8$.
Combine: $\hat{x}_i^{0,LR} = 0.8 \times (\text{upsampled latent}) + 0.2 \times \epsilon_i$.
The resulting $8\times8$ latent is fed to the decoder, yielding a high‑resolution video frame.
Even a single diffusion step ($t_r=0.2$) is enough to inject fine‑grained stochastic detail while preserving the upsampled structure.
Why does the Refiner use a separate low‑noise cache instead of the same cache as the backbone?
The backbone’s high‑noise cache is optimized for propagating coarse motion over long horizons, which would blur fine spatial details. A low‑noise cache gives the Refiner a clean high‑resolution reference, allowing it to restore texture without destabilizing the temporal prior.
Inference Acceleration and Agentic Control
Fast inference combines TurboDiffusion tricks with a layered attention and an agentic pipeline for avatars.
Real‑time avatar generation stalls on attention and linear layers, which dominate latency. By importing TurboDiffusion tricks and reorganizing execution, we cut both compute and memory overhead while keeping video quality.
TurboDiffusion schedules module execution on a shared timeline so that different parts of the pipeline occupy the GPU at non‑overlapping moments, turning many short kernels into a continuously busy stream.
Time 0 ms: RMSNorm runs (2 ms).
Time 2 ms: SparseAttention runs (5 ms).
Time 7 ms: Linear‑GEMM runs (3 ms).
Total wall‑clock time = 10 ms, whereas running them sequentially without overlap would sum to 10 ms + launch overhead.
Scheduling removes idle gaps between kernels, so the GPU stays busy and launch latency becomes negligible.
Why does re‑ordering modules improve latency if the total compute stays the same?
Latency is dominated by kernel launch and synchronization costs. By arranging modules so that each launch follows the previous one without waiting for memory transfers, we amortize those fixed costs over the whole step, turning many short launches into a single, continuously streamed execution.
Different diffusion layers tolerate approximation differently, so we assign the cheapest attention variant to insensitive layers and reserve exact attention for the critical ones.
How does the hybrid strategy avoid the quality drop that pure sparse attention would cause?
Only layers that the paper’s ablation shows to be insensitive to approximation receive the sparse variant. The remaining layers keep full‑resolution attention, so the overall generation retains the same PSNR as a fully dense model.
We replace BF16/FP16 matrix multiplies with a custom W8A8 GEMM that scales each block independently, limiting the impact of outlier values.
Why not simply quantize the whole matrix globally instead of per‑block?
Global quantization is dominated by a few extreme values, forcing the majority of the matrix into very low precision. Per‑block scaling isolates those extremes, preserving detail where it matters while still gaining the speed of 8‑bit arithmetic.
By merging adjacent operators into a single Triton/CUDA kernel and replaying a captured CUDA graph, we eliminate most host‑side launches and intermediate memory traffic.
Does kernel fusion change the numerical results of the fused operations?
No. The fused kernel computes the exact same arithmetic as the separate kernels; it only reduces the overhead of writing intermediate tensors to global memory.
We split the diffusion batch across GPUs, distributing activation memory while quantizing inter‑GPU tensors to keep communication cheap.
Why does quantizing the tensors exchanged between GPUs not degrade the final video quality?
The tensors exchanged are activation statistics that are later de‑quantized; the quantization error is absorbed by the diffusion model’s stochastic nature, and empirical tests show negligible impact on PSNR.
The VLM agent reads user instructions and images, writes detailed prompts for the avatar model, and iteratively refines them by inspecting generated frames.
How does the agent know which parts of the prompt to keep unchanged across steps?
The agent maintains a state dictionary of attributes (e.g., held object, clothing) that were explicitly mentioned in previous prompts and automatically copies them forward unless the user explicitly modifies them.
**Figure 3** Agentic System Pipeline. The VLM agent reads user instructions and images, writes prompts for Vidu S2-Avatar, and checks generated video frames to guide later prompts. The pipeline is schematic.
Real-Time Video Editing
The section details Vidu S2‑Editing’s frame‑aligned attention trick for real‑time video editing.
Real‑time video editing demands that each incoming frame be processed faster than a human can perceive a lag. Existing diffusion models incur high latency because they treat the whole clip as a batch, wasting GPU cycles on idle modules.
It treats each target frame as a token that attends only to its matching source frame and to the reference image, then streams causally with self‑replay forcing to keep temporal consistency — like a film editor who aligns each new cut with the original shot before applying the desired effect.
At diffusion step $t=0$, sample noisy latents $x_1,x_2,x_3$ for the three target frames.
Frame‑aligned attention computes, for each $i$, an intermediate representation $h_i$ by attending $x_i$ to its matching source token $s_i$ and the global reference token $r$.
Self‑Replay Forcing: after a forward rollout producing $h_1,h_2,h_3$, the model re‑noises these intermediates and predicts again, receiving DMD supervision that aligns the replayed trajectory with the original rollout.
The final denoised latents $\hat{x}_1,\hat{x}_2,\hat{x}_3$ are output, ready for decoding into edited frames.
This toy example shows that each frame only interacts with its own source frame, preventing cross‑talk that would blur edits, while the replay step enforces consistency across the streaming horizon.
How does Vidu S2‑Editing differ from a standard Diffusion Transformer that attends globally?
Standard DiT lets every token attend to all others, which can cause edited regions to bleed into unchanged areas and prevents streaming because the model must see the entire clip. Vidu S2‑Editing restricts attention to temporally aligned source‑target pairs and adds a causal block‑wise mask plus Self‑Replay Forcing, preserving unedited content and enabling frame‑by‑frame inference.
**Figure 4.** Reference and Accessory Control. (a) Initial images plus a blue-mug reference illustrate object generation (left) and replacement (right). (b) The character leaves the original scene and enters a new one. (c) Taking off and putting back on the same hat.
By sharing GPU resources across encoder, backbone, refiner, and decoder modules, the system eliminates idle periods and reduces end‑to‑end latency, making real‑time editing feasible on commodity hardware.
Spatial Video Generation
We convert real-time monocular video into synchronized stereoscopic streams via depth‑driven warping.
Real‑time video generation currently delivers only flat, monocular streams, which limit the sense of depth and presence. To achieve immersive experiences we must turn each frame into a pair of slightly offset views, one for each eye, while keeping latency low enough for interactive avatars.
We treat the monocular output as a central view, estimate per‑frame depth, and warp it left and right to synthesize synchronized stereoscopic streams.
How does this depth‑based warping differ from a naïve stereoscopic duplication?
Naïve duplication simply copies the same image to both eyes, offering no disparity and thus no depth cue. Our method derives per‑pixel disparity from a learned depth map, warps the image in opposite directions, and fills occlusion gaps, producing true binocular parallax that conveys distance.
The editing pipeline reuses the same conversion stage. For monocular input, Vidu S2‑Editing first modifies the stream (style transfer, virtual try‑on, etc.) and then passes the result through the depth‑warping conversion. For stereoscopic input, the left and right views are concatenated horizontally, edited jointly, and split back into separate streams.
**Figure 13.** Qualitative results of Vidu S2 spatial video generation. Each example shows the generated left- and right-eye views.
Deploying this pipeline at scale raises two technical hurdles: high‑resolution rendering to fill a wide field‑of‑view headset, and sub‑30 ms end‑to‑end latency to keep head motion in sync. Future work aims to extend the approach to panoramic spatial video, enabling full‑head turning while preserving real‑time performance, which could dramatically reduce the need for traditional 3D asset pipelines.
Experimental Setup
We detail tasks, benchmarks, baselines, and evaluation protocols for streaming character generation and video editing.
This section specifies the two streaming tasks, the benchmark suites used for public and internal evaluation, and the set of baseline systems against which Vidu S2 is compared.
Runway’s GWM‑1 is a commercial avatar‑generation service that produces a single‑person video from an audio track and a reference image, but it updates the output only at fixed intervals.
PixVerse is an open‑source avatar system that synthesizes a talking head from audio and a single portrait, using a diffusion model that operates on each frame independently.
**Table 1.** Comparison of different models across OpenVE (GS, BC, Ovr.), RefVIE Ovr., and Joint Ovr. metrics.
**Figure 5.** Duration-stratified mean ratings for digital character generation. Absent tail points indicate unavailable ratings.
Quantitative Benchmark Results
Benchmark numbers show Vidu S2 dominates across all public metrics.
The paper’s core premise is that Vidu S2 delivers real‑time video generation by fusing a diffusion transformer with an accelerated inference engine. This section reports how that claim translates into concrete benchmark numbers.
Vidu S2‑Avatar outperforms every baseline on all nine StreamAV‑Bench metrics.
Table 1 shows the highest scores for $VA$, $VQ$, $PQ$, $AQ$, $AVAlign$, $AVSync$, $AIF$, $SC$, and $BC$ among 14 evaluated models.
For video‑editing benchmarks, Vidu S2‑Editing also leads the field. Table 2 reports its scores on Sparkle‑Bench, where it attains the top overall rating and excels on every listed dimension.
Vidu S2‑Editing achieves the highest overall score on Sparkle‑Bench.
Table 2 shows a score of 3.74, surpassing the second‑best model (Decart‑Lucy2.5) at 3.67.
Human Preference Evaluation
Human judges overwhelmingly prefer Vidu S2‑Avatar over all baselines across multiple criteria.
Vidu S2‑Avatar is preferred over all evaluated baselines, achieving 85.7 % preference versus Runway Character GWM‑1 and unanimous (100 %) preference versus both PixVerse Image Avatar and HeyGen.
Human‑preference GSB comparisons on a 5‑dimension rating suite (overall quality, motion, expression, semantic adherence, video quality) across multiple duration settings.
Beyond overall preference, Vidu S2‑Avatar maintains high scores across all five dimensions as stream length increases, showing stable identity preservation, motion continuity, and expressive behavior up to 90 seconds.
**Table 3** Joint OpenVE and RefVIE evaluation. “-” denotes an unavailable measurement. **Bold** indicates the best value in each column, <u>underlining</u> indicates the second-best value, light-gray shading denotes offline models, pale-blue shading denotes streaming models, and darker-blue shading highlights our streaming method. ↑/↓ denote higher/lower is better.
Qualitative Analysis and Conclusion
Qualitative results and concluding observations highlight Vidu S2’s superior visual fidelity and real‑time performance.
Vidu S2‑Avatar is preferred or judged perceptually equivalent over Runway in 85.7 % of cases.
Human preference evaluation on temporal‑consistency tests.
Beyond the headline numbers, Vidu S2‑Avatar consistently beats PixVerse and HeyGen in audio‑visual synchronization, and Vidu S2‑Editing shows clear advantages across all four editing tasks.
**Figure 8.** Qualitative comparisons for streaming digital character generation. Vidu S2-Avatar preserves identity, appearance attributes, fine-grained geometry, and temporal structure, while the closed-source baselines exhibit facial drift, body-proportion changes, hairstyle or attire changes, and local distortions in hair, eyebrows, fingers, or hand structure. (a) Female-character case with cosmetics and hand interaction. (b) Male-character case with facial and attire consistency.
**Figure 9.** Style-transfer cases. Vidu S2-Editing changes the brushwork and palette while preserving facial layout, subject silhouette, pose, and temporal texture attachment across the stream.
**Figure 10.** Virtual try-on cases. Vidu S2-Editing transfers the target garments while preserving body motion, garment boundaries, material texture, and hand–cloth occlusion relationships. (a) White-shirt virtual try-on. (b) Denim virtual try-on.
**Figure 11.** Subject-replacement cases. Vidu S2-Editing transfers the reference identity while preserving the source pose camera trajectory, scene geometry, and coherent boundaries around hair and limbs.