LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing

Xinyu Wang, Chongbo Zhao, Fangneng Zhan, Yue Ma

LiveEdit enables real-time streaming video editing by distilling bidirectional diffusion priors into a 4-step causal model with spatial caching.

How can we adapt high-quality bidirectional diffusion models for real-time streaming video editing without sacrificing background stability or latency?

State-of-the-art video editing models rely on bidirectional attention that requires seeing the entire video at once, making them too slow for real-time streaming applications. LiveEdit solves this by distilling these complex bidirectional priors into an efficient, unidirectional streaming editor that processes video in small, causal chunks. This approach achieves state-of-the-art visual quality while reaching 12.66 FPS, enabling responsive, high-fidelity editing for live augmented reality.

Paper Primer

The core mechanism is a three-stage distillation pipeline that transfers editing knowledge from a heavy foundation model to a lightweight student. The student is further optimized using Distribution Matching Distillation (DMD) to compress the generation process into just four inference steps.

To handle the computational load, the system uses an AR-oriented Mask Cache: it calculates the L2 distance between source and edited latents to identify static regions, then reuses cached features for those areas instead of recomputing them. This is like a video editor who only repaints the moving subjects in a scene while keeping the static background layers locked in place.

LiveEdit achieves real-time inference speeds suitable for interactive applications.

The framework reaches 12.66 FPS, significantly outperforming offline baselines that require full-sequence processing.

The AR-oriented Mask Cache maintains high background consistency without recomputing static regions.

Ablation studies show that caching Self-Attention features preserves structural integrity, whereas caching Feed-Forward Network features causes severe blurring. 70% of redundant spatial tokens are pruned during inference.

Why is it difficult to simply use existing video generation models for streaming editing?

Existing models are designed for free-form motion synthesis rather than strict spatial alignment. Applying them to editing causes structural degradation and flickering because they lack the precise, region-specific control required to preserve unedited background details.

What is the primary trade-off when using the Mask Cache?

The cache trades off full-frame recomputation for speed. By identifying and reusing static tokens, the model avoids redundant calculations, but it must be applied specifically to Self-Attention layers to avoid losing high-frequency spatial details contained in the Feed-Forward layers.

Introduction and Motivation

We define streaming video editing’s latency‑stability trade‑off and its core challenges.

Streaming video editing must edit frames on the fly while keeping unedited background content stable across time. Two obstacles block deployment: (1) bidirectional diffusion models rely on future frames, so causal inference leads to forgetting or flickering, and (2) most of the scene is static, yet dense attention and feed‑forward passes are repeatedly applied, inflating latency.

A bidirectional diffusion model processes the entire video sequence at once, using full temporal context to generate high‑fidelity edits that respect both past and future frames.

**Figure 1. Gallery of various editing results and efficiency comparisons.** We propose the LiveEdit, a novel streaming video editing framework capable of performing causal, chunk-by-chunk manipulation with ultra-low latency and strict background preservation. By synergizing a progressive three-stage architectural distillation pipeline with an AR-oriented Mask Cache, LiveEdit effectively resolves the architectural incompatibilities and computational bottlenecks inherent in streaming editing paradigms.

**Figure 2. Comparison of video editing paradigms.** Unlike bidirectional models that suffer from inefficient inference, and past streaming models that fail to preserve accurate unedited content, our proposed streaming editing model leverages a Causal DiT with a mask-guided cache mechanism to achieve high-fidelity and efficient editing.

The central design tension is balancing ultra‑low latency with temporal consistency of unedited regions.

Related Work

We position our streaming editing approach among prior video generation, streaming models, and diffusion distillation work.

Recent video generation pipelines such as VACE, EditVerse, and UNIC achieve high‑fidelity control by embedding source video and dense multimodal conditions into massive joint representations. Model families like InsV2V and LucyEdit further improve fidelity using synthetic data or channel‑wise concatenation, while reinforcement‑learning tricks align outputs with user intent. All of these approaches process the full temporal context offline, incurring quadratic compute and prohibiting real‑time interaction.

InsV2V treats video‑to‑video translation as a per‑frame injection of synthetic edits, preserving the original spatial layout while applying learned transformations.

LucyEdit leverages a learned latent‑space encoder to inject user‑specified edits directly into the diffusion latent, enabling high‑quality video manipulation.

VideoCoF (Video Conditional Flow) provides a lightweight conditioning mechanism that aligns high‑level semantic cues with frame‑wise diffusion steps.

StreamV2V adapts video‑to‑video diffusion to a streaming setting by processing frames in small overlapping windows and reusing cached latent states.

Streaming autoregressive video models such as Diffusion Forcing, StreamDiffusion, and their successors (MAGI‑1, SkyReels‑V2, StreamingT2V) replace full‑sequence denoising with block‑wise sequential processing, enabling massive‑scale generation. Extensions like Rolling Forcing, Stable Video Infinity, and Self Forcing improve stability and reduce drift, while StreamDiffusionV2 introduces sink‑token KV caches for ultra‑low latency. However, these mechanisms are tuned for unconditional generation and rely on serial feedback from previously generated frames, which conflicts with the strict spatial fidelity required for editing.

Our AR‑based Cache stores only the most recent latent activations needed for the next causal step, discarding the serial feedback that harms editing fidelity.

To accelerate inference without sacrificing edit quality, we adopt a three‑stage distillation pipeline reminiscent of FlashVSR and PersonaLive but tailored to editing. Stage 1 compresses the bidirectional diffusion prior into a lightweight autoregressive model; Stage 2 refines the AR model using our AR‑based Cache; Stage 3 fine‑tunes the pipeline with reward‑guided control to preserve semantic fidelity. This integration yields unprecedented real‑time performance for complex video edits.

The LiveEdit Framework

Method details the core LiveEdit mechanisms and their efficient streaming implementation.

Two inefficiencies dominate naive adaptation of offline video diffusion models to streaming editing: (1) truncating future tokens collapses the bidirectional attention distribution into a uniform blur, and (2) recomputing every spatial token for each incoming frame wastes compute and breaks background consistency.

When a bidirectional model loses its future context, attention spreads evenly over past frames, erasing the localized focus that preserves structure.

With future frames available, the model concentrates 60 % of its attention on the immediate predecessor (frame 3).

When future keys are removed, the softmax must redistribute the 60 % mass across frames 0‑3, yielding roughly equal weights ≈ 0.25 each.

The uniform distribution blurs temporal cues, making frame 4’s output indistinguishable from earlier frames.

Uniform attention eliminates the model’s ability to anchor edits to the most recent context, which is why a dedicated distillation step is needed.

Streaming editing only changes a small subset of pixels per frame; recomputing every token repeats work and destabilizes static background.

LiveEdit stitches three distilled stages together and adds an autoregressive mask cache so editing runs in real time while keeping background fidelity.

**Figure 4.** Overview of the proposed streaming video editing framework. Our approach features a three-stage distillation pipeline that transfers editing capabilities from a bidirectional DiT to a 4-step causal model. Furthermore, an AR-oriented Mask Cache accelerates real-time inference by dynamically decoupling computation and reusing tokens in unedited background regions.

Replacing the bidirectional attention mask with a causal mask forces the model to predict only from past frames, but naïve replacement destroys the learned priors.

How does Causal DiT differ from simply applying a causal mask to the pretrained bidirectional DiT?

Applying the mask alone leaves the weights tuned for future context, so attention flattens. Teacher forcing retrains the model under the mask, forcing it to learn a new representation that preserves locality despite the missing future tokens.

The pipeline progressively transfers knowledge from a full‑capacity bidirectional prior to a lightweight causal editor, ensuring each step respects the temporal constraints of streaming.

Stage 1: Train a bidirectional DiT ($\epsilon_{\text{bid}}^{\theta}$) on video‑to‑video editing with loss $\mathcal{L}_{\text{bid}}$.

Stage 2: Initialize a causal DiT ($\epsilon_{\text{causal}}^{\theta}$) and train with teacher forcing under mask $M_{\text{causal}}$, minimizing $\mathcal{L}_{\text{causal}}$.

Stage 3: Distill the causal model into a 4‑step generator $G_{\theta}$ using Distribution Matching Distillation, optimizing $\mathcal{L}_{\text{DMD}}$.

We first teach a bidirectional DiT to edit videos by matching the injected noise, establishing a strong offline prior.

We retrain the model under a causal mask, forcing it to reproduce the bidirectional representations despite lacking future context.

DMD aligns the generator’s output distribution with that of a frozen real‑score model, compressing inference to four steps.

**Figure 3.** Visualization of the attention distribution shift. Left: The bidirectional prior exhibits localized attention gathering. Right: Direct causal truncation forces attention to spread uniformly across all historical frames.

The cache reuses previously computed features for static background tokens, letting the causal model focus compute on edited regions only.

Autoregressive Mask Cache – per‑chunk update.

Why can’t we simply recompute all tokens each frame and rely on the model’s internal consistency?

Recomputing every token discards the exact feature values that already encode the static background, causing small numerical drift that accumulates into visible flicker; the mask cache preserves those exact values, guaranteeing temporal stability.

**Figure 5.** Visualization of the temporal consistency analysis and mask generation process. The left panels show (from top to bottom) the source video frames, the synthesized video frames, the computed difference matrices, and the resulting binary masks. The right panels display the statistical distributions of Temporal IoU and Pixel Difference across the sequence, with mean values of 0.016% and 0.126%, respectively, indicating high structural stability.

Experimental Results and Ablations

LiveEdit’s mask‑cache cuts latency to sub‑100 ms while beating all baselines on quality.

LiveEdit reduces per‑frame latency to 79 ms, a ≈25× speed‑up over prior streaming editors.

Latency drops from 197 ms (Stage 1) and 200 ms (Stage 2) to 7.89 ms (Stage 3) per chunk, enabling real‑time 79 ms frame‑wise inference.

The AR‑oriented Mask Cache dynamically routes computation based on where edits actually occur, preserving background fidelity while saving work.

**Table 1.** Quantitative comparison of video editing methods. We evaluate across six metrics: Text Alignment (TA), Background Consistency (BC), Motion Smoothness (MS), Dynamic Degree (DD), Aesthetic Quality (AQ), and Imaging Quality (IQ). Red and Blue denote the best and second-best results, respectively.

**Figure 6.** Qualitative comparison of streaming video editing performance. The source videos and instructions are displayed at the top. While existing methods exhibit significant limitations, leading to structural collapse or an inability to accurately follow the text, our approach precisely modifies the target regions and preserves the visual quality and temporal coherence of the original scenes.

**Table 2.** Ablation of our three-stage distillation pipeline. Comparison of generation configurations and inference efficiency across the different training stages.

**Figure 7.** Visual comparison of different cache locations. With the instruction "Change the red currants to deep purple grapes with a thin layer of frost on their skins".

**Table 3.** Quantitative results of the cache-mechanism ablation. Comparing the performance of applying the caching mechanism to either Self-Attention or FFN layers. Red denote the best results.

**Figure 8.** Distribution of token cosine similarity between consecutive denoising step.

**Figure 1.** User study results. Volunteers ranked the generated videos from our method and six baselines across three metrics: Instruction Consistency, Background Preservation, and Overall Quality. The line plots indicate the proportion of top-3 selections. Our proposed approach overwhelmingly dominates the evaluations, securing the vast majority of absolute "Best" rankings across all dimensions.

Additional Results and Analysis

Additional qualitative results, user study findings, and extended comparisons.

This supplementary material expands on the qualitative behavior of LiveEdit, details the user study, and provides extra visual comparisons.

In the discussion we contrast LiveEdit with Self‑Forcing and EgoEdit, highlighting that LiveEdit avoids costly ODE initialization and preserves spatial alignment via the AR‑oriented Mask Cache.

**Figure.** Qualitative results of the LiveEdit method, demonstrating various image editing tasks based on user input.

**Figure.** Qualitative results of the LiveEdit method across seven different image editing scenarios. Each row displays a set of four "User input" images on the left and a corresponding sequence of four "LiveEdit" output images on the right, demonstrating progressive or varied editing based on the provided text prompts.

**Figure.** Examples of image editing using the LiveEdit framework, showing various transformations based on user input.

The user study with 20 participants ranked LiveEdit highest on instruction consistency, background preservation, and overall quality, achieving near‑perfect top‑3 rates.

**Figure.** Qualitative comparison of video editing results. The user input prompt is: "Change the Canada goose's feathers from grayish-brown to iridescent deep blue with metallic sheen."

**Figure.** Qualitative comparison of video editing results. The user input prompt is: "Change the dark shorts to navy blue gym shorts with a reflective stripe on the side." The figure displays frames from various methods (InsV2V, LucyEdit, VideoCoF, StreamV2V, Stream., Stream. V2, and LiveEdit) compared against the original User Input.

Section C points to additional image‑editing cases (Figures 2‑4) that further illustrate LiveEdit’s versatility across diverse visual domains.

Section D references extra comparative figures (Figures 5‑6) that benchmark LiveEdit against baseline methods on a broader set of editing scenarios.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers