JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion

Yicheng Xiao, Wenxun Dai, Xinran Qin, Lin Song, Maoquan Zhang, Hang Xu, Yukang Chen, Yitong Li, Guohui Zhang, Yuan Zhang, Xuying Zhang, Tommy Zhang, Jianlong Yuan, Peihao Li, Shuai Lu, Siming Fu, Chuyang Zhao, Xin Han, Jie Huang, Wenbo Li, Guoqing Ma, Wei Huang, Xiaojuan Qi, Haoyang Huang, Nan Duan

A 16B-parameter autoregressive diffusion model enabling real-time, open-ended video editing at 30 FPS.

How can we transform video editing from a batch post-production process into a real-time, streaming-compatible autoregressive task?

Streaming video editing requires causal output and bounded memory, but existing models struggle to maintain editing consistency over long durations, often suffering from accumulated temporal drift and source-fidelity loss. JoyAI-Video-Edit addresses this by combining chunk-wise autoregressive adaptation with source-anchored distillation, which anchors the model to the input video to prevent error propagation. The system achieves 720p editing at 30 FPS on a single Nvidia B200 GPU, outperforming existing streaming editors while remaining competitive with strong offline systems.

Paper Primer

The core mechanism hinges on Source-Anchored Distribution Matching Distillation (SA-DMD), which compresses the iterative diffusion process into a two-step generator. SA-DMD acts like a navigator correcting a ship's course: it independently guides the model along text-conditioning and source-fidelity axes, using the aligned source chunk to counteract drift from imperfect autoregressive history.

JoyAI-Video-Edit significantly outperforms existing streaming video editors in both short- and long-video editing tasks.

On the LongV2VBench benchmark, the model achieves an overall score of 3.30, exceeding the strongest streaming baseline (XMax-X2.0) by 1.59 points. 44.4% faster throughput than the next-best streaming baseline at a higher resolution.

The model maintains high editing quality and temporal consistency over extended, open-ended streams.

Human evaluation shows a strong preference for JoyAI-Video-Edit over all compared streaming editors, with 81% to 90% of preference votes in pairwise comparisons.

Why is causalizing an offline editor insufficient for this task?

Offline editors rely on bidirectional temporal context that is unavailable in streaming scenarios. Simply causalizing them leads to a train-inference mismatch where the model consumes its own imperfect predictions, causing reconstruction errors and color deviations to accumulate into long-term drift.

What is the role of the "global sink" in the model's attention mechanism?

The first chunk of the video is retained as a global sink to serve as a persistent anchor for long-horizon generation, ensuring that the model maintains a stable reference point regardless of the total stream duration.

Real-Time Streaming Video Editing

The paper reframes video editing from offline batch processing to streaming causal generation.

High‑quality video editing has historically relied on offline pipelines that ingest an entire clip, apply bidirectional temporal processing, and emit the result only after full denoising. This batch‑centric design blocks real‑time use because it cannot produce frames before future content arrives and its memory and compute grow with clip length, leading to temporal drift when forced into a streaming setting.

Streaming video editing demands a model that generates each output frame causally, with constant per‑frame cost, while staying faithful to the incoming source and avoiding drift over arbitrarily long streams.

**Figure 1** JoyAI-Video-Edit enables real-time, high-quality streaming video editing.

**Figure 2.** JoyAI-Video-Edit supports diverse video editing tasks.

**Figure 12.** Additional qualitative results of JoyAI-Video-Edit across diverse editing tasks.

The key shift is moving from batch post‑production editing to a streaming, autoregressive approach that preserves fidelity and consistency with bounded compute.

Architecture and Related Work

JoyAI‑Video‑Edit combines a multimodal LLM, a causal VAE, and an MM‑DiT diffusion backbone for streaming video editing.

JoyAI‑Video‑Edit is built from three cooperating modules that together enable causal, instruction‑guided video editing.

The MLLM reads the first video frame and the editing prompt, then emits a compact sequence of condition tokens that steer the diffusion process.

How does the MLLM differ from a standard language model that only processes text?

Standard LMs lack visual encoders, so they cannot incorporate the source‑frame appearance. The MLLM adds a vision backbone and fuses its embeddings with the text embeddings before tokenization, enabling the diffusion model to respect both visual context and textual instruction.

The causal VAE compresses consecutive video frames into a latent sequence while preserving the temporal order needed for autoregressive generation.

Why is the VAE called “causal” instead of a regular VAE?

In a causal VAE the encoder processes frames strictly forward in time, never looking ahead. This guarantees that the latent representation of frame $t$ depends only on frames $\le t$, which is essential for streaming generation where future frames are unavailable.

MM‑DiT is the diffusion core that iteratively denoises latent tokens, guided by both condition and vision streams.

Is MM‑DiT just a standard diffusion transformer with extra inputs?

No. Standard diffusion transformers treat all tokens uniformly. MM‑DiT separates condition tokens from visual tokens into distinct streams, applies separate linear layers and RoPE, and then merges them with learned gates, which lets the model treat instruction cues differently from video content.

First the model learns static image generation, then it extends to video by adding temporal modeling while keeping the spatial knowledge.

Why not train the video model from scratch instead of building on T2I?

Training directly on video would require far more data and compute to learn basic texture and composition. By reusing the T2I checkpoint, the model inherits strong spatial priors, accelerating convergence and preserving image quality when extending to the temporal domain.

Bidirectional V2V pre‑trains the model with full‑clip supervision, establishing a strong editing baseline before imposing streaming constraints.

Why train with a bidirectional loss if the final system must run causally?

Bidirectional training lets the model see the full edited clip, establishing strong source preservation and temporal coherence. Those weights then serve as a warm‑start for the later causal autoregressive adaptation, reducing error accumulation when streaming.

**Figure 3** Architecture of JoyAI-Video-Edit. Our autoregressive diffusion architecture consists of an MLLM, a causal video VAE, and an MM-DiT diffusion backbone. The MLLM extracts condition tokens from textual and visual inputs, while the VAE projects videos and optional reference images into a shared latent space. The MM-DiT jointly models the condition tokens and latent visual tokens to generate edited-video latents, which are subsequently decoded by the VAE into the final output video.

Together, these components enable JoyAI‑Video‑Edit to edit videos frame‑by‑frame while respecting both past context and user instructions.

Chunk-wise Autoregressive Adaptation

How JoyAI‑Video‑Edit streams video edits with bounded compute.

Streaming video editing demands a model that can emit edits frame‑by‑frame without ever looking ahead, yet naïve causal adapters quickly drift because they lose the global context that a full‑clip batch provides.

Think of reading a book page‑by‑page while keeping the first page on a desk as a permanent reference – each new page can see the current and earlier pages, but never the future ones, and the first page stays visible as a “global sink”.

Chunk 0 (sink) is always attended to; its latent stays unchanged throughout the stream.

When processing chunk 3, the model attends to chunks 2, 3 (current) and chunk 0 (sink) – chunk 1 is outside the window and thus invisible.

Chunk 4 can attend to chunks 3, 4 and the sink; the attention context size stays at 3 chunks regardless of position.

Each chunk produces one output latent frame, which is immediately emitted to the downstream video decoder.

The window guarantees $O(W)$ attention cost per step, while the sink preserves a stable reference to the original source, eliminating the drift that pure causal models suffer.

How does this differ from a standard sliding‑window transformer?

A vanilla sliding‑window transformer simply drops older tokens; our design keeps the first chunk as a permanent global sink and retains bidirectional attention inside each chunk, so early‑stage source information never disappears.

**Figure 4** The Training Pipeline of JoyAI-Video-Edit. LHAD indicates the long-horizon autoregressive distillation.

To compress the bidirectional teacher into a fast causal generator we use Source‑Anchored Distribution Matching Distillation (SA‑DMD), which guides the student toward a teacher that is explicitly sharpened on the aligned source chunk.

Imagine rehearsing a long speech by practicing short segments, then stitching the rehearsed pieces together; the rehearsal of each segment sees the already‑practiced earlier parts, exposing the speaker to the cumulative fatigue that will appear in the full delivery.

Clip 1 processes chunks 0‑1, computes gradients, then clears the graph.

Clip 2 processes chunks 2‑3, re‑uses the cached sink and the two most recent chunks (1‑2) as history.

Clip 3 processes chunks 4‑5, again with sink 0 and the two latest chunks (3‑4) visible.

Gradients from the three clips are summed before the optimizer step, so the update reflects errors across the full six‑chunk horizon.

Segmented back‑propagation lets the model experience the same drift it will face at inference time, while never storing more than $W$ chunks of activations.

Why not simply train on full‑length rollouts and accept the memory cost?

Full‑length back‑propagation would exhaust GPU memory for long videos; LHAD’s segmented approach yields the same gradient signal with only $O(W)$ memory, making real‑time training feasible.

Deployment and Data Pipeline

How JoyAI‑Video‑Edit streams video chunks, caches states, and builds paired editing data for real‑time editing.

Real‑time video editing demands a pipeline that can ingest a live stream, edit each chunk instantly, and keep memory bounded.

The pipeline slices the incoming stream into fixed‑size chunks, processes each with a causal VAE and a few‑step diffusion model, then reuses the resulting key‑value cache for the next chunk, guaranteeing constant compute per frame.

Chunk 1: FP8‑quantized encode = 2 ms, DiT edit = 5 ms, decode = 1 ms, KV cache = 1 ms, pseudo encode = 0.5 ms → total 9.5 ms.

Chunk 2: reuse KV from Chunk 1, same per‑step timings → total 9.5 ms.

Chunk 3: reuse KV from Chunk 2, same per‑step timings → total 9.5 ms.

Sliding‑window cache holds the KV of the most recent two chunks; memory never exceeds the size of two KV states.

Overall latency for the 24‑frame stream = 28.5 ms, well below the 33 ms budget for 30 FPS.

Because each chunk reuses a bounded KV cache, the memory footprint stays constant regardless of stream length, and the per‑chunk compute is independent of how many chunks have already been processed.

How does this pipeline differ from a naïve full‑clip processing approach?

Full‑clip processing would require encoding the entire video before any edit, incurring memory that grows with video length and latency that scales with the total number of frames. The streaming pipeline instead works on fixed‑size chunks, reusing a small KV cache, so both memory and latency stay constant per frame.

Two complementary routes turn static image‑editing supervision into high‑quality video‑editing pairs: one edits a representative keyframe then propagates the edit across the video, the other generates paired videos from an original‑edited image pair while sharing early diffusion latents.

Keyframe 30 (1 s) is extracted and edited to replace gravel with grass.

The edited keyframe and the full 120‑frame video are input to the propagation model, which copies the background change to all frames while keeping original subject motion.

In parallel, an original image of the scene and an edited version (grass) are fed to the latent‑shared I2V generator; early diffusion latents are shared, then the later stages condition on the two images, yielding two videos with identical motion but different backgrounds.

Both generated pairs are passed through the quality filter; the pair passes because background consistency, motion fidelity, and edit correctness exceed thresholds.

The MLLM reviews the pair, confirms the edit instruction “replace gravel with grass”, and stores the pair for training.

Sharing early diffusion latents guarantees that the two videos remain temporally aligned, so the only difference is the intended edit, which is crucial for training a model that learns to apply edits consistently across time.

**Figure 6. Paired video editing data pipeline.** Paired videos are synthesized through keyframe-guided edit propagation or latent-shared I2V generation, and are subsequently filtered and recaptioned.

**Figure 7.** Data distributions of T2V continual training (CT) and supervised fine-tuning (SFT), and V2V supervised fine-tuning (SFT). The T2V data cover diverse semantic domains, while the V2V data include global, local, and subject-level editing tasks.

The table compares the performance of various video editing methods across several metrics, including resolution, throughput (FPS), and qualitative scores for overall performance, background change, global style, local add, local change, and local remove. The proposed method, JoyAI-Video-Edit, is highlighted in the final row.

Long-Video Editing Benchmarks

JoyAI‑Video‑Edit dominates the new LongV2VBench, excelling across all editing categories.

JoyAI‑Video‑Edit achieves an overall score of 3.30 and ranks first in all five editing categories on LongV2VBench.

Table 3 shows it surpasses the strongest baseline XMax‑X2.0 by 1.59 points overall.

**Figure 8.** Overview of LongV2VBench. The benchmark contains 229 long-video editing items across five categories: background change, global style editing, local addition, local modification, and local removal.

Quantitative Results

JoyAI‑Video‑Edit outperforms all streaming baselines on LongV2VBench.

JoyAI‑Video‑Edit attains an overall editing score of 3.30, more than double the best streaming baseline (1.71).

Table 3 shows JoyAI‑Video‑Edit scoring 3.30 versus XMax‑X2.0’s 1.71 on the same benchmark.

The efficiency gain stems from bounded‑history causal inference, which reuses a fixed temporal state and keeps per‑chunk computation constant as the video length grows.

Ablation and Efficiency Analysis

We isolate each strategy’s impact on editing quality and system speed.

We now ablate each proposed strategy to quantify its effect on editing quality and on the end‑to‑end system speed.

These gains stem from four orthogonal optimizations: (1) reduced‑precision quantization of both transformer and autoencoder, (2) compiled VAE paths with autotuning, (3) pipelined execution that overlaps host‑side work with device computation, and (4) causal KV caching that avoids recomputing already‑generated content.

**Figure 5.** Runtime analysis of JoyAI-Video-Edit on an Nvidia B200 GPU.

SA‑DMD improves global style by +0.63 points.

Global style rises from 3.61 to 4.24 when SA‑DMD is enabled.

LHAD adds +0.06 to the overall editing score.

Overall score increases from 3.00 to 3.06 with LHAD.

SA‑DMD anchors the teacher to the source chunk, while LHAD stabilizes long‑horizon rollouts; together they address complementary failure modes and deliver the strongest edits.

Human Preference Study

Human judges overwhelmingly prefer JoyAI‑Video‑Edit and it runs faster than rivals.

JoyAI‑Video‑Edit is preferred by up to 90 % of evaluators over existing streaming editors.

Human judges chose JoyAI‑Video‑Edit over LiveEdit (90 %), SANA‑Streaming (87 %), XMax‑X2.0 (81 %), and StreamDiffusionV2 (87 %).

**Figure 9.** Pairwise human preference between JoyAI-Video-Edit and competing methods. Red and blue denote preferences for JoyAI-Video-Edit and the competing method, respectively; the hatched region denotes ties.

Conclusion and Impact

Authors and qualitative results showcase JoyAI‑Video‑Edit’s capabilities and contributors.

JoyAI‑Video‑Edit turns video editing into a continuous, interactive workflow, preserving identity, layout, motion, and unrelated regions while new frames arrive. The causal pipeline gives instant visual feedback, enabling on‑the‑fly instruction changes during playback.

Core contributors (equal contribution) are Yicheng Xiao, Wenxun Dai, Xinran Qin, Lin Song (project leader), Maoquan Zhang, Hang Xu, Yukang Chen, Yitong Li, Guohui Zhang, Yuan Zhang, Xuying Zhang, Tommy Zhang, Jianlong Yuan, Peihao Li, Shuai Lu, Siming Fu, Chuyang Zhao, Xin Han, Jie Huang, Wenbo Li, Guoqing Ma, Wei Huang, Xiaojuan Qi, Haoyang Huang (corresponding author), and Nan Duan (corresponding author). Additional contributors, listed alphabetically, include Anson Li, Bi Cheng, Bin Li, Bo Wang, Boyang Li, Dongyan Yang, Feice Huang, Fengyuan Shi, Fan Lin, Haoran Li, Haoyu Wu, Hu Yu, Jia Shi, Jiachen Liu, Jiaqi Wang, Jiawei Li, Jianhui Liu, Jiayi Deng, Jiangmiao Pang, Junhao Zhuang, Kangliang Chen, Libing Fang, Lichen Ma, Liang Lin, Lingjie Li, Lixin Wang, Nan Jiang, Nanhua Lai, Nick, Pan Wang, Qingyi Si, Qiushi Yang, Ruofan Lv, Shaonan Wu, Tong He, Wanyan Yu, Wei Tang, Xiaoxiao Huo, Xing Pan, Xi Yang, Xuan Yang, Yan Li, Yanfei Tang, Yichen Wang, Yijun Yang, Yipeng Sun, Yuhang Li, Yujia Liang, Yue Ma, Zeyue Xue, and Zuopeng Dong.

**Figure 10.** Qualitative comparison with streaming video editing methods.

**Figure 11.** Additional qualitative results of JoyAI-Video-Edit across diverse editing tasks.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers