StreamEval: Evaluating Interactive Streaming Video Generation Models
StreamEval is a unified benchmark for interactive streaming video generation, evaluating request execution and state consistency across 300 test cases.
How do we systematically measure the performance of interactive streaming video generation models across diverse, evolving user tasks?
Interactive streaming video systems must generate content that responds to user inputs in real-time while maintaining visual identity and scene consistency. Existing benchmarks focus on static quality or short clips, failing to capture the difficulty of executing timed requests while preserving prior state over long sessions. StreamEval introduces a unified evaluation framework across five tasks—dialogue, performance, world generation, and two types of editing—using 300 scripted test cases with explicit timing and preservation requirements. It assesses systems on ten metrics covering task execution, state consistency, visual fidelity, and audiovisual coordination. Results from 37 models show that high visual quality does not guarantee reliable request execution, with task-specific success rates remaining low even for top-performing systems.
Paper Primer
The benchmark uses a scheduler to release multimodal inputs—such as speech, text, or visual references—incrementally during 15–300 second sessions. An authored oracle defines the "contract" for each test case, specifying exactly when a system must modify a target, retain existing content, or restore a previous state.
Request execution is the primary bottleneck for current streaming video models.
Across all tasks, models struggle to balance visual fidelity with the "contract" of following specific, timed instructions. The highest reported Edit Success score in reference editing is only 15.00 out of 100, and Following scores in avatar performance peak at 36.90.
The benchmark reveals a clear divergence between visual quality and functional reliability. Models that excel at maintaining identity or temporal stability often fail to execute specific user-requested actions, suggesting that current architectures prioritize smooth rendering over responsive, stateful control.
Why is this benchmark needed if existing video benchmarks already exist?
Existing benchmarks like VBench or WBench focus on static quality or specific world-model environments; StreamEval is the first to unify avatar, world, and editing tasks under a shared protocol that explicitly measures the ability to execute timed, multi-turn requests while preserving state.
What does the "Overall" score actually represent?
The Overall score is the equal-weight mean of applicable content metrics (Quality, Stability, Following, etc.), plus a +5 "sweet-spot" bonus for systems that support custom avatar initialization, which is a core requirement for interactive dialogue and performance tasks.
Introduction to StreamEval
We expose the unmet challenge of reliable request execution in interactive streaming video.
Real‑time interaction is becoming a core requirement for streaming video generation. Existing systems often produce high‑quality frames but fail to execute user requests reliably as sessions progress, exposing a gap between visual fidelity and stateful responsiveness.
It is a generation setting where a model must continuously ingest new user inputs and update the video output while preserving previously generated content and internal state.
StreamEval formalizes this problem with five interactive tasks, 300 scripted sessions (15–300 s each), and ten metrics spanning task execution, state consistency, visual fidelity, and audiovisual coordination. The benchmark reveals that the highest request‑execution scores are modest: 36.90 for avatar‑performance following, 25.93 for world‑generation following, and 15.00 for reference‑editing success.
The shift from static generation to real‑time interactive video is the key obstacle for future omni‑modal systems.
The StreamEval Benchmark
Describes the StreamEval benchmark setup, tasks, and data composition.
The benchmark defines a concrete experimental protocol for measuring interactive video generation across five distinct tasks.
StreamEval bundles five interactive video tasks—avatar dialogue, avatar performance, world generation, instruction editing, and reference editing—into a single evaluation suite that measures both generation quality and state‑preservation over time.
How does StreamEval differ from existing video generation benchmarks?
Existing benchmarks typically evaluate a single static task (e.g., pure video synthesis or dialogue) and ignore the need to retain and update state. StreamEval combines five interactive tasks, enforces a scheduler that interleaves inputs, and measures both the quality of new outputs and the fidelity of previously generated content.
Author an oracle that specifies desired states, cancellations, and restorations at each checkpoint.
Generate multimodal inputs (image synthesis, speech synthesis, licensed videos) using Qwen3.8‑Max, GPT‑Image‑2, Qwen‑Image, and Qwen3.5‑Omni.
Run automated visual, speech, and duration checks; regenerate any failing media (e.g., hand‑defect correction).
Release the screened inputs to the scheduler, which timestamps them and advances the source video.
Package the full timeline, oracle, and media assets for public release.
Scheduler releases the text command at t=10 s; the model updates the target region’s color attribute to “purple”.
At t=30 s the image reference is released; the model refines the texture to match the reference while preserving the new color.
At t=45 s the removal request is released; the model restores the original yellow color, confirming that the earlier edit was correctly undone.
Scoring checks that the final frame matches the original source video (color restored) and that intermediate frames respect the intermediate edits.
This walk‑through shows how successive controls interact: later inputs must both apply new changes and respect earlier state, exposing any failure to preserve or correctly revert prior edits.
**Figure 1.** StreamEval overview. Five tasks span avatar interaction, world generation, and real-time video editing. Avatar panels show recorded frames; the world-generation panel illustrates a scenario. Editing uses a shared source video with text instructions or reference images that change over time.
**Figure 2.** Benchmark construction and verification. Model-assisted authoring and input synthesis precede modality-specific checks. A Qwen-Image portrait with hand defects is regenerated and passes the same screening.
**Figure 3.** **Benchmark composition.** (a) Task and function composition. (b) Planned input types. (c) Inputs per session. (d) Gaps between adjacent scheduled inputs. The 200 cases shown contain 562 online events, excluding initialization and continuous source frames. Gap statistics exclude pairs containing a trigger.
The benchmark’s five‑task structure forces models to handle diverse interactive scenarios while preserving state across time.
System Evaluation and Metrics
We detail the experimental protocol, variables, and scoring pipeline for the five StreamEval tasks.
The evaluation follows a three‑stage pipeline: capture reference inputs, let a Vision‑Language Model (VLM) observe output frames, and finally convert those observations into numeric scores that are aggregated into task‑specific Overall numbers.
A Metric Profile is the collection of per‑task scores (quality, stability, identity, etc.) that together describe how well a system fulfills the requirements of a given interactive streaming task.
Is the Metric Profile simply the arithmetic mean of all ten metrics?
No. Each task averages only the metrics that apply to it, and the averaging is equal‑weight, not weighted by importance or by the number of metrics present.
Compute the repeat average: (80 + 85 + 78) / 3 = 81.
Since this stratum contains only this case, the stratum average equals 81.
With only one stratum, the final aggregated score Sₘ is 81.
This example shows how the multi‑level averaging smooths variability across repeats and strata before producing the final metric value.
State Preservation measures whether a system keeps the correct scene, object, and motion state across successive edits and user inputs, independent of visual quality.
Why is State Preservation reported separately from the overall Quality score?
Quality focuses on visual fidelity (sharpness, color, identity), while State Preservation checks that the underlying scene and motion remain consistent across edits—two orthogonal aspects of interactive streaming performance.
Record the input request (e.g., a new avatar pose or a world‑generation prompt) and the corresponding ground‑truth reference frames.
Run the system to generate output frames; feed the video to a Vision‑Language Model (VLM) that produces structured observations (quality, identity, reference matches, etc.).
Apply fixed scoring rules to each observation, converting them into numeric metric values per test case.
Aggregate scores across three repeats, then across all test cases within each duration stratum, and finally across the five strata using the formula for $S_m$.
Combine the relevant metric aggregates into a task‑specific Quality‑Overall, add the avatar‑customization bonus when applicable, and report the final Overall score.
**Figure 4.** Evaluation workflow. Recorded inputs and output frames from reference-editing case 234 illustrate quality, stability, reference, and preservation checks. Fixed rules convert structured VLM observations into run metric scores. Aggregation averages repeats, conditions, and duration strata before computing task-specific Overall.
Model Performance Results
Vidu S2 Avatar dominates across all five interactive streaming video tasks.
Vidu S2 Avatar consistently outperforms every other system across the five interactive streaming video tasks.
It records the highest Overall scores in Digital‑Human Dialogue (93.98), Avatar Performance (81.71), World Generation (33.36), Instruction Editing (45.43), and Reference Editing (39.02).
All models were evaluated on the StreamEval benchmark with an identical Metric Profile, ensuring a fair comparison across tasks.
**Table 1.** Comparison of various avatar models across multiple performance metrics including Quality, A-V Sync, Stability, Dialogue, Identity, Q-Overall, Custom, and Overall scores.
**Table.** Evaluation of various avatar models across metrics including Quality, A-V Sync, Stability, Following, Identity, Q-Overall, Custom, and Overall score.
**Table 1.** The distribution of metrics across both leading and mid-tier systems highlights the multi-constraint structure of streaming instruction editing. While Following assesses the requested operation and target, Edit Success evaluates complete editing requirements including scope, timing, persistence, and protected-content requirements (Following / Edit Success: 72.10 / 57.20 for Vidu S2 Editing, 63.89 / 43.52 for JoyAI-Video-Edit, and 12.04 / 4.26 for EditaLive (14B)). Meanwhile, Quality (up to 22.99) and Stability (up to 13.94) remain low even for the highest-scoring systems, indicating the difficulty of sustaining clean temporal rendering during live edits.
The provided image contains a table comparing five models (EditaLive (14B), JoyAI-Video-Edit, SANA-Streaming, Vidu S2 Editing, and XMax X2.0) across six metrics: Quality, Stability, Following, Edit Success, Preservation, and Overall.
The table presents a comparison of three video editing models: JoyAI-Video-Edit, Vidu S2 Editing, and XMax X2.0, across six metrics: Quality, Stability, Reference, Edit Success, Preservation, and Overall.
Performance varies widely across the five tasks, indicating that excelling in one metric does not guarantee overall superiority.
Analysis and Qualitative Results
We contextualize prior streaming video work and introduce reference‑editing as a new editing primitive.
Streaming video models must generate frames incrementally while reusing previously produced content, which drives research on efficient sampling and temporal consistency.
Generation approaches such as StreamDiffusion, Diffusion Forcing, CausVid, Self Forcing, Causal Forcing, Causal Forcing++, rCM, Causal‑rCM, speculative decoding, StreamDiT and StreamDiffusionV2 each trade sampling steps for speed, enabling interactive use.
Editing‑focused works—InstructPix2Pix, TokenFlow, StreamV2V, Live2Diff, LiveEdit, SANA‑Streaming, StreamEdit, EgoEdit, JoyAI‑Video‑Edit, VACE, EditaLive, Vidu S2‑Editing—extend these generators with mechanisms to incorporate external references or instructions while preserving the underlying source video.
Evaluation research (VBench, VBench++, EvalCrafter, T2V‑CompBench, GenAI‑Bench, VBench‑2.0, Video‑Bench, MLLM‑as‑a‑Judge, TC‑Bench, NarraStream‑Bench, MBench, OpenTrans‑360, ViewBench, WorldScore, WorldMark, WBench, EditWorld, StreamAV‑Bench, SyncNet, THEval, Full‑Duplex‑Bench, VideoFDB, TDVE‑DB, EditVerseBench, LiveEdit, EgoEditBench, CharEdit‑Bench, LongV2VBench) provides metrics for visual quality, temporal consistency, and interactive behavior, motivating the comprehensive StreamEval benchmark.
Reference‑editing lets a model replace or remove a previously supplied visual element while keeping the rest of the scene unchanged.
Frame 1 → output retains the red cape (baseline).
Frame 2 (after Reference B) → the model swaps the red cape pixels for gold‑satin pixels while keeping the horse’s motion unchanged.
Frame 3 (after removal request) → the model inpaints the cape region with the original background, yielding a cape‑free horse.
Reference‑editing must both insert new visual content and later erase it without leaving residual colors, demonstrating the need for precise temporal alignment and background reconstruction.
**Figure 5.** Reference editing under changing inputs. Reference A is preloaded, B arrives at 10 s, and removal is requested at 20 s. Columns pair source and output frames at recorded session times. Insets magnify the outlined cape regions, showing grey retained in both B-phase outputs instead of the requested gold satin.
Conclusion
We summarize findings, highlight challenges, and outline future directions for interactive streaming video.
StreamEval unifies five interactive streaming video tasks in a benchmark of 300 scripted cases, linking evolving requests to explicit execution and state‑preservation requirements. Our evaluation of 37 models reveals complementary strengths across tasks, yet online action following and reference‑based editing remain challenging, with the highest Edit Success score of 15.00 on a 0–100 scale. Improving request execution together with visual fidelity and state continuity, and gathering human pairwise judgments to validate the rubric, are key directions toward omni‑modal real‑time interactive applications.