VIS-Ground: Video Interactive Storytelling with Contextual Grounding
Bingxuan Li, Yiwen Song, Xueqing Wu, Yanzhou Pan, Yang Li, Kuang Su, Jingyun Liu, Sebastian Ko, Huan Zhang, Tong Zhang, Nanyun Peng, Tomas Pfister, Yale Song
VIS-Ground uses structured dependency modeling to ensure interactive video continuations remain faithful to source material and prior visual states.
How can we enable viewers to interactively steer video generation while ensuring the resulting story remains consistent with both the original source material and the video's existing narrative history?
Interactive video generation systems often struggle to maintain consistency when a viewer intervenes, as new requests can conflict with established narrative facts or previously rendered visual events. VIS-Ground addresses this by compiling heterogeneous inputs—the grounding source, video prefix, and viewer request—into a structured dependency model that projects instance-specific constraints onto the generation process. Across three video backbones, this framework consistently outperforms direct generation and agentic baselines, achieving an average absolute improvement of 10.3 points on composite semantic metrics.
Paper Primer
The core mechanism treats contextual grounding as a compilation problem: it extracts predicates from the source, story, and video, then derives a dependency set that dictates what a valid continuation must respect. The system acts as a controller: it plans a script, verifies it against these dependencies, and iteratively repairs any conflicts before rendering.
VIS-Ground significantly improves contextual grounding and semantic consistency in interactive video storytelling.
Composite scores across three backbones (Omni-1.1-Flash, Veo 3.1, and MiniMax-H3) show consistent gains over the strongest baselines. Average absolute improvement of 10.3 points, with specific gains of 12.3, 15.3, and 3.4 points per backbone respectively.
The framework reduces performance variance across different video generators by 82% compared to direct generation, suggesting that the dependency-guided planning layer effectively decouples contextual reasoning from the underlying synthesis model's native capabilities.
Why is direct generation insufficient for interactive video storytelling?
Direct generation treats the source, prefix, and request as implicit inputs, which frequently leads to incompatible state changes, contradictions of previously shown information, or the satisfaction of requests in ways unsupported by the grounding source.
What is the role of the "dependency model" in this framework?
It serves as an explicit interface between contextual understanding and generation, allowing the system to identify which facts, events, or prerequisites are relevant to a specific viewer request and to reject proposed continuations that violate these constraints.
The framework assumes that the video prefix is a sufficient record of earlier instructions, and it relies on the ability of the underlying video generator to faithfully realize the planned script; if the renderer fails to visually establish a required transition, the system must rely on iterative output-level repair.
For researchers building interactive video systems, this paper demonstrates that performance gains come from explicit constraint induction rather than just scaling the underlying generation model.
The Challenge of Grounded Interaction
We expose the core problem: viewer requests are unconstrained by the source material, leading to context drift.
Interactive video storytelling promises viewers the ability to steer a narrative after seeing part of a video, but current systems suffer from “context drift”: viewer interventions often ignore the grounding source and the story’s prior visual state. Video Interactive Storytelling with Contextual Grounding (VIS‑Ground) frames this gap as a problem of aligning heterogeneous inputs—source material, rendered prefix, and new request—into constraints that guide the next continuation.
Contextual Grounding means building a unified representation that captures what the viewer’s request, the already rendered video, and the grounding source each allow to change or must keep fixed.
Structured Context Abstraction extracts entities, events, and relations from the source, video prefix, and request into aligned states that make dependencies explicit.
Generation Constraint Induction converts the aligned states into concrete constraints that any candidate continuation must satisfy.
**Figure 1 | Video interactive storytelling with contextual grounding.** A viewer's request can depend jointly on the grounding source and the rendered video prefix. These dependencies become generation constraints when projected onto a candidate future continuation. VIS-Ground structures the context, induces candidate-specific constraints, and generates and verifies the video continuation.
The core problem is that viewer interventions are currently unconstrained by the source material.
Abstract
VIS‑Ground raises interactive video generation scores by 10.3 points, outperforming all baselines.
Viewer interventions introduce latent dependencies on the grounding source and current video state, which only become apparent when the source, rendered history, and new viewer intent are considered jointly. Existing systems either prioritize following viewer instructions or aligning generated content with an external narrative, leaving unclear which context a model should ground on during interactive continuation and how heterogeneous, unstructured inputs can be transformed into such grounding. We introduce VIS‑Ground (Video Interactive Storytelling with Contextual Grounding), which performs Structured Context Abstraction, Generation Constraints Induction, and Constrained Video Generation through planning, verification, revision, and rendering, and report a 10.3‑point composite score improvement across three video generation backbones, with gains in narrative and knowledge grounding and remaining challenges in dependency extraction and faithful rendering.
Mechanism: Context Abstraction and Repair
Explicitly tracking dependencies enables the system to repair continuations until all constraints are satisfied.
When a viewer intervenes, the generation often drifts because the latest instruction alone cannot recover the hidden dependencies that tie the story to the source video. By exposing those dependencies explicitly, the system can detect mismatches and iteratively repair the continuation.
Instead of treating a continuation as a free‑form script, the method keeps a concrete set of grounded dependencies and forces every proposed script to respect them, looping until no violation remains.
Step 1: The prefix records the transfer event “A gives key to B”, updating possession to $k_A = 0$, $k_B = 1$.
Step 2: Candidate script proposes “A opens cabinet now”. The activation step checks the constraint “requires key owned by actor” and finds a violation because $k_A = 0$.
Step 3: Repair inserts “B returns key to A” before the opening event, yielding a new script.
Step 4: Re‑activation validates the repaired script: $k_A = 1$ at the moment of opening, so the constraint passes.
Step 5: Video rendering is inspected; if the key‑hand‑over is visually absent, the verification flags a missing transition, triggering another repair.
Grounded dependencies expose mismatches that a naïve “keyword‑check” would miss, forcing the system to add the missing hand‑over rather than silently ignoring the violation.
Extract and align video, story, and source predicates; derive the dependency set $R$.
Propose a candidate continuation script $P(k)$ conditioned on the grounded context and the current request.
Project $R$ onto constraints $c_j$, form the active set $C(k)$, and evaluate each constraint.
Collect failed or unresolved constraints, generate targeted feedback, and revise the script accordingly.
Render the revised script and verify that the realized video satisfies all applicable constraints.
Repeat steps 3–5 until verification succeeds or the computation budget is exhausted.
How does Continuation Repair differ from a standard constraint‑satisfaction planner that simply enforces hard constraints?
Standard planners treat constraints as immutable facts; Continuation Repair treats many constraints as *conditional* on the evolving script. When a new event is added, the set of active constraints is recomputed, allowing previously irrelevant dependencies to become active and be repaired on the fly.
Results and Performance Analysis
How we score videos and what the numbers mean.
Recall that VIS‑Ground prevents context drift by keeping generation anchored to a structured state that respects both the source video and the narrative history.
VIS‑Ground attains an overall composite score of 82.5, the highest among all evaluated methods.
Figure 5 (left panel) shows the box‑and‑whisker distribution where the “Ours” line tops the chart with a mean of 82.5.
**Figure 5.** Cross-backbone comparison. Left: overall composite-score variation. Right: performance across the six evaluation dimensions, averaged over settings for each backbone.
**Figure 6 |** *VIS-Ground* Performance gain over the baselines. Left: performance across six metrics for three video generators under equal grounding weights. Right: our method's JCS gain over the strongest baseline in narrative and knowledge settings (percentage points).
**Figure 8.** The comparator is the IF-leading baseline, held fixed across all four metrics in each panel.
**Figure 9.** Score distribution in the 72-output analysis sample.
SF measures how accurately the generated continuation preserves factual and relational information from the original source video.
How does SF differ from a simple “does the object appear?” check?
SF goes beyond presence; it requires the correct factual relationship (e.g., “the hero picks up the key”) and temporal ordering, whereas a mere presence check would accept any frame showing the object regardless of context.
JCS is the arithmetic mean of SF, Story Continuity (SC), and Interaction Fulfillment (IF), summarizing how well a video satisfies the three core semantic constraints together.
Why does the paper report both the mean JCS and a conjunctive “all ≥ 75” variant?
The mean can hide severe failures in one dimension (e.g., high SF but low IF). The conjunctive variant forces every metric above a quality threshold, making trade‑offs explicit.
VIS‑Ground consistently outperforms baselines in JCS across all backbones.
Baseline Comparisons
Adapting four baselines to support contextual grounding under a shared video‑generation interface.
The four baselines—Direct Generation, MM‑StoryAgent, MovieAgent, and VideoGen‑of‑Thought—originate from distinct planning paradigms. To evaluate VIS‑Ground’s contextual grounding, each pipeline must be adapted so that a viewer’s intervention can be inserted after a partially rendered prefix.
Baseline methods are existing video‑generation pipelines that differ in how they structure story planning, reasoning, and rendering, and they must be reshaped to accept a new request after an intermediate prefix.
These adaptations let us probe whether each planning paradigm can respect the shared state representation required for contextual grounding, while keeping the rendering stage identical across baselines.
Human Evaluation and Formative Study
Human studies quantify learning gains and validate automatic metrics against annotator judgments.
Interactive condition yields a substantially higher learning gain than the non‑interactive condition.
Mean gain 47.9 pp with interaction vs. 32.1 pp without, a 15.8‑point advantage.
**Figure 2 | Formative Study Results.** The study enrolled nine volunteers. *Left:* Learning gains for the seven participants who passed the attention test. *Right:* Experience ratings from four respondents.
Figures 10–12 illustrate the workflow participants followed during the formative study, from the pre‑test to the interactive questioning phase.
**Figure 10.** Pre-test interface. The pre-test screen asks participants to answer ten questions using their current knowledge, without correctness feedback. Each visible question includes an “I don’t know” option.
**Figure 11.** Noninteractive viewing interface. The noninteractive screen presents the video, subtitles, scene title, and viewing progress. The screenshot shows an Apollo 13 example.
**Figure 12 | Interactive viewing interface.** The interactive screen provides a question field and an “Add to Queue” control below the video. The displayed instructions ask participants to submit at least three questions and state that playback pauses while they type.
Human annotators achieve strong inter‑rater reliability across all five metrics.
Krippendorff’s $\alpha$ ranges from 0.78 to 0.86 on the ordinal rating scale.
Gemini’s automatic evaluation aligns closely with human judgments.
$\alpha$ ranges from 0.75 to 0.83 when compared to the median human rating.
Claude’s automatic evaluation also shows substantial agreement with human judgments.
$\alpha$ ranges from 0.64 to 0.76 against the median human rating.
Implementation Details and Prompts
Prompt templates that extract facts, align context, and plan multi‑scene video continuations.
G.1 Atomic source facts: this template extracts ordered facts from the original source, preserving exact supporting quotes. It captures narrative events, actions, relationships, and knowledge claims, assigning IDs (F001, F002, …) in source order. The result is a SourceFacts JSON.
G.2 Observed prefix ledger: the template inspects the complete video prefix and builds a conservative state ledger for the immediate continuation. It records identities, appearances, locations, object states, relationships, and unresolved events, distinguishing persistent from momentary state. The output is a PrefixStateLedger JSON.
G.3 Source–prefix alignment: this template aligns each atomic source fact to the prefix ledger, partitioning IDs into shown, partially shown, unseen, or uncertain categories. “Shown” means fully realized in the prefix; “partially shown” covers events underway at the cut. The alignment is returned as SourceAlignment JSON.
G.4 Instruction requirements: the template decomposes the viewer instruction into atomic, assessable requirements and maps each to relevant unseen or partially shown source facts. It flags unsupported or contradictory requests and outputs an InstructionRequirements JSON.
G.5 Two‑scene planning: acting as a source‑grounded narrative director, this template expands a frozen interactive‑video continuation into exactly two coherent 24‑second scenes. It enforces strict use of source‑supported facts, maintains persistent state, and produces a TwoSceneNarrativePlan JSON.
G.6 Six‑shot expansion: the template turns the two‑scene plan into six generation‑ready 8‑second shots, specifying dominant beats, focal actions, composition, camera movement, and video prompts. The result is a TwoSceneShotPlan JSON.
G.7 Plan review: this template audits the six‑shot continuation against the public source, instruction, and prefix, checking for material defects such as missing actions, identity drift, or unsupported speech. It reports concrete defects per shot and accepts the plan only if no defect is found.
G.8 Plan repair: based on the review findings, this template repairs the six‑shot plan while preserving the 48‑second, two‑scene structure. It enforces fixed fields (shot IDs, durations, continuation types) and uses only listed assets, outputting a corrected plan JSON.
**Figure 4.** Dataset Construction and Dataset Statistics