KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation

Yuqi Tang, Tengfei Liu, Yizheng Lai, Yuran Wang, Yang Shi, Wanshun Su, Zhuoran Zhang, Qixun Wang, Xiaohan Zhang, Xinlei Yu, Xuehai Bai, Xuanyu Zhu, Bohan Zeng, Bozhou Li, Shujie Li, Yifan Dai, Yujie Wei, Shixuan Liu, Haotian Wang, Jialu Chen, Yuanxing Zhang

KeyFrame-Compass is a diagnostic benchmark for evaluating how faithfully video models reproduce ordered sequences of keyframes.

How can we systematically evaluate whether video generation models accurately follow a sequence of keyframe constraints?

Video generation models are increasingly tasked with following "visual plans" defined by multiple keyframes, but existing benchmarks only measure generic quality or single-image preservation. There is currently no way to verify if a model correctly places specific images at the right time or maintains their order throughout a video. KeyFrame-Compass provides a diagnostic framework that decomposes this task into six metrics covering keyframe presence, fidelity, temporal ordering, and persistence. It uses a combination of multimodal large language models and specialized perception tools to score both plan execution and overall video quality. The benchmark reveals a persistent trade-off: models that strictly adhere to keyframe constraints often struggle to synthesize natural transitions, while those prioritizing smooth motion tend to drift from the prescribed visual anchors.

Paper Primer

The benchmark evaluates 386 curated samples across five controlled factors, including keyframe density (3 to 12 frames) and video structure (one-take vs. multi-shot). It forces models to prove they can interpret both multi-image lists and storyboard grids, exposing a significant gap in how open-source models handle structured visual inputs.

Proprietary models significantly outperform open-source systems in keyframe comprehension.

The leading proprietary system outscores the best open-source model by 0.153 in overall performance, with most open-source models failing to decompose storyboard-grid inputs into temporally ordered sequences.

Instruction adherence degrades as keyframe density increases.

Instruction-adherence scores drop from 0.849 for 3-keyframe sequences to 0.756 for sequences with 9 or more keyframes.

Why is a new benchmark necessary if we already have general video generation metrics?

Existing benchmarks collapse performance into aggregate scores that cannot detect specific failures like keyframe omission, temporal misplacement, or incorrect ordering. KeyFrame-Compass isolates these failures to diagnose whether a model is failing to follow the plan or failing to synthesize coherent motion.

What is the primary bottleneck for current state-of-the-art models?

The primary bottleneck is synthesizing plausible transitions between visual anchors. Even models that achieve high keyframe fidelity often produce abrupt, slideshow-like transitions or melting artifacts when forced to connect disparate keyframes.

The benchmark highlights that "semantic compliance" is not the same as "visual grounding." Some models follow the prompt's narrative instructions perfectly while completely hallucinating new character appearances or scene layouts that do not match the input keyframes.

Researchers should shift focus from generic video quality to the "transition bottleneck," as current models treat keyframes as loose semantic suggestions rather than strict visual constraints.

Introduction

We expose why current video generators fail to follow multi‑keyframe plans and introduce the KeyFrame‑Compass benchmark.

Video generation models that accept multiple keyframes often cannot preserve the intended visual sequence, leading to missing or misplaced frames and unnatural transitions. This limitation hampers workflows where creators rely on precise storyboard control.

The core challenge is to make a model faithfully reproduce an ordered set of reference images while still generating a smooth, plausible video.

**Figure 1.** Illustration of KeyFrame-Compass. The benchmark spans three application domains (daily capture, product visualization, and cinematic narrative) and covers both realistic and stylized visual content. Each test case provides an ordered set of keyframes that the generated video must reproduce at the intended moments.

Shifting from text‑only prompts to keyframe‑conditioned generation fundamentally changes the controllability challenge for video models.

Related Work

Related work positions KeyFrame-Compass among video generation benchmarks and highlights its unique focus.

Image‑conditioned video generation augments text‑to‑video models with visual references to preserve subject appearance, layout, and composition. Recent work expands beyond a single input image to storyboards, predefined keyframes, arbitrary frame guidance, and interleaved multimodal inputs, thereby strengthening control over event states, shot composition, and narrative pacing.

Evaluations for these methods still rely on generic video quality, reference fidelity, transition smoothness, or cross‑shot consistency, without a unified diagnostic that checks whether an externally provided sequence of keyframes is faithfully realized in order and at the intended temporal locations.

General video‑generation benchmarks such as TC‑Bench, VBench, EvalCrafter, VBench‑2.0, and audio‑video suites like VABench, T2AV‑Compass, AVGen‑Bench, LongAV‑Compass, and MSVBench evaluate final outputs rather than ordered keyframe fidelity. In the image‑conditioned domain, AIGCBench, UI2V‑Bench, Video‑CanvasBench, and ViStoryBench focus on visual‑condition preservation or story visualization, but not on ordered keyframe‑conditioned generation. Table 1 shows that KeyFrame‑Compass uniquely provides a diagnostic benchmark for this task.

**Table 1.** Comparisons between KeyFrame-Compass and relative image-conditioned video generation benchmarks. “–” indicates that the aspect is not reported or not used as an explicit benchmark dimension.

Benchmark Design

Defines the benchmark tasks, dimensions, and data pipeline for KeyFrame-Compass.

It is a benchmark that measures how faithfully a video generation model reproduces a prescribed sequence of keyframes while preserving the intended temporal narrative.

Define the task: a model receives an ordered list of keyframe images and a text prompt, and must generate a video that respects the keyframe order.

Assign keyframe locations: for multi‑shot videos, label each keyframe as first, last, or representative within its shot; for one‑take videos, assign a target timestamp.

Annotate each sample along five dimensions: prompt control level, video structure, keyframe count, input format, and application domain.

Specify two prompt variants: a minimal prompt containing only keyframe order and story synopsis, and a segment‑specific prompt that details temporal placement, subject states, camera language, and narrative content.

Choose video structure: either a single continuous shot (one‑take) or multiple shots (multi‑shot) to test motion generation versus shot transition handling.

Select keyframe count (3, 6, 9, 12) to vary the density of visual constraints.

Provide input in two formats: an ordered list of separate images or a storyboard grid that packs the sequence into one image.

Assign an application domain (daily capture, product visualization, cinematic narrative) to diversify content.

Collect the six keyframe‑response metrics and four quality metrics for each generated video.

Step 1: The benchmark assigns keyframe 1 as the first frame of shot A, keyframe 3 as the representative frame of shot B, and keyframe 6 as the last frame of shot C.

Step 2: The model receives the storyboard grid (a single image containing the six ordered keyframes) and the minimal prompt.

Step 3: The model generates a video of three shots, placing the first keyframe at the start of shot A, the third keyframe somewhere inside shot B, and the sixth keyframe at the end of shot C, while synthesizing motion between them.

Step 4: The evaluation pipeline extracts frames, matches them to the reference keyframes using the three‑stage matching pipeline, and computes the six keyframe‑response metrics.

This concrete run shows how the benchmark isolates the model’s ability to respect temporal anchors while still allowing freedom in the intervening motion.

**Figure 2.** **Overview of KeyFrame-Compass.** **Benchmark Design:** KeyFrame-Compass evaluates keyframe-conditioned video generation from a text prompt and an ordered set of keyframes, covering diverse domains. **Evaluation Framework:** Keyframe response is assessed by a staged matching pipeline that combines MLLM-assisted shot segmentation with pixel-level and semantic matching to locate keyframe occurrences within exact temporal windows. General video quality is evaluated by specialized perception models and evidence-grounded MLLM judges. Together, they yield 6 keyframe-response metrics and 4 general quality groups for diagnosing controllability, temporal alignment, and overall generation quality.

**Figure 3.** Data construction of KeyFrame-Compass. Narrative sources are curated and converted into structured story summaries, which are transformed into scene specifications for keyframe generation and video prompt construction. Candidate samples are then filtered through multimodal verification, human review, and safety screening to ensure quality, consistency, specification compliance, and safety.

Evaluation Metrics

Defines how generated videos are quantified across fidelity, timing, and overall quality.

This section details the concrete procedures used to turn a generated video into a set of scalar scores.

The pipeline first discovers the actual shot structure of a generated video, then aligns each input keyframe to a temporal window, finally picks the most semantically similar frame as the canonical match.

Stage 1: Gemini 3.1 Pro reports shot boundaries at 0 s, 2.4 s, 5.1 s, 8.3 s, 10.0 s, 12 s.

Stage 2: K₁ is assigned to window [0 s, 2.4 s] (one‑to‑one), K₂ to [5.1 s, 8.3 s] (one‑to‑one), K₃ to [10.0 s, 12 s] (one‑to‑one).

Stage 3: Within each window, DINOv3 similarity scores are computed for every frame; the frame with the highest score (and PSNR > 30 dB) is chosen as the match.

This concrete walk‑through shows how structural recovery, temporal windowing, and dual‑criteria filtering combine to produce a single, comparable frame per keyframe.

How does this pipeline differ from a naïve frame‑wise nearest‑neighbor match?

Naïve matching ignores shot boundaries and would compare every generated frame to every keyframe, conflating structural errors with visual similarity. The pipeline first enforces temporal coherence, then restricts similarity search to the appropriate window, isolating the two failure modes.

These metrics answer whether each input keyframe appears in the generated video, how faithfully it is rendered, and whether its temporal behavior matches the specification.

Why is Kendall’s $\tau$ used for order consistency instead of a simple count of out‑of‑order pairs?

Kendall’s $\tau$ normalizes the count of discordant pairs by the total possible pairs, yielding a symmetric score in $[-1,1]$ that is robust to varying numbers of keyframes. The linear shift to $[0,1]$ makes it comparable to the other metrics.

These metrics evaluate the video as a whole, covering visual fidelity, temporal coherence, adherence to the prompt, and how well audio and video stay in sync.

How does the “lowest‑scoring 40 % frames” rule differ from averaging over all frames?

By focusing on the worst‑aligned subset, JavisScore highlights brief desynchronizations that would be washed out by a full‑video average, ensuring the metric is sensitive to transient audio‑visual mismatches.

**Figure 4.** Keyframe matching pipeline of KeyFrame-Compass. Since generated videos may not preserve the assumed one-keyframe-per-shot structure, we first use Gemini 3.1 Pro to recover the actual shot structure and assign input keyframes to the corresponding generated segments. Expected temporal windows are then derived for one-to-one, multi-to-one, and unassigned cases. Within each window, candidate frames are filtered by DINOv3 semantic similarity and PSNR/SSIM pixel fidelity, and the valid candidate with the highest DINOv3 similarity is selected as the canonical match for evaluation.

Dataset Statistics

Dataset composition and diversity are quantified for the benchmark.

The benchmark now spans three video structure‑duration settings and three application domains.

Table 2 lists 386 test cases covering one‑take, short multi‑shot, and long multi‑shot videos.

Table 2 breaks down the 386 cases by video setting, duration, and application domain, showing a predominance of daily‑use videos and a balanced mix of one‑take and multi‑shot structures.

**Figure 5.** Dataset statistics of KeyFrame-Compass. From left to right: the scene-type word cloud, the distribution of characters per story, the distribution of keyframe counts across one-take and multi-shot samples, and the co-occurrence patterns between camera motions and shot scales. These statistics summarize the diversity of story structure, scene semantics, and camera-control requirements in the benchmark.

Main Results

Key findings on short‑video benchmark and model trade‑offs.

Video generation models often trade off between preserving input keyframes and producing high‑quality video; the Main Results section quantifies these trade‑offs on the KeyFrame‑Compass benchmark.

Seedance 2.0 achieves the highest overall score on the short‑video leaderboard.

Overall score = 0.807, which is 0.063 points above Gemini‑Omni‑Flash.

Seedance 2.0’s advantage stems from balanced performance: it leads in Spatiotemporal Coherence and Instruction Adherence and ranks second in the remaining four dimensions, whereas LTX‑2.3 excels in Keyframe Fidelity and Temporal Organization but falls to fourth overall due to lower Video Quality.

Seedance 2.0 is a proprietary video generator that jointly optimizes keyframe preservation, temporal consistency, and overall visual quality, yielding a well‑rounded score across all benchmark dimensions.

Gemini‑Omni‑Flash prioritizes raw video quality and audio‑video coordination, achieving top scores on those metrics while sacrificing keyframe fidelity.

Analysis and Insights

We dissect where open‑source video models excel and where they still falter.

This section isolates the precise strengths and weaknesses revealed by our benchmark, focusing on keyframe fidelity versus transition quality.

LTX‑2.3 is an open‑source video generator that conditions on a set of user‑provided keyframes and strives to reproduce those frames faithfully while stitching them into a coherent video.

How does LTX‑2.3 differ from standard diffusion‑based video generators?

Standard diffusion models typically condition on a single prompt and generate frames sequentially, whereas LTX‑2.3 explicitly aligns each generated frame to multiple visual anchors via a dedicated keyframe‑response loss, separating appearance fidelity from motion synthesis.

LTX‑2.3 attains the highest keyframe‑response scores yet its transition quality lags behind, as shown by low DVQ ($0.453$) and PR ($0.417$) values.

DVQ and PR are minimal‑mode metrics that capture temporal smoothness; LTX‑2.3’s scores are markedly lower than its keyframe‑response numbers.

**Figure 6.** Representative failure modes of open-source models. (a) *Slide-like transitions* and (b) *implausible morphing*: LTX-2.3 reproduces the input keyframes faithfully but connects them with abrupt cuts or physically implausible dissolves instead of natural motion. (c) *Static grid-like outputs*: the storyboard grid is reproduced as a whole rather than decomposed into temporally ordered shots. (d) *Irrelevant visual artifacts*: content unrelated to the intended shot intrudes into generated frames.

Instruction‑adherence scores drop as the number of keyframes increases, falling from $0.849$ (3 keyframes) to $0.756$ (9 & 12 keyframes).

Table 5 reports the per‑keyframe instruction averages; the decline is driven primarily by VMA, while AMA stays stable.

Open‑source models fail to parse storyboard grids, achieving Hit Rates below $0.28$ and often outputting near‑static videos.

Hit Rate measurements across daVinci, HunyuanVideo, SkyReels, and Wan2.2‑I2V remain under $0.28$, indicating inability to decompose grid inputs into ordered shots.

Instruction‑adherence scores correlate perfectly with human win rates (Spearman $\rho_s = 1.00$, $p = 0.017$), confirming the metric’s diagnostic value.

Table 6 lists the rank‑alignment statistics; Instruction Adherence attains the highest correlation among all dimensions.

Keyframe‑response loss forces the generated first and last frames to match $K_1$ and $K_2$ exactly.

The motion‑synthesis head predicts a linear interpolation of pixel values between the two poses.

Because the scene change is large, the interpolation yields a sudden cut: frames abruptly jump from $K_1$ to $K_2$ without smooth motion.

This toy illustrates why LTX‑2.3 excels at reproducing static anchors yet produces slide‑like transitions when the visual gap between keyframes is large.

Experimental Settings

This section details the KeyFrame-Compass benchmark composition and the evaluation setup for nine video generation models.

The benchmark comprises stories averaging 3.73 unique characters, with most narratives involving three to five characters.

It contains 75, 112, 110, and 89 video samples for 3, 6, 9, and 12 keyframes respectively, providing a graded difficulty spectrum.

Scene annotations are dominated by action, establishing, interaction, and payoff labels, which together account for 67.2 % of all scene tags.

After discarding static‑camera cases, shot‑level camera‑language annotations reveal 730 motion‑scale pairs; the most frequent motions are truck/track, follow shot, pan, and push‑in, while medium, full, and wide are the prevalent shot scales.

We evaluate nine representative video generation models—four proprietary (Gemini‑Omni‑Flash, Kling‑3.0‑Omni, Seedance 2.0, Wan2.7) and five open‑source (daVinci‑I2V, MagiHuman‑1080p‑I2V, HunyuanVideo1.5‑I2V, LTX‑2.3, SkyReels‑V2‑I2V, Wan2.2‑I2V‑A14B)—in a single‑pass setting; only Kling‑3.0‑Omni and Seedance 2.0 employ their agent‑mode pipelines for longer videos.

For each duration split, inputs are adapted to the model’s conditioning interface: multi‑image models receive a list of keyframes, while single‑image models get a storyboard grid; all models run at 720P except daVinci‑MagiHuman‑1080p‑I2V, which outputs 1080P, and aspect ratios are matched to the nearest supported ratio.

Generated videos are assessed with the unified pipeline from Section 3.3, reporting results separately for the minimal and segment‑specific prompts; because minimal prompts lack temporal and audio specifications, Keyframe Position Accuracy (KPA) and Audio Modality Adherence (AMA) are evaluated only for the segment‑specific prompt.

Human Alignment and Conclusion

Human judgments expose alignment gaps while the new benchmark quantifies them.

Human rankings align strongly with automatic metrics, with Spearman’s $\rho$ ranging from 0.70 to 1.00 across models.

Table 6 reports the correlation range for the five evaluated models.

The evaluation protocol—three blinded experts per comparison, tie‑half‑win accounting, and decisive‑majority agreement filtering—remains identical across all models, isolating model behavior as the only variable.

**Figure 7.** An example of a fable under minimal prompt. All models consistently preserve the visual style and closely follow the intended story progression specified by the input. Notably, Seedance 2.0 further enriches the sequence by introducing close-up shots of the lion’s attack from multiple viewpoints, resulting in a more dynamic and cinematic presentation.

**Figure 8.** An example of a cinematic story under minimal prompt. The outputs of all models generally reproduce the shot composition and camera staging specified by the input keyframes.

**Figure 9.** An example of a cinematic story under segment-specific prompt.

**Figure 10.** An example of a commercial under segment-specific prompt.

**Figure 11.** An example of a commercial advertisement under segment-specific prompt. This sample features rapid scene transitions with drastic differences between scenes. Although LTX 2.3 reproduces every single frame, its overall effect resembles PowerPoint slide transitions. The other models mostly produce static grid-like collage images.

**Figure 12.** An example under segment-specific prompt. In this sample, the visual differences between scenes corresponding to different shots are relatively small. Besides LTX 2.3, HunyuanVideo 1.5-I2V and daVinci-MagiHuman-1080p-I2V also produce some non-static results that go beyond simple storyboard-grid collages.

**Figure 13.** An example under minimal prompt. Some models produce abnormal and irrelevant content in this sample. Surprisingly, daVinci-MagiHuman-1080p-I2V correctly generates a single-shot video, although it does not fully respond to every input keyframe.

**Figure 14.** An example under minimal prompt. LTX 2.3 suffers from repeated texture artifacts, but it still achieves the best overall performance.

Calibration and Computation

Defines how keyframe–response matches are quantified and thresholded.

We compare each keyframe to every generated frame by measuring the cosine similarity of their DINOv3 embeddings; the highest‑scoring pair is taken as the candidate match.

To decide when a similarity score counts as a true match we calibrate a threshold on a held‑out validation set of 50 samples covering both prompt modes and all four short‑video models.

For each validation keyframe we take the top‑ranked pair from the similarity computation, label it matched if the pair shares at least one salient subject, object, or event participant, and then sweep candidate thresholds to find the point where false‑positive and false‑negative rates are equal.

The sweep yields an equal‑error point at $\gamma = 0.7837$ (FPR = 5.3 %, FNR = 5.4 %). For all benchmark runs we adopt the rounded, slightly more conservative threshold $\gamma^{*}=0.80$, which keeps the validation FPR at 5.3 % and raises the FNR to 6.4 %.

For each matched keyframe we compute pixel‑level similarity using PSNR and SSIM, normalizing each to the $[0,1]$ range and taking the maximum.

Finally we fuse semantic and pixel similarity into a single response score.

Judge Sampling

We assess how frame‑rate and repeated judge calls affect metric stability.

MLLM‑based metrics are evaluated on uniformly sampled frames at 8 FPS, a rate that preserves short‑lived motion while keeping input size tractable. To probe temporal sensitivity we also run the same videos at 4 FPS and 2 FPS; metric scores shift only marginally, so we fix 8 FPS for all reported results. Finally, we repeat the full evaluation five times on the 115‑sample leaderboard set, observing negligible variance in the macro‑averaged groups.

The table presents performance metrics across five experimental runs for three categories: Video Quality, Spatiotemporal Coherence, and Instruction Adherence. Each row provides the individual scores for Run 1 through Run 5, followed by the calculated mean and standard deviation for that metric.

Human Alignment Protocol

Describes how human judges evaluate generated videos against segment‑specific prompts and keyframes.

For each of the 60 cases we collect full video outputs from all five evaluated models and compute automatic metrics; the final subset contains 10 one‑take and 50 multi‑shot cases, with 20, 20, 10, and 10 cases containing 3, 6, 9, and 12 keyframes respectively, yielding 600 unique model‑pair comparisons.

Three annotators (out of a pool of twelve experts) view the input keyframes, the segment‑specific prompt, and two generated videos shown in randomized left–right order; model identities and automatic scores are hidden.

The human rubric mirrors the automatic scopes from Sec. 3.3: Video Quality combines static and dynamic visual criteria; Spatiotemporal Coherence aggregates Attribute Consistency, Spatial Orientation Consistency, and Physical Rationality; Instruction Adherence follows the video‑ and audio‑modality rubrics; Audio‑Visual Coordination evaluates intrinsic semantic and temporal agreement, analogous to ImageBind Similarity and JavisScore.

A tie is recorded when annotators cannot consistently prefer one video over the other for a given dimension.

After mapping the randomized left–right labels back to model identities, a win contributes 1 to the selected model, 0 to its opponent, and a tie contributes 0.5 to each; a model’s human win rate per dimension is the mean over its four opponents across all 60 cases and three annotators, and the overall human score is the mean of the four dimension‑level win rates.

Krippendorff’s $\alpha$ is computed separately for each unordered model pair, treating Tie as a third category; the median $\alpha$ across pairs is reported in Table 8.

Automatic scores are formed by setting GVQ = GST = GIA = GAV = 1 and then weighting: 2 × (SVQ + DVQ), 3 × (AC + SOC + PR), 2 × (VMA + AMA), 2 × (IB + JavisScore). The model‑level automatic score for each dimension is the mean of per‑case scores over the same 60 cases, and the overall automatic score is the mean of the four dimension‑level scores.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers