MultiRef-Compass: Towards Comprehensive Evaluation of Multi-Reference-to-Audio-Video Generation

Xiaohan Zhang, Yuqing Wen, Junlin Chen, Yuqi Tang, Yiting He, Lizhuo Shao, Weiming Zhu, Tengfei Liu, Yang Shi, Jialu Chen, Yuanxing Zhang, Huaxiong Li

MultiRef-Compass provides a unified benchmark and hybrid evaluation framework for multi-reference audio-video generation.

How can we systematically evaluate models that generate audio-video content from multiple, diverse reference inputs?

Current video generation models increasingly accept multiple reference images, but existing benchmarks only evaluate single-reference fidelity or isolated audio-visual alignment. This leaves a gap in assessing how models jointly reason over multiple references to bind entities and compose coherent audio-visual events. MultiRef-Compass introduces a taxonomy-driven pipeline to construct 350 samples across four boards, evaluating models on multi-view preservation, multi-entity binding, and audio-visual consistency. It uses a hybrid framework that combines automatic metrics with a rejudging-enhanced MLLM-as-a-Judge protocol to provide interpretable, audit-ready scores. Experiments on eight representative systems reveal that while proprietary models outperform open-source ones, all systems struggle with cross-reference understanding and natural visual integration, often resorting to copy-paste artifacts or identity splitting.

Paper Primer

The core challenge in multi-reference-to-audio-video (MR2AV) generation is compositional binding: models must correctly associate multiple referenced entities with prompt-specified roles and interactions. MultiRef-Compass addresses this by using a structured asset-composition pipeline that forces models to handle multi-view subject preservation and heterogeneous entity interactions.

The evaluation framework is a dual-level system: it routes samples to specific metrics based on prompt metadata and applies a rejudging stage to ensure semantic consistency across MLLM-based scores. This rejudging acts as a calibration layer: it compares initial judgments side-by-side to resolve discrepancies in how the judge applies criteria across different model outputs.

Current MR2AV systems exhibit significant performance gaps in reference consistency and instruction following, particularly in open-source models.

Performance on complex tasks like "Action" and "Wearing" requirements shows larger variance across models than static scene preservation, with open-source models like SkyReels-V3 and Wan2.1-VACE trailing proprietary systems. Proprietary models like Seedance 2.0 lead across all four evaluation dimensions, while open-source models show compressed capability profiles and frequent identity splitting.

Why is a new benchmark needed if we already have video generation benchmarks like VBench?

Existing benchmarks primarily assess text-to-video or single-reference fidelity; they lack the multi-reference conditioning required to test if a model can bind multiple distinct images to specific roles or maintain identity across different views.

How does the "rejudging" stage improve the reliability of MLLM-as-a-Judge?

It mitigates the tendency of MLLMs to apply criteria inconsistently across samples by jointly examining the original evaluation prompt, the initial scores, and the rationales for all models, correcting only clear semantic inconsistencies.

The Challenge of MR2AV Evaluation

We expose the missing evaluation of multi‑reference audio‑video generation and introduce the MultiRef‑Compass benchmark.

Multi‑reference‑to‑audio‑video (MR2AV) generation demands that models fuse several heterogeneous inputs—images, audio, video—and textual instructions into a single coherent output. Existing benchmarks concentrate on text‑driven generation or single‑reference fidelity, leaving the MR2AV setting largely unmeasured.

Because MR2AV requires cross‑reference understanding, compositional binding, and natural visual integration, current protocols miss critical failure modes such as identity splitting (treating multiple views of one subject as different entities), identity swaps, attribute leakage, and copy‑and‑paste artifacts that break coherence.

**Table 1.** Comparison between MultiRef-Compass and representative (audio-)video generation benchmarks. MultiRef-Compass focuses on MR2AV generation, where each reference can be represented by multiple images and combined with optional video and audio references. It requires models to effectively fuse diverse reference modalities, establish correct reference binding, and preserve multi-modal reference consistency.

The field must shift from text‑driven evaluation toward explicit multi‑reference control.

Introducing MultiRef-Compass

Introduces the MultiRef‑Compass benchmark and its four evaluation dimensions.

The benchmark supplies a scalable, taxonomy‑driven pipeline that assembles multi‑view, multi‑entity, and human‑object‑scene compositions. It couples this pipeline with a hybrid evaluation that blends automatic metrics and re‑judging‑enhanced MLLM‑as‑a‑Judge scoring. Together they expose model weaknesses across four complementary dimensions.

**Figure 1.** **Overview of the MultiRef-Compass framework.** The benchmark evaluates MR2AV tasks through four complementary dimensions—Basic Quality, Reference Consistency, Audio-Visual Consistency, and Instruction Following, enabling fine-grained diagnosis of multi-reference-conditioned audio-visual generation failures beyond aggregate leaderboard scores.

It is a curated suite of 350 samples that deliberately combine multiple visual and audio references so that a model must truly fuse heterogeneous inputs to succeed.

How does MultiRef‑Compass differ from existing video‑generation benchmarks?

Existing benchmarks provide a single reference (often just a text prompt) and evaluate only overall quality. MultiRef‑Compass requires multiple heterogeneous references, enforces explicit reference binding, and reports performance across four orthogonal dimensions, exposing failure modes that aggregate scores hide.

MultiRef‑Compass offers 350 taxonomy‑driven, multi‑modal samples evaluated across four dimensions.

Dataset Construction Pipeline

Describes how MultiRef-Compass samples are curated, composed, and prompted for evaluation.

The MultiRef‑Compass benchmark is built in three systematic stages: curating diverse asset packs, recombining them per board configuration, and generating structured prompts that tie assets to instructions. Each stage enforces reproducibility while allowing fine‑grained control over reference types and task difficulty.

Asset‑Pack Curation – collect scene, video, and voice clips from royalty‑free sources (Pexels, Freesound); build subject packs either by extracting multi‑view frames from a source video or by generating 3–6 auxiliary views with AIGC tools, then manually verify identity, visibility, and quality.

Board‑Specific Asset Composition – for each benchmark board (B1–B4) recombine assets: B1 draws multiple views from a single subject pack; B2 mixes subject, object, and scene assets; B3 merges entities from multiple subject packs; B4 adds reference video and audio inputs for joint audio‑video conditioning.

Structured Prompt Generation – GPT‑5.5 first drafts three modality‑specific fields (video description, audio description, speech transcript) conditioned on the board’s difficulty level, then merges them into a single generation prompt; human reviewers subsequently check that the prompt faithfully reflects the selected assets and that all constraints are coherent.

**Figure 2.** Data construction pipeline of MultiRef-Compass. The benchmark is built from licensed asset packs covering subjects, objects, scenes, videos, and voices. These assets are filtered, annotated, and routed into different reference-conditioned tracks. After structured prompt generation and manual review, the evaluation framework combines automatic metrics with a rejudging-enhanced MLLM-as-a-Judge protocol, producing diagnostic scores across four major dimensions.

Select subject view S₁ (portrait with left‑hand raised) and object clip O₁ (ball bouncing), then pair them with scene background C₁ (park).

Feed S₁, O₁, C₁ into GPT‑5.5; it produces a video description (“a person raises their left hand while a ball bounces in a park”), an audio description (“soft ambient park sounds, occasional ball thuds”), and a speech transcript (“Watch the person interact with the ball in the park”).

Merge the three fields into the final prompt and attach metadata indicating “subject‑view S₁”, “object‑clip O₁”, “scene C₁”, and difficulty level “medium”.

Human reviewer checks that the prompt mentions the left‑hand raise, the ball’s trajectory, and the park setting; all three are present, so the sample passes verification.

This concrete flow shows how heterogeneous assets are bound together: the prompt explicitly enumerates each reference, ensuring the model must attend to all modalities rather than ignoring any component.

Hybrid Evaluation Framework

Defines the metric dimensions and the dual‑level Hybrid Judge evaluation process.

The evaluation framework operationalizes the four metric dimensions and a dual‑level judging process for each generated sample.

It combines an automatic MLLM‑as‑a‑Judge scoring pass with a second re‑judging pass that double‑checks uncertain scores, yielding more reliable MOS ratings — like a two‑stage jury where a junior juror gives an initial verdict and a senior juror reviews only the doubtful cases.

How does this hybrid judge differ from simply averaging multiple MLLM scores?

Instead of running two full passes on every sample, the hybrid judge runs a cheap first pass on all samples and a costly second pass only on those flagged as unreliable, preserving efficiency while still correcting outliers.

After the first MLLM score, a rule‑based router flags low‑confidence outputs; those are sent to a second MLLM pass that re‑evaluates with stricter criteria — like a spell‑checker that first highlights suspicious words and then asks a human editor to review only the highlighted ones.

What happens if the router mistakenly flags a reliable sample?

The extra re‑judging incurs minimal overhead and may slightly adjust the score, but the final MOS remains within the original confidence interval, preserving overall evaluation fidelity.

Collect the generated video/audio together with all provided references (images, audio clips, text prompts).

Run the rule‑based router to activate only the metric dimensions applicable to the current prompt.

Compute Basic Quality metrics: VTQ via DOVER++, ATQ via Audiobox, AQ via an MLLM checklist.

Compute Reference Consistency metrics: EF (YOLO‑World + SigLIP embeddings), DP and BC via MLLM, then apply the paste‑naturalness coefficient $P_{\text{paste}}(r)$ to EF.

Compute Audio‑Visual Consistency metrics: SLS (filter frontal faces with MLLM then apply LatentSync), ESM and SC via MLLM.

Aggregate each dimension’s scores into a 1–5 MOS rating.

Apply the Re‑judging Protocol to any dimension whose confidence is low; replace its MOS with the re‑judged value if the deviation exceeds the threshold.

Store the final MOS values for downstream analysis.

**Figure 3.** Statistics of MultiRef-Compass. The benchmark features complex condition combinations, including four reference structures, six audio settings, and fine-grained prompt constraints, while maintaining a balanced prompt-length distribution.

Evaluation Dimensions

Describes the six evaluation metrics and a rejudging stage that enforces item‑level consistency.

This section details how the evaluation pipeline measures six aspects of multi‑reference generation and how a lightweight rejudging stage resolves inconsistencies across model outputs.

Reference Consistency checks whether the generated output aligns with each provided reference (audio, video, or text) in the aspects that the references explicitly define.

How does Reference Consistency differ from a generic similarity score?

A generic similarity score collapses all modalities into one number, losing the ability to pinpoint which reference (audio, visual, or textual) is violated. Reference Consistency keeps each modality separate, so a failure in timbre does not mask a correct visual action, and vice versa.

Audio‑Visual Consistency ensures that the temporal and semantic relationship between the generated audio track and the visual sequence matches the joint specification of the references.

Why can’t we evaluate audio and visual quality independently and then average the results?

Independent evaluation treats the two streams as unrelated, ignoring the cross‑modal constraints that define the task (e.g., a door‑closing sound must coincide with the visual door closing). Averaging would reward a model that gets one modality right while completely failing the other, violating the definition of Audio‑Visual Consistency.

Compute the six metric scores (TS, IF, VT, AT, SCA, TO) for each generated sample using the prescribed sub‑modules.

Feed each score and its accompanying rationale into the MLLM‑as‑a‑Judge to obtain an initial judgment per sample.

Run the rejudging stage: for every evaluation item, collect the initial judgments across all models and compare their scores and rationales side‑by‑side.

If a discrepancy is detected (e.g., one model receives a high score while another receives a low score for the same criterion), the rejudge revisits the original video to verify the evidence.

When the rejudge can resolve the inconsistency, it overwrites the offending score and rationale with a corrected judgment; otherwise the original judgment is retained.

Report the final, rejudged scores as the official evaluation results.

The rejudge gathers the TS and VT scores from three competing models for this sample.

Model A’s TS score is 0.62 (rationale above); Model B’s TS score is 0.45 with the rationale “Speaker embedding below threshold.” Model C’s TS score is 0.60 with a missing rationale.

Because Model B’s TS score deviates sharply from the others, the rejudge re‑examines the audio track, recomputes the ECAPA‑TDNN similarity, and finds the correct score should be 0.61.

The rejudge updates Model B’s TS score to 0.61 and adds the rationale “Speaker embedding now meets threshold after re‑evaluation.”

The example shows how rejudging can correct an outlier judgment while leaving consistent scores untouched, preserving the original evaluation intent.

The rejudging stage therefore improves item‑level consistency without expanding the metric set, ensuring that each semantic criterion is applied uniformly across all model outputs.

Experimental Setup

Experimental setup and key findings for eight MR2AV models across quality dimensions.

Across the evaluated models, visual and audio quality vary markedly, with Gemini‑Omni and HappyHouse 1.1 achieving the highest scores.

Tables 2 and 3 show Gemini‑Omni leading visual metrics and HappyHouse 1.1 leading audio metrics.

We evaluate eight representative models that accept multi‑image references for omni‑modal video generation: six closed‑source (Seedance 2.0, Kling 3.0, HappyHouse 1.1, Gemini‑Omni, Wan 2.7, Vidu Q3‑Mix) and two open‑source (Wan2.1‑VACE, SkyReels‑V3). Audio metrics are omitted for the open‑source models because they lack native audio outputs.

All experiments run on eight NVIDIA A800 GPUs. Proprietary models are accessed via their official APIs; when possible they generate 10‑second, 1080p videos. Open‑source models follow their recommended inference configurations to avoid degrading generation quality.

Each model receives three reference views; for models that accept only a single image we stitch the three views into one composite image. All outputs for a given sample are evaluated with identical checklists and prompts, and Gemini 3.1 Pro judges all MLLM‑based metrics with repeated runs to assess consistency.

Results are reported across three boards using 13 metrics grouped into Basic Quality, Reference Consistency, Audio‑Visual Consistency, and Instruction Following. Timbre Similarity is evaluated only on board B4 and reported in Appendix 6.

Basic Quality findings show Gemini‑Omni excelling in visual fidelity and HappyHouse 1.1 in audio quality, while Seedance 2.0 attains strong structural realism but occasionally produces face‑region artifacts. Reference Consistency results highlight Kling 3.0’s highest EF score and Seedance 2.0’s superiority on BC and DP, revealing that embedding‑based similarity (EF) does not always align with fine‑grained preservation.

Open‑source models still lag in natural visual integration and cross‑reference understanding. SkyReels‑V3 copies reference images verbatim, inflating conventional similarity scores; our Ppaste metric discounts this inflation and uncovers identity‑splitting errors when multiple views of the same subject are treated as distinct entities. HappyHouse 1.1, while polished, appears noticeably synthetic, and Veo struggles with identity preservation for Asian faces, a gap partially closed by the later Gemini‑Omni release.

Evaluation of 8 representative models (closed and open source).

Related Work

Current benchmarks miss evaluating how models fuse multiple heterogeneous references in MR2AV generation.

Image‑conditioned video generation began with animating a single reference image and has since expanded to accept multiple images and heterogeneous modalities such as audio or video clips.

Early works emphasized core animation capabilities—subject customization, pose control, identity consistency, and localized motion—while newer methods add compositional spatiotemporal conditioning and trajectory guidance, enabling finer‑grained control.

Because models now must relate several references and obey textual instructions, evaluation protocols need to assess cross‑reference understanding, compositional binding, and multimodal consistency rather than only single‑reference fidelity.

A general‑purpose video generation benchmark that measures overall generation quality and alignment with textual prompts.

Extended version of VBench with additional metrics for temporal coherence and visual realism.

Benchmark suite for evaluating generated video content against reference videos using perceptual similarity metrics.

Framework for evaluating video generation with emphasis on fidelity, diversity, and temporal consistency.

Unified benchmark covering multiple video generation tasks, including text‑to‑video and image‑to‑video.

Benchmark for subject or reference consistency in image‑conditioned video generation.

Evaluation suite focusing on user‑provided image references for video synthesis.

Automatic and MLLM‑based evaluation of text‑to‑audio‑video generation.

Benchmark for multimodal generation focusing on audio‑visual quality and synchronization.

Comprehensive suite measuring audio‑visual generation across several quality dimensions.

Evaluation framework for long‑form text‑to‑audio‑video, image‑to‑audio‑video, and video‑to‑audio‑video generation.

Benchmark that jointly assesses multi‑reference consistency and cross‑modal alignment in audio‑visual generation.

Automatic Metric Results

Key automatic‑metric gains are highlighted and evaluation conditions are kept constant.

Gemini‑Omni* attains the highest VTQ score among all evaluated models.

Gemini‑Omni* scores 0.2373 VTQ, while the next‑best Seedance 2.0* scores 0.2225.

All eight models were evaluated on the MultiRef‑Compass benchmark using the same four automatic metrics (VTQ, ATQ, EF, SLS) and identical reference‑consistency and audio‑visual consistency checks.

**Table 2.** Results obtained with automatic tools, grouped by the available evaluation dimensions. The best result is shown in bold, and the second-best result is underlined. Due to content-safety filtering, Seedance 2.0* and Gemini-Omni* are evaluated on 282 and 245 samples, respectively.

Analysis of Automatic Metrics

Automatic metrics reveal Seedance 2.0* leads across all three evaluation dimensions.

Seedance 2.0* attains the highest AQ score among the evaluated models.

Table 2 shows Seedance 2.0* achieving AQ 3.9452, while the next‑best Gemini‑Omni* scores 3.9100.

All models were evaluated with the same Hybrid Judge and identical automatic metrics; only Seedance 2.0* and Gemini‑Omni* were run on fewer samples due to content‑safety filtering.

Model Capability Visualization

Visual comparison of model capabilities on the MR2AV benchmark.

SC outperforms ESM on the primary capability metric.

SC achieves a peak score of 4.9441 versus ESM’s 4.8528.

All systems share the same training data, hyper‑parameter settings, and evaluation pipeline; only the model architecture varies.

MLLM-Based Metric Results

Quantitative comparison reveals which models dominate each evaluation dimension.

Seedance 2.0 ranks first overall across the four MLLM‑based evaluation dimensions.

Table 3 shows Seedance 2.0 achieving the highest scores in Basic Quality, Reference Consistency, Audio‑Visual Consistency, and Instruction Following.

Gemini‑Omni attains the highest lip‑synchronization (SLS) among all models.

Audio‑Visual Consistency analysis reports Gemini‑Omni’s SLS as the strongest.

Seedance 2.0 leads the ESM sub‑metric, while HappyHouse 1.1 leads SC, indicating that semantic event‑audio matching and correct sound‑source attribution remain distinct challenges.

Kling 3.0 excels on audio and speech‑related instruction compliance, Wan 2.7 performs best on temporal tasks, and Seedance 2.0 shows the strongest visual instruction handling, confirming that current models exhibit modality‑specific strengths rather than uniform performance.

**Figure 4.** **Model capability visualization.** Fig.(a) and Fig.(b) show four-dimensional rankings for closed-source and open-source models. (using ranking due to different scale of auto and mllm In the radar charts, ranks range from R1 (best) at the outermost ring to R8 (worst) at the center. Fig.(c) and Fig.(d) decompose visual task following into Action, Wearing, Basic Setting, and Detail using original MLLM scores.

Summary of Model Capabilities

Key patterns in model capabilities across closed‑ and open‑source systems.

Seedance 2.0 attains the top rank because its capability profile is comparatively balanced, whereas competing systems concentrate strength in only a subset of dimensions.

Score distributions across the four sub‑metrics (Basic Setting, Detail, Action, Wearing) show Seedance’s values are evenly spread, while other models peak in one or two areas.

Open‑source models cluster toward the inner rings of the capability space, indicating uniformly weaker performance across all dimensions compared with the more dispersed profiles of closed‑source systems.

SkyReels‑V3 surpasses Wan2.1‑VACE overall, yet both remain substantially behind closed‑source baselines on every evaluated attribute, especially Reference Consistency and Instruction Following.

Closed‑source systems exhibit stable scores on Basic Setting and Detail, while Action and Wearing produce the largest inter‑model separations, suggesting that static scene attributes are reliably preserved but fine‑grained actions and clothing constraints are less dependable.

The same reliability pattern is amplified for open‑source models: both SkyReels‑V3 and Wan2.1‑VACE score lower across all four sub‑dimensions, with particularly incomplete fulfillment of Action and Wearing requirements.

**Figure 5.** Reference consistency breakdown across all evaluated models. BC and DP are reported with MLLM-based scores, while EF reports automatic entity fidelity.

Extended Metric Analysis

Ablation analyses detail metric behaviors across boards, rejudging effects, and evaluation stability.

Table 6 extends the main benchmark with Board 4 results, showing how Seedance 2.0 and Kling 3.0 trade off basic quality, reference preservation, and audio‑visual fidelity.

**Figure 10.** Representative Entity Fidelity case. This case illustrates the operation of EF tools.

**Figure 11.** Board-level Speech-Lip Synchronization analysis. (a) SLS scores across the three boards for Gemini-Omni, Kling, and Seedance 2.0, grouped by model. (b) Number of valid videos retained after filtering.

**Table 7.** Representative rejudging cases. The rejudge stage corrects both overly strict and overly permissive initial judgments by checking whether the score is consistent with the visual evidence and the checklist semantics.

**Table 8.** Stability of MLLM-dependent evaluation over five repeated runs on a 60-sample subset. Score reports the mean and standard deviation of the five run-level average scores. MAE denotes the average item-level absolute difference over all pairs of runs.

D.1 Entity Fidelity analysis reveals that Board 1’s average EF (0.6930) drops to 0.5893 on Board 2 and 0.5748 on Board 3, driven mainly by declines in the Human branch (0.5522 → 0.5409).

D.2 Speech‑Lip Synchronization shows that Gemini‑Omni retains 84 valid videos after filtering, whereas Kling and Seedance 2.0 retain only 47, indicating that current sync metrics are sensitive to motion and pose variations.

D.3 Speech Content Accuracy highlights language‑specific performance: Kling reaches 90.6 % English accuracy and 97.6 % Chinese accuracy, while Gemini‑Omni excels in English (98.1 %) but drops to 76.7 % in Chinese.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers