HarnessEval-W: Agentifying the Evaluation of Visual Worlds

Weiliang Chen, Haowen Sun, Jun Gao, Jiawei Chi, Hanyang Wang, Qiyu Dai, Yihao Li, Hao Li, Jingnan Gao, Yi-Hsin Hung, Xingzhuo Guo, Shangchen Miao, Zhiyuan Shi, Xiang Li, Fengrui Tian, Weihua Du, Ziqi Huang, Shenyuan Gao, Siqiao Huang, Mingyu Liu, Yifei Li, Shizun Wang, Xi Wang, Tianqi Zhang, Xue Luo, Xiyin Ren, Jinshan Ren, Xiaoyang Shen, Xiaobo Hu, Zhiyang Dou, Mingyu Ding, Yichao Yan, Xinchao Wang, Yizhou Wang, Shilong Liu, Wenzhao Zheng, Yueqi Duan, Yuan Gong, Ziwei Liu, Ming-Yu Liu, Jialong Wu, Jiangran Lyu, Fangfu Liu

HarnessEval-W replaces static rubrics with an agentic pipeline that decomposes world model evaluation into verifiable reasoning chains.

How can we move beyond scalar scores in world model evaluation by using an agentic, reasoning-based harness that decomposes complex tasks into evidence-grounded sub-questions?

Existing world model benchmarks rely on fixed, black-box metrics that provide a scalar score without explaining why a model failed or whether its physical reasoning is sound. HarnessEval-W treats evaluation as an agentic workflow: it interprets the context of each rollout, decomposes the task into specialized sub-questions, and uses sub-agents to gather grounded evidence before aggregating a final, transparent verdict. This approach aligns closely with human preferences, achieving a Spearman rank correlation of 0.93 on intentional transition tasks while providing an auditable reasoning trace for every score.

Paper Primer

Evaluating world models is difficult because each test case—such as a physical intervention or a long-horizon navigation—requires different criteria, making a one-size-fits-all rubric inherently fragile. HarnessEval-W solves this by routing each case to a library of reusable skills, where specialized sub-agents inspect specific visual evidence like bounding boxes or temporal consistency to justify their scores.

HarnessEval-W significantly improves human alignment over traditional evaluation protocols.

In a head-to-head comparison with WBench, HarnessEval-W increased pairwise accuracy on physical transition tasks from 31.9% to 71.7% and reduced the draw rate from 52.2% to 1.8%. Substantial improvement in discriminative power and human-preference correlation.

The evaluator is robust across different execution rounds.

When run multiple times, HarnessEval-W's correlation with human strength remained stable (0.928–0.964), whereas the baseline protocol's correlation fluctuated significantly (0.646–0.780).

Why is an agentic approach necessary for world models compared to standard video benchmarks?

World models are interactive and context-dependent; a fixed rubric cannot account for the unique physical actions or temporal structures of every generated world. Agentic evaluation allows the system to dynamically select the right tools to verify specific causal or geometric transitions as they occur.

What does the "evidence tree" provide that a standard benchmark score lacks?

The evidence tree records the complete logical chain of the evaluation, including which sub-agents were called, what visual grounding they used, and how they validated the outcome. This makes the final score auditable and provides actionable insights into specific failure modes.

Evaluation is shifting from static rubrics to executable, self-improving agentic systems. Researchers should treat the evaluation harness as a living component that grows alongside the models it measures.

The Need for Reasoning-Based Evaluation

We expose the gap: benchmarks lack reasoning, and introduce an agentic pipeline to provide evidence.

Benchmarks serve as the North Star for progress, yet current world‑model evaluations are fragile: they report only a single scalar and offer no insight into why a model succeeds or fails.

Standard benchmarks collapse rich model behavior into an opaque score, omitting the reasoning chain that would let a user verify or debug the result.

To close this gap we propose HarnessEval‑W, an agentified benchmark that interprets each case, breaks the evaluation question into measurable sub‑problems, and dispatches specialized sub‑agents to gather evidence before a parent agent synthesizes a final verdict.

The image displays a collection of institutional logos representing various universities and organizations, including MirroS, Tsinghua University, Peking University, NVIDIA, xbench, Carnegie Mellon University, Institut Polytechnique de Paris, University of Michigan, Shanghai Jiao Tong University, Zhejiang University, The University of North Carolina at Chapel Hill, National University of Singapore, Nanyang Technological University, University of California, Berkeley, and Massachusetts Institute of Technology.

Evaluation must provide reasoning, not just scores.

Prior Approaches to World Model Evaluation

Survey of video generation, interactive world models, and evaluation benchmarks relevant to HarnessEval‑W.

Recent video generation models such as Video Diffusion Models, Align Your Latents, CogVideoX, HunyuanVideo, and Wan have pushed visual fidelity and temporal coherence, yet they remain open‑loop generators. Interactive world models like Genie, GameNGen, DIAMOND, MineWorld, YUME 1.5, HY‑World 1.5, LingBot‑World, and Matrix‑Game 3.0 close this gap by conditioning on actions and maintaining a persistent state. To assess such systems, benchmarks ranging from VBench and EvalCrafter to WorldScore and WorldArena evaluate observation quality, transition correctness, and world persistence, motivating the WBench suite introduced here.

WBench isolates the three core evaluation dimensions for interactive world models, giving a concise yardstick for comparing approaches.

**Figure 2. Overview of the HarnessEval-W evaluation pipeline.** Given a case with its prompt and evaluation setting, the agentic planner routes the case to applicable skills from the skill library, recording an evidence-grounded reason for every activated and skipped skill (e.g., the Offscreen Evolution Verifier is skipped because all requested actions remain visible). Each activated skill spawns sub-agents that inspect the world model rollout for specific sub-questions, such as target visibility and final state validity; here, the Intentional Change Verifier detects an unrelated human intervention picking up the cube and zeroes the no-extra-event score. The collected evidence is finally aggregated into per-dimension scores and an interpretable final score.

The HarnessEval-W Pipeline

Decompose evaluation into hierarchical agents that route cases to reusable skills and trace evidence.

The standard benchmarks give only a single scalar, leaving us blind to why a model fails. Our harness therefore breaks the monolithic task into concrete, traceable sub‑problems, ensuring each failure can be pinpointed.

Instead of asking a model to produce one overall score, the harness routes each test case through a hierarchy of agents that decompose the problem, answer concrete sub‑questions, and then stitch the answers back into a transparent evidence tree.

Sub‑agent 1 scans the three frames, finds the ball in frame 2, and returns score 1 with the comment “target visible in frame 2”.

Sub‑agent 2 checks the ball’s final position against the expected left‑to‑right trajectory, detects a rightward shift, and returns score 0 with the comment “target moved opposite to instruction”.

The parent skill aggregates the scores (1 + 0) → overall skill score 0.5 and records the two diagnostic messages in the evidence tree.

This tiny example shows how a single high‑level skill can be broken into concrete, inspectable checks, turning an opaque scalar into a traceable set of failures.

The library is a catalog of reusable high‑level skills, each targeting one of the three evaluation axes and encapsulating the right set of sub‑questions for that axis.

How does this Skill Library differ from a traditional set of evaluation metrics?

Metrics give a single number per axis, but the library supplies agents that actually interrogate the rollout and return discrete, interpretable scores for each sub‑question. The difference is that the library produces a reasoning trace, not just a summary statistic.

Sub‑agents do not guess; they ground every answer in concrete visual evidence extracted from the rollout, returning both a numeric score and a short justification.

Is evidence‑grounded reasoning just a fancy name for post‑hoc visual inspection?

No. Post‑hoc inspection looks at the rollout after a score is produced, often without a systematic link to the score. Evidence‑grounded reasoning integrates the inspection into the scoring process, guaranteeing that every numeric answer is accompanied by the exact visual evidence that justified it.

**Figure 1** HarnessEval-W. The agentified benchmarks for interactive world models. Given an evaluation case HarnessEval-W routes it to the appropriate skills, decomposes each skill into measurable sub-questions answered by specialized sub-agents, and aggregates the validated evidence into a final score that traces back to the exact sub-questions that failed.

**Figure 3** Hierarchical structure of sub-agent reasoning for high-level skills. The Intentional Change Verifier skill decomposes its evaluation into eight measurable sub-questions, each answered by a dedicated sub-agent that inspects the rollout evidence and returns a discrete score with its reasoning.

Automated Case Generation

How we automatically build diverse, verifiable evaluation cases for world models.

Building a benchmark that stresses world models requires many diverse, diagnosable cases, but hand‑crafting each one does not scale.

The pipeline turns a high‑level scene description and a probe family into a fully‑specified evaluation case by chaining three lightweight agents.

Sampler selects the six axis values and verifies that a person can plausibly sit at the table (semantic compatibility).

Image Generator receives the prompt “a first‑person view of a living‑room with a person standing by a low‑clutter table” and produces a 256×256 RGB image.

Planner receives the image and a probe family “Intentional Transition”; it outputs the instruction “pick up the cup”, a short camera pan, and a 2‑second arm‑trajectory script.

Validator checks that the cup is visible, the person can reach it, and the planned trajectory will be captured in the rollout; all checks pass, so the case is accepted.

The example shows how each symbolic axis directly constrains the next agent, preventing impossible or ambiguous cases from ever entering the benchmark.

Sample a compatible scene taxonomy tuple.

Generate the initial image from the sampled metadata.

Plan a concrete action grounded in the image and probe family.

Validate the image–action pair; if it fails, return to sampling.

Store the verified case (ID, init image, prompt, action) for inclusion in the benchmark.

**Figure 4** Overview of the HarnessEval-W data construction pipeline. We first sample an initial world setup from a predefined scene taxonomy and probe family, and then employ a series of agents for world generation, action planning, and case validation.

Main Leaderboard and Performance

Benchmarking 18 world models with HarnessEval‑W and presenting the leaderboard.

We evaluate 18 world models on a frozen set of 330 HarnessEval‑W cases, translating each interaction into the model’s native input form while preserving the case intent.

Seedance 2.0* achieves the highest overall score of 75.5, the top among all 18 models.

Table 2 lists Seedance 2.0* with an overall score of 75.5, outperforming every other entry.

This table outlines evaluation criteria for world-state models, categorized by "Evaluation Axis," "Detail Evaluation Settings," and "Core World-state Question."

**Figure 5.** Statistics of the HarnessEval-W cases. (a) Case taxonomy distribution in HarnessEval-W. (b) Keyword frequency over scene descriptions. (c) Probe family distribution.

HarnessEval‑W provides a consistent ranking across diverse world models.

Human Alignment and Robustness

Key robustness and performance findings of HarnessEval‑W across models.

Recall that HarnessEval‑W evaluates world models via an agentic pipeline of evidence‑grounded sub‑questions; its robustness envelope is 4.9× narrower than that of WBench.

Figure 9 shows three independent runs; the shaded envelope around HarnessEval‑W’s regression line is 4.9× tighter than the envelope for WBench.

**Figure 6.** Human alignment of HarnessEval-W and its comparison with WBench. (a) Model-level human alignment: each point is one of the evaluated models; we fit a linear curve for both Intentional and Physical Transition. (b) Controlled comparison with the closest WBench protocols on the same data: we report pairwise accuracy (higher is better), draw rate, and Brier score (both lower is better).

**Figure 8** Robustness of evaluation. We run the benchmark three times and fit a linear curve for each run. The shaded band is the envelope of the three fits. The envelope of HarnessEval-W is 4.9× narrower than that of WBench.

Human Alignment Validation

Human judgments confirm that the agentic evaluation outperforms prior protocols.

We first measured how well the proposed agentic evaluator aligns with human preferences by collecting 5,000 pairwise A/B judgments across nine representative models.

HarnessEval‑W aligns closely with human judgments on both Intentional and Physical transitions.

Spearman $\\rho$=0.93 (Intentional) and $\\rho$=0.87 (Physical); Kendall $\\tau$=0.82 and $\\tau$=0.74 respectively.

Replacing HarnessEval‑W with the WBench protocols dramatically lowers pairwise accuracy on Physical transitions.

Accuracy drops from 71.7 % to 31.9 % when using Event Edit instead of HarnessEval‑W.

The draw rate explodes under the WBench protocols.

Draw rate rises from 1.8 % to 52.2 % when the evaluator is switched to Event Edit / Causal Fidelity.

HarnessEval‑W exhibits far lower evaluation variance across repeated runs than WBench.

Envelope width spans 0.33 Bradley–Terry units for HarnessEval‑W versus 1.61 units for WBench (≈5× wider).

**Figure 7** Complete reasoning traces produced by HarnessEval-W. Each card records the full trajectory of one evaluation: the case specification, the generated rollout, the selected evaluation route with evidence-grounded reasons for each skill. For the long-horizon navigation case (left), the planner selects drift analysis as the core skill; for the intentional state-change case (right), it selects the intentional-change verifier while skipping persistence and physics-specific skills.

**Figure 9** Correlations among evaluation axes. Each cell reports the Pearson correlation between two axes. The results are reported across all 18 models.

**Figure 10. Capability shifts under fine-tuning.** For each pair (Wan 2.2 $arrow$ DreamX-World and HunyuanVideo 1.5 $arrow$ HY-WorldPlay 1.5), we report the per-axis difference $\Delta = S_{\text{fine-tuned}} - S_{\text{original}}$.

Evaluator Statistics

We report how a range of world‑model systems fare on the agentic evaluation suite.

The evaluator distinguishes three families of world models: Prompt‑I2V generators, native‑action simulators, and camera‑pose estimators. Each system is run through the same suite of agentic sub‑questions, producing a scalar score per dimension.

Across the eight evaluation dimensions (Obs‑P, Trans‑E, Trans‑I, Trans‑P, Pers‑D, Pers‑R, Pers‑O, Overall) the models achieve a spread of performance, with the highest overall scores clustered among the Prompt‑I2V family.

Conclusion and Future Directions

We outline three avenues to extend the evaluation harness and recap its core contributions.

Evaluation cannot remain a fixed rubric; as models grow more capable, the evaluator must evolve in lockstep.

Test‑Time Scaling for Agentic Benchmarks means allocating more compute at evaluation time to perform finer skill decomposition, deeper sub‑agent search, and repeated verification, thereby making verdicts strictly more complete.

Scaling Skill Libraries requires continuously expanding a repository of human‑curated evaluation skills so that new, high‑fidelity scenarios can be assessed without rebuilding bespoke rubrics.

Recursively Self‑Improving Agentic Benchmarks turn skill gaps encountered during evaluation into actionable expansion signals, allowing the evaluator to acquire missing capabilities and write them back into its library.

In this work we introduced HarnessEval‑W, an agentic pipeline that decomposes world‑model evaluation into evidence‑grounded sub‑questions, aggregates transparent reasoning traces, and aligns closely with human preferences across 330 cases and 18 models.

We release HarnessEval‑W as an open‑source, living benchmark and invite the community to contribute new skills and evaluation cases as world models continue to evolve.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers