PlayWorld: Benchmarking World Models with Agent Players Over Long-Horizon Objectives

Kaixin Ding, Xi Chen, Minghong Cai, Zhiyuan Xu, Yiyang Wang, Yuxiang Lu, Junyi Li, Shuyang Chen, Yuan Gao, Xin Tao, Pengfei Wan, Hengshuang Zhao

PlayWorld uses multi-modal agents to benchmark interactive world models on long-horizon objectives.

How can we reliably benchmark video world models on long-horizon, interactive tasks rather than just passive video generation?

Interactive world models are difficult to compare because they respond to user actions with varying granularity, making fixed, predefined action trajectories unreliable for cross-model evaluation. PlayWorld addresses this by using a multi-modal Agent Player that observes generated frames and adaptively adjusts action execution—Keep, Stop, Extend, Correct, or End—to ensure each model pursues the same long-horizon objective. Experiments across nine state-of-the-art models reveal that while current systems handle short-term interactions well, they struggle significantly with persistent state evolution and global spatial consistency over long rollouts.

Paper Primer

The core mechanism is a closed-loop interaction protocol where an Agent Player (e.g., Claude Haiku) acts as a human surrogate. It receives a shared basic action sequence as a reference and dynamically modifies the control stream based on the model's real-time visual output, ensuring the intended objective is pursued regardless of the model's specific response speed or action scale.

Current world models fail to maintain persistent state evolution and global spatial consistency over long-horizon interactions.

Rubric-based evaluation across 171 scenarios shows consistently lower scores in out-of-sight and insight evolution dimensions compared to basic visual quality metrics. Even top-performing models like Genie 3 show significant performance gaps when evaluated on complex, multi-step objectives rather than simple frame-level aesthetics.

The benchmark uses a VQA rubric verifier to score performance on a 1–5 scale across four dimensions: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution. This approach correlates more strongly with human preference than traditional pixel-level or basic controllability metrics.

Why is a multi-modal agent necessary instead of just using fixed action sequences?

Fixed sequences cause different models to overshoot or undershoot the intended state due to varying action granularity; the agent adapts execution online to preserve the evaluation intent across diverse architectures.

Does the choice of agent model (e.g., Claude vs. Gemini) bias the benchmark results?

The authors found only minor differences in trajectory quality across agent models, as the task is constrained to adapting actions toward a provided objective rather than planning from scratch.

Introduction

Introducing the need for interactive evaluation of video world models.

Video world models simulate future states conditioned on current observations and user actions. Recent systems have demonstrated impressive video consistency and action controllability over long sequences. However, fairly comparing these interactive models remains challenging.

Passive benchmarks fix a trajectory in advance, while interactive evaluation lets an agent adapt actions to achieve a goal, exposing whether the model truly maintains world coherence.

In practice, a human player typically evaluates a world model by pursuing long‑horizon objectives through interaction. For example, a user may turn around 360° to check whether the environment remains consistent, or walk into water to see if realistic ripples are generated.

Fixed action‑conditioned evaluation is unsuitable because the same command sequence can produce vastly different outcomes across models. Action granularity varies, so a three‑command script may complete a full rotation in one rollout but only a partial turn in another, making geometry‑consistency scores incomparable.

To address this, we employ a multi‑modal Agent Player that interacts with world models toward specified long‑horizon objectives. Building on this paradigm, PlayWorld provides 171 human‑annotated scenarios, each with a defined objective.

Performance is assessed along four core dimensions—geometry consistency, interaction fidelity, out‑of‑sight evolution, and insight evolution—plus basic ability metrics for video quality and controllability.

Experiments across nine state‑of‑the‑art world models reveal that current systems remain unreliable on long‑horizon interactive objectives, especially in maintaining spatial consistency and persistent state evolution.

**Figure 1.** **PlayWorld evaluates world models from the perspective of a human player.** We define long-horizon objectives such as “turning around 360 degrees” or “walking into the water”, and use an Agent Player to interact with the world models. We then evaluate them from multiple perspectives, including geometry consistency, interaction fidelity, and persistent state evolution.

The shift from passive video generation to interactive world simulation is essential for measuring true agent‑centric capabilities.

Related Work

We survey prior world‑model systems and benchmarks, highlighting their limitations and introducing our evaluation concepts.

World models have proliferated across autonomous driving, robotics, and open‑ended interactive settings, each generating future observations conditioned on actions or user inputs.

WorldScore quantifies how consistently a model follows a prescribed camera or action trajectory while preserving scene geometry.

WorldDreamer is a generative model that imagines future observations conditioned on a sequence of actions, effectively “dreaming” possible world evolutions.

By exposing these gaps, PlayWorld’s adaptive Agent Player and VQA‑based rubric provide a unified, closed‑loop evaluation that preserves intent across diverse models.

The PlayWorld Benchmark

Design of the PlayWorld benchmark, its Agent Player, and VQA rubric for interactive evaluation.

PlayWorld introduces a multi‑modal Agent Player that drives each evaluated world model and a VQA‑based rubric that checks four core capabilities.

PlayWorld Benchmark defines long‑horizon, goal‑directed tasks that require sustained interaction with a world model.

How does PlayWorld differ from traditional passive video consistency benchmarks?

Traditional benchmarks only check that generated frames look plausible in isolation, while PlayWorld forces a model to follow a goal‑directed action sequence and be evaluated on whether the world stays coherent throughout the interaction.

The Agent Player repeatedly observes the world model’s output and decides how to adapt the next action to stay on track with the long‑horizon objective.

Why not just let the world model follow a pre‑written script without the Agent Player?

A fixed script removes the ability to react to unexpected changes; the Agent Player’s observation‑conditioned decisions let the evaluation capture whether the model can adapt its behavior when the scene deviates from the planned trajectory.

The VQA Rubric asks targeted Yes/No questions about the generated video to verify specific world‑model capabilities.

Why use a VQA rubric instead of purely pixel‑level metrics?

Pixel metrics cannot capture whether objects retain their identity or whether interactions obey physics; the VQA rubric directly queries these semantic properties, providing a more meaningful assessment of long‑horizon coherence.

Receive the current frame, previously executed actions, scene description, and long‑horizon objective.

Agent model decides to keep, stop, extend, correct, or end the active action.

Agent interface translates the decision into the world model’s native controls and executes the action.

Capture the resulting frame and feed it back to the agent model.

Repeat until the model signals “End” or the $40$‑step interaction budget is exhausted.

Step 1 – The agent observes the initial frame and chooses “keep” for the first action (move forward).

Step 2 – After reaching the sink, the frame shows water spilling, so the agent “extends” the raise‑arm action to hold longer.

Step 3 – The cup is still empty; the agent “corrects” by inserting a new “tilt cup” action before waiting.

This trial shows how the Agent Player can dynamically modify a predefined plan in response to unexpected visual feedback, preserving the long‑horizon objective.

**Figure 2.** Benchmark construction pipeline of PlayWorld. Starting from diverse initial worlds, annotators define long-horizon objectives based on scenario-specific attributes, write basic action sequences, and construct sample-specific VQA rubrics. The resulting benchmark spans over 170 human-annotated cases and 50 action patterns over 10–60-second rollouts, with more than 820 questions applied to over 1,400 interactive videos.

**Figure 3.** Composition of PlayWorld. The benchmark comprises over 170 human-annotated cases across diverse scenario categories. The cases span four evaluation dimensions and diverse subject types, with dimension-specific VQA questions tailored to their evaluation criteria.

**Figure 4.** **Representative cases in PlayWorld.** Each case combines an initial world, a scene-grounded long-horizon objective, and a sample-specific VQA rubric. The expected trajectory is visualized alongside each case; for insight evolution, the camera remains stationary and observes the scene for 60 seconds.

Benchmark Results

We report model performance on PlayWorld’s out‑of‑sight and insight evolution criteria.

The PlayWorld benchmark measures how well models preserve world coherence during interactive, long‑horizon tasks, using two complementary rubrics: out‑of‑sight evolution and insight evolution.

Closed‑source models outperform open‑source models on out‑of‑sight evolution by a noticeable margin.

Best closed‑source score 2.74 versus best open‑source score 2.40.

Beyond these evolution scores, we also report basic ability metrics (Aesthetic Quality, Motion Smoothness, etc.) and action controllability (Translation and Rotation Pass Rates) to give a holistic view of model performance.

Experimental Setup and Validation

We evaluate nine world models with the adaptive Agent Player and report trajectory‑validation pass rates.

All nine models receive the same initial world, long‑horizon objective, and basic action sequence, while the same Agent Player observes generated frames and issues actions for each model.

Replacing passive video consistency with interactive trajectory validation uncovers a sizable performance gap between closed‑source and open‑source models.

Table 3 shows closed‑source models achieving higher pass rates on geometry consistency, interaction fidelity, and out‑of‑sight evolution.

Trajectory Validation checks whether a model’s rollout maintains coherent world dynamics across time, ensuring that objects move plausibly, interactions respect physics, and unseen regions evolve consistently.

The evaluation protocol runs rollouts lasting roughly 10–60 seconds, then applies the VQA‑based rubric verifier (Sec. 3.3) and the basic ability metrics (Sec. 4.4) to each rollout.

Agent Player Analysis

We evaluate control strategies and agent models, showing that preset + agent adaptation outperforms other approaches.

The central premise of PlayWorld is that interactive evaluation, not passive video consistency, is needed to judge world models. This section revisits that idea while focusing on how the adaptive Agent Player and VQA rubric affect trajectory quality.

Control strategies differ in how they use the shared basic action sequence. Preset Only follows the sequence verbatim, Agent Only predicts every action online, and Preset + Agent blends the two by adapting the preset to model‑specific granularity.

Across both world models, Preset + Agent attains the highest Trajectory Scores and Human Preference percentages, confirming that online adaptation of a shared action prior yields more reliable goal achievement than either extreme alone.

We next examine how the choice of multi‑modal agent model influences the trajectories produced under the Preset + Agent strategy. Three models—Claude Haiku, Claude Sonnet, and Gemini 3.1 Pro—are evaluated on the same Genie 3 cases.

Model choice yields only modest differences: Claude Sonnet scores highest on the Trajectory metric, while Claude Haiku leads in Human Preference and exhibits the lowest decision latency, making it the preferred default for the current benchmark.

For each comparison we aggregate judgments from five independent raters with video‑generation experience. Trajectory Score is assigned by Gemini 3.1 Pro on a 0–2 scale; Human Preference aggregates pairwise preferences, granting full credit for wins and half for ties; Decision Latency records the average end‑to‑end time per model call.

World Model Performance Analysis

Key failures reveal why current world models still lag on long‑horizon coherence.

Sustained world evolution remains the primary bottleneck, with out‑of‑sight evolution scoring markedly lower than geometry consistency.

Table 2 shows Genie 3 leads on geometry consistency and interaction fidelity, yet out‑of‑sight evolution lags behind. ≈20% lower out‑of‑sight score than geometry consistency

Long‑horizon revisitation reveals global spatial inconsistency: when the camera orbits the Taj Mahal, the model repeatedly regenerates the monument at new viewpoints, breaking spatial uniqueness despite each frame looking plausible.

Interaction fidelity is limited to simple collision responses; third‑person agents stop at obstacles, but first‑person agents still pass through solid objects, and complex interactions like walking into water often produce incorrect visual or physical responses.

Trajectory control and world‑model capability can diverge: SANA‑WM achieves a high trajectory‑control pass rate yet receives modest rubric scores, indicating it follows the required path but fails to preserve memory or generate plausible interactions.

Automatic metrics primarily assess basic video quality; high scores on Video Quality do not guarantee successful completion of long‑horizon objectives, as models can score well while producing little camera motion or failing to reach target states.

**Figure 5.** Representative failures across four evaluation dimensions. Current video world models can follow camera or action controls, but still struggle to simulate coherent and realistic world dynamics.

Agent Interface and Robustness

We assess VQA scoring stability and the practical limits of the web‑based agent interface.

The paper’s core claim is that passive video consistency misses the interactive dynamics needed for agent tasks; PlayWorld instead evaluates world coherence during goal‑directed interaction using an Agent Player and a VQA‑based rubric.

Gemini’s VQA scores show a mean per‑model sample variance of 0.0112 between two independent scoring passes, indicating that a single fixed pass yields stable aggregates, though the authors note that averaging several passes can further reduce verifier variance when resources permit.

The agent interface for web‑based models bridges the lack of a local inference API: it opens an authenticated browser session, loads the initial‑world condition, dispatches model‑specific controls (Keep, Stop, Extend, Correct, End), monitors generation, captures each frame, and feeds the observation back to the Agent Player for the next decision.

During execution the interface performs deterministic steps—login, page navigation, submission, rendering‑state checks, and output collection—while the Agent Player solely decides which control to issue, ensuring that the interaction logic is isolated from the browser plumbing.

Recorded screenshots are direct crops from the browser session; all browser chrome, system controls, and assistant overlays are removed, leaving only the generated world content untouched.

**Figure 7.** Agent interface execution on a web-based world model. (a) The agent interface enters the initial-world condition; (b) waits until the generated world becomes interactive; (c)–(d) dispatches controls and captures observations for the Agent Player’s online decisions; and (e) terminates the episode while preserving the final video and execution record.

Human annotators construct the VQA rubrics by screening initial worlds, verifying captions, assigning viewpoints, writing long‑horizon objectives, and defining basic action sequences; only observable evidence is used, and any question lacking a determinable answer from the rollout is discarded.

The human‑alignment study involved five participants evaluating 120 video pairs (30 per dimension) under anonymized codes, yielding 600 pairwise judgments; unanimous agreement ranged from 16.7 % to 43.3 % while majority agreement exceeded 90 % across all dimensions, with Fleiss’ $\kappa$ values between 0.323 and 0.483.

Overall, at least three raters concurred on 95.8 % of items (Fleiss’ $\kappa$ = 0.434), and Spearman correlations between human preference and VQA scores were high ($\rho$ = 0.933 overall, up to 0.983 for geometry consistency), confirming that the VQA rubric aligns well with human judgments.

Human Alignment Study

Key implementation choices and evaluation pipelines for PlayWorld.

We first assess whether the VQA metrics used in the rubric reflect human judgments. Six hundred pairwise human judgments were collected across four evaluation dimensions, and Spearman’s $\rho$ was computed for each of the nine models; all correlations are positive, indicating that the VQA scores preserve the human ranking.

**Figure 6.** Alignment between human preferences and VQA metrics.

The evaluation loop supplies every model with the same initial frame, long‑horizon objective, and a human‑annotated basic action sequence. The Agent Player observes the generated frame, the remaining action sequence, and recent history, then returns one of {Keep, Stop, Extend, Correct, End}, iterating until End or a 40‑step budget is exhausted.

Models are run either through their public web interfaces (Genie 3, LingBot‑World, etc.) or via local adapters that generate chunks and feed them to the model (SANA‑WM, Hunyuan‑GameCraft‑2, etc.). HY‑World2 requires a global scene image and skips cases lacking such input; Hunyuan‑GameCraft‑2 demands a non‑empty action input, so insight‑evolution cases that involve only waiting are sometimes omitted.

Basic ability metrics combine seven higher‑is‑better Video Quality scores (Aesthetic, Imaging, Motion Smoothness, Temporal Flickering, Temporal Consistency, Depth Stability, Subject Consistency) with two Action Controllability pass rates (Translation < 0.3, Rotation < 45°), rank each model on each metric, average the nine ranks, and convert to a percentage.

The VQA rubric verifier is Gemini 3.1 Pro. For geometry consistency, interaction fidelity, and out‑of‑sight evolution, rollouts are first validated against the required trajectory; invalid rollouts receive the minimum score of 1. Insight evolution bypasses trajectory validation because it relies on stationary observation.

To balance temporal coverage and spatial detail, the verifier receives two streams: a primary 10 FPS stream resized to 384 × 216 and grouped into 5 × 5 grids, and a detail stream at 0.5 FPS resized to 800 × 450 and grouped into 2 × 2 grids. Both streams, together with the case objective and rubric, are fed to Gemini for scoring.

Figure 1’s radar charts plot the four rubric‑based dimensions on a 1–3 scale and Basic Ability on a 0–1 scale; the differing axis ranges mean the charts are not directly comparable across dimensions.

Table 7 reports inter‑rater agreement for the human‑alignment study: unanimous agreement percentages, majority agreement percentages, and Fleiss’ $\kappa$ for each of the four dimensions, with Out‑of‑sight Evolution achieving 100 % majority agreement and Insight Evolution attaining the highest $\kappa$ of 0.483.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers