WorldReward: Reward Modeling for Camera-Conditioned World Models

Yibin Wang, Zehan Wang, Junshu Tang, Zhimin Li, Yujie Zhou, Jiazi Bu, Pengyang Ling, Feng Han, Zhixiong Zhang, Long Xing, Shengyuan Ding, Ziang Li, Cheng Jin, Yuhang Zang, Jiaqi Wang, Tianyu Pang

WorldReward uses chunk-level VLM reasoning to unify action-consistency and visual-quality evaluation for world models.

How can we build a reward model that accurately evaluates whether a camera-conditioned world model's generated video matches the user's commanded camera trajectory?

Camera-conditioned world models struggle to balance precise action execution with visual coherence; existing rewards either ignore visual quality or decouple it from the commanded motion, leading to drift and artifacts over long horizons. WorldReward is a VLM-based reward model that decomposes videos into action-aligned chunks, evaluating each segment for both action adherence and visual quality before aggregating the results via voting. On the new WorldReward-Bench, this approach outperforms proprietary models like GPT-5.5 and Gemini 3.1 Pro, providing a more reliable signal for reinforcement learning post-training.

Paper Primer

The core challenge is that whole-video evaluation dilutes local action evidence, while separate geometry and image rewards fail to capture the coupling between motion and visual stability. WorldReward solves this by grounding judgments in local action-video chunks: it is a VLM-as-judge system that treats the video as a sequence of action-aligned segments, ensuring that local failures—like flickering or incorrect camera turns—are explicitly attributed to the specific action that caused them.

WorldReward achieves the highest agreement with human preferences on action consistency, appearance quality, and motion quality.

Evaluated on WorldReward-Bench (760 paired generations), it outperforms proprietary VLMs and specialized geometric/image-based reward models. It exceeds GPT-5.5 by 3.42, 1.45, and 3.56 percentage points on action, appearance, and motion dimensions, respectively.

WorldReward provides a superior reinforcement learning signal for world-model post-training.

Post-training HY-WorldPlay 1.5 using WorldReward consistently improves both action execution and visual quality across short- to long-term horizons compared to the WorldCompass baseline. Combined-action accuracy improves by up to 2.78 points, with simultaneous gains in visual quality scores.

Why is a chunk-level approach necessary instead of just feeding the whole video to a powerful VLM?

Direct whole-video judging creates a noisy, overly long context where short-lived local action evidence is easily missed or diluted by the global impression. Chunking forces the model to inspect specific action-aligned visual evidence, making it easier to identify and attribute local motion errors or visual artifacts.

How does this model handle the "coupled" nature of action and visual quality?

Unlike heterogeneous systems that use separate models for geometry and image quality, WorldReward derives both action-consistency and visual-quality preferences from a single VLM interpretation of the same action-aligned visual evidence, ensuring the two signals are mutually consistent.

WorldReward demonstrates that reward models for world models should be grounded in local, action-aligned reasoning rather than global video impressions. This shift enables more stable, long-horizon interactive generation by providing a unified, high-fidelity signal for RL post-training.

Introduction and Motivation

We expose why existing rewards miss camera‑trajectory alignment and introduce the WorldReward motivation.

Camera‑conditioned world models aim to generate videos that faithfully follow user‑specified camera motions while preserving geometry, appearance, and dynamics. Current reward models split this requirement, using geometry‑based scores to check trajectory execution but ignoring visual fidelity, and image‑based scores to assess frame quality while overlooking whether the commanded actions were actually performed. This separation leaves a gap: a unified evaluation that simultaneously verifies precise camera‑trajectory alignment and high‑quality visual output is missing, motivating the WorldReward approach.

The gap between open‑ended video generation and precise camera control drives the need for a unified reward.

Related Work and Context

We situate WorldReward among prior world models and outline its chunk‑level reward approach.

Camera‑conditioned world models predict future frames given user‑specified camera motions, while RL‑based post‑training aligns diffusion generators to human preferences. Visual reward models either learn a scalar scorer or employ a VLM as a judge, and existing reward signals focus on geometry or task completion rather than visual fidelity of camera‑driven videos. WorldReward builds on these lines by pairing a VLM‑as‑judge with chunk‑level comparisons that assess both action consistency and visual quality.

HY‑WorldPlay 1.5 is a lightweight extension of the original HY‑WorldPlay that adds a per‑chunk visual consistency check without increasing the model’s parameter count.

Chunk construction addresses three key difficulties: (1) the multimodal context becomes too long if the full video is fed to the VLM; (2) a monolithic input weakens the link between a specific action and its visual evidence; and (3) local motion errors or artifacts can be washed out by global averaging, making them hard to attribute.

For each chunk we provide four components: the source image $x_0$ to anchor scene identity, a frame‑grid overview showing start‑middle‑end frames of the chunk, action‑level panels that juxtapose the first and last frames of the corresponding action from both videos, and the caption $d$ to preserve semantic context. This layout lets the VLM focus on the precise visual transition induced by each local action segment.

Structured‑evidence ablations reported in Table 6 confirm that removing any of these components—source image, frame‑grid, or action‑level panels—consistently reduces agreement with human preferences, underscoring their necessity for reliable chunk‑level reward reasoning.

Chunk-Level Reward Reasoning

Chunk-level reward reasoning evaluates action consistency and visual quality per video chunk.

Camera‑conditioned world models often drift from the commanded camera trajectory, producing videos that look plausible but fail to follow the intended motion.

Think of a movie critic who watches each scene separately, judging whether the camera moves as instructed and whether the picture looks realistic before forming an overall verdict.

Chunk 0: Action 1 – Video A keeps the vehicle idle while yaw‑left rotates the scene; Video B does the same but with weaker depth cues. Decision: Tie.

Chunk 0: Action 2 – Video A maintains a consistent rightward shift; Video B shows the shift but with flat motion. Decision: Video A wins.

Chunk 0: Action 3 – Video A preserves texture stability; Video B exhibits minor texture sliding. Decision: Video A wins.

Chunk 0 visual quality – Temporal consistency, dynamic generation, and artifact checks all favor Video A. Decision: Video A wins.

Repeat for Chunks 1‑2, accumulating votes. Final tally: Action votes 4 : 2 : 0 (A : B : Tie), Visual votes 6 : 0 : 0.

Chunk‑level voting isolates the exact actions and visual aspects where a model fails, enabling targeted improvements rather than a single opaque score.

**Figure 1.** Chunk-level reward reasoning in WorldReward. A paired video is split into action-aligned chunks. Each chunk is judged from a six-image input (source image, frame-grid overview, and four action-level panels), yielding action and visual-quality winners in {A, B, Tie} that are aggregated by voting into video-level preferences.

Across the six chunks, Video A wins four action votes and all six visual‑quality votes, while Video B secures only two action votes and none of the visual votes.

**Figure 2.** Reasoning-augmented preference-data pipeline. (1) Input preparation over image content, style, and trajectory groups. (2) Pairwise generation from eight world-model variants, with per-model coverage and pair-assignment density. (3) Chunk-level reasoning annotation by a VLM, followed by agent auditing and human review.

Agent‑Harness Quality Control (Algorithm 1)

**Table 1. Quality-control outcomes.** Top: chunk-level annotations retained or revised by the agent harness, with a breakdown of revisions. Bottom: agent-proposed revisions confirmed or rejected by human reviewers.

Training the Reward Model

How WorldReward is trained and then used to steer a camera‑conditioned world model.

Camera‑conditioned world models often drift from the exact user‑specified trajectory; the paper solves this by training a reward model that reasons at the chunk level.

We fine‑tune a Vision‑Language Model (VLM) to predict, for each video chunk, both the action‑control reasoning and the visual‑quality reasoning, then output a categorical preference $\{A, B, \text{Tie}\}$.

Why does the training use teacher forcing instead of feeding back the model’s own predictions?

Because the reward model must learn to articulate the correct visual and action reasoning; feeding its own predictions would corrupt the conditioning context and make the categorical preference noisy.

Compute $\log p_{\theta}(\text{"act"}\mid z_1)$, $\log p_{\theta}(\text{"high"}\mid z_1,\text{"act"})$, and $\log p_{\theta}(\text{A}\mid z_1,\text{"act"},\text{"high"})$.

Sum the three log‑probabilities for chunk 1; repeat for chunk 2 with its two tokens.

Average the negative sum over the two chunks to obtain $\mathcal{L}_{\text{SFT}}$.

Even a tiny toy example shows that the loss aggregates over both action and visual tokens, forcing the model to learn a joint representation.

After the reward model is trained, it supplies reinforcement signals for post‑training of the world model.

Sample a shared autoregressive prefix $x_{1:n-1}$ from the EMA policy $\bar{\pi}_{\phi}$ conditioned on the previous actions $a_{1:n-1}$, the initial frame $x_0$, and the camera description $d$.

For each of $G$ candidates, sample a target clip $x^{(i)}_{n}$ from $\bar{\pi}_{\phi}$ conditioned on the prefix, the current action segment $a_n$, $x_0$, and $d$.

Construct the reward input for every unordered pair $\{i,j\}$ by concatenating $x_0$, its caption, the local action sequence, a frame‑grid overview, and comparison panels.

Feed each pair to WorldReward, which returns an action winner and a visual‑quality winner.

Compute $\omega_1(y^{(\text{act})}_{12}) = 1$ and $\omega_2(y^{(\text{act})}_{21}) = 0$.

Since $G-1 = 1$, $R^{(\text{act})}_1 = 1$ and $R^{(\text{act})}_2 = 0$.

The win‑rate reduces to a binary indicator when only two candidates are compared, making the reward signal very sharp.

These dimension‑specific rewards are then standardized and combined to guide diffusion training.

Finally, we evaluate the trained reward model on a dedicated benchmark.

We build a human‑annotated suite of 760 paired video comparisons that isolates trajectory execution and visual quality, letting us measure how well a reward model reproduces human preferences.

Evaluation Benchmark

Key dataset and training details that enable WorldReward’s chunk-level reasoning.

We construct a large paired video dataset and a held‑out benchmark to evaluate chunk‑level reward reasoning.

WorldReward training consumes roughly 100 k annotated chunks, trimming the raw 150 k candidate pool by a factor of 1.5.

After generating 150 k candidate chunks we subsample to ≈100 k examples for reward‑model training.

WorldReward‑Bench is a curated collection of 760 video pairs that tests a reward model’s ability to reason about chunk‑level consistency of action, appearance, and motion.

**Figure 3.** Composition of WorldReward-Bench. (a) Trajectory groups, (b) visual styles, and (c) source world models of the 760 video pairs.

Benchmark Performance

WorldReward’s agreement with human preferences on WorldReward‑Bench is reported.

Recall that WorldReward tackles the camera‑trajectory alignment problem by performing pairwise, chunk‑level reward reasoning that evaluates both action consistency and visual quality. This section quantifies how well it matches human preferences on the WorldReward‑Bench.

WorldReward‑RL attains the highest long‑term combined‑action accuracy, reaching 78.84%.

Table 4 shows the long‑term combined‑action column where WorldReward‑RL outperforms HY‑WorldPlay 1.5 and WorldCompass.

**Table 3.** Agreement with human preferences on WorldReward-Bench (%). Three-way accuracy for action consistency (Act.), appearance quality (App.), and motion quality (Mot.); Tie is an explicit label. Best and second-best are bold and underlined; - marks a dimension the predictor does not model.

**Table 1.** Generalization across trajectories and visual domains. Across trajectory groups, WORLDREWARD achieves the highest action and motion agreement for translation, rotation, and compound controls, together with the highest appearance agreement on rotation and compound trajectories. The strong performance on compound trajectories is particularly notable, as these cases require the model to associate multiple consecutive controls with their corresponding visual changes, indicating that its action judgments remain reliable beyond isolated movements. Across visual domains, WORLDREWARD leads all three dimensions on game/anime content, achieves the highest action and motion agreement on photorealistic content, and attains the highest appearance agreement on artistic content, remaining competitive on the other dimensions. Together, these results indicate that WORLDREWARD generalizes reliably across varied trajectories and visual domains.

RL Post-Training Results

Post‑training boosts combined‑action accuracy by up to 2.78 points.

WorldReward post‑training improves combined‑action accuracy over the baseline.

Table 4 reports a gain of up to 2.78 points across three generation horizons.

It aligns generated video frames with target camera motions by evaluating pairwise action consistency and visual quality.

**Figure 4.** Qualitative comparison with HY-WorldPlay 1.5 [9]. Top: HY-WorldPlay 1.5; bottom: WorldReward-RL (labeled WorldReward). Icons mark the translation key and rotation direction active at each frame.

**Figure 5.** Qualitative comparison with WorldCompass [10]. Top: WorldCompass; bottom: WorldReward-RL (labeled WorldReward). Layout as in Figure 4.

**Figure 6.** Pairwise evaluation of post-trained world models. Left-win / tie / right-win rates (%) judged by WorldReward, GPT-5.5 [23], and human annotators for action consistency (top) and visual quality (bottom) on the 200 human-study pairs. WorldReward-RL denotes HY-WorldPlay 1.5 post-trained with WorldReward.

Ablations and Limitations

Ablations show annotation refinement, visual evidence, reasoning supervision, and combined rewards each boost WorldReward performance.

Human review raises the average agreement to 77.33 % on WorldReward‑Bench.

Table 5 shows the progression from 68.69 % (distillation) to 75.94 % (agent‑harness QC) and finally 77.33 % after human review.

Structured visual evidence ablation demonstrates three complementary components. Removing the source image hurts appearance quality by 2.37 points, while preserving action and motion scores. Dropping the frame‑grid overview causes the largest average decline ( 2.48 points) and reduces action and motion agreement by 2.71 and 3.29 points respectively. Eliminating the action‑level panels primarily lowers action agreement by 2.82 points, with a smaller 1.35‑point drop in motion quality.

Reasoning supervision ablation isolates the benefit of structured supervision. A model trained with only preference labels attains 72.47 % average agreement. Adding an overall comparison between video chunks lifts the average to 74.94 %, a 2.47‑point gain. Incorporating per‑video analysis further improves average agreement by 2.39 points, reaching the highest scores across all three dimensions.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers