PAWBENCH: How Far Are We From Probabilistically Aligned World Modeling?

Yuandong Pu, Le Zhuo, Sayak Paul, Gabriel Jorge Menezes, Avram Đorđević, Shiyang Li, Yifan Zhou, Bin Fu, Wenlong Zhang, Junjun He, Yu Qiao, Yihao Liu, Jingbo Xing, Xi Chen

PAWBench evaluates whether video generators capture the true distribution of possible physical futures.

Do current video generation models accurately represent the probability distributions of physical outcomes, or do they merely produce single plausible-looking trajectories?

Current video generators are often treated as world models, but they are evaluated only on the plausibility of single, isolated video clips. This ignores whether a model correctly captures the range and relative likelihood of multiple physically valid outcomes from the same starting point. The authors introduce PAWBench, a diagnostic benchmark that uses repeated rollouts to map generated videos to terminal physical outcomes. It measures "probabilistic alignment" by comparing these empirical distributions against known reference probabilities or valid outcome sets. Across eleven state-of-the-art models, no system consistently matches reference probabilities or recovers the full range of valid futures. Plausible individual videos do not imply a correctly aligned underlying distribution.

Paper Primer

PAWBench formalizes probabilistic alignment as a distributional criterion: a world model must reproduce the conditional distribution of possible futures, not just a single plausible trajectory. The core move is to treat the generator as a stochastic sampler, using a fixed initial observation and action to generate $K=50$ rollouts, which are then mapped to discrete terminal outcomes via an automated rubric.

Current video generators fail to achieve probabilistic alignment.

Across 50 scenarios, no model simultaneously achieves accurate probability-mass matching (Calibration) and full support recovery (Coverage) while maintaining reliable scene-level performance. Observed calibration errors (Total Variation Distance) average 31.2, significantly exceeding the ~8.3 error expected from finite sampling alone.

The authors probe the source of this failure by intervening at three levels: language prompts, initial noise sampling, and model training. They find that while these interventions can steer individual rollouts, they do not reliably reshape the model's learned predictive distribution to match the physical reality of the scene.

Why is single-sample evaluation insufficient for world modeling?

A model can produce visually plausible videos while collapsing to a narrow subset of outcomes or assigning incorrect probabilities to valid futures, masking its failure to understand the underlying stochastic dynamics.

Does this benchmark rely on subjective human judgment?

No; PAWEval uses an automated rubric applied by a vision-language model to map rollouts to terminal outcomes. This protocol agrees with human judgment on 81.3% of clear outcomes, confirming the benchmark's reliability.

Introduction: The World Model Gap

We define probabilistic alignment and introduce PAWBench to evaluate video models' distributional fidelity.

Video generation models are often judged by the visual plausibility of a single rollout, but many physical processes are inherently stochastic. Consequently, a true world model must capture the full distribution of possible futures given the same observation and action, not just one plausible trajectory.

A model is probabilistically aligned when, for a fixed initial observation and action, the set of generated videos reflects both which futures are physically possible and how likely each one is.

**Figure 1.** **One plausible future is not enough.** A single rollout from a video generation model can appear physically plausible, yet repeated rollouts from the same initial observation and action may reveal a probabilistically unaligned world: outcomes concentrate on a narrow subset of futures instead of matching the valid stochastic support. PAWBench evaluates this induced distribution or support over repeated futures, rather than judging a model by one sampled video.

Using PAWBench and the PAWEval protocol, we evaluated eleven contemporary video generators across 50 scenarios. No model consistently reproduced the reference probabilities or recovered the full set of valid futures, revealing a systematic gap between visual plausibility and distributional accuracy.

The key distinction is that single‑sample plausibility does not guarantee correct distributional modeling; evaluating the full outcome distribution is essential for truly aligned world models.

The PAWBench Framework

PAWBench evaluates video models on calibrated and coverage alignment via repeated rollouts.

PAWBench measures whether a video generator reproduces the full distribution of physically possible outcomes for a fixed initial scene and action.

How does PAWBench differ from the usual single‑sample visual plausibility tests?

Single‑sample tests only check that a model can produce one plausible video; PAWBench repeatedly samples the same scenario and compares the full outcome distribution to a reference, revealing both probability misallocation and missing futures.

PAW‑Calibration checks whether the probabilities a model assigns to each possible outcome match a known reference distribution.

Why does the benchmark use Total Variation Distance instead of KL divergence?

TVD treats over‑ and under‑estimation symmetrically and yields a bounded, interpretable score (0–1), whereas KL can be infinite and is asymmetric, which complicates comparison across scenarios.

PAW‑Coverage verifies that a model’s rollouts actually exhibit all qualitatively distinct valid outcomes.

Fix a source image $x$ and an action prompt $a$ for the scenario.

Query the video model $K$ times with the identical pair $(x,a)$.

Map each generated video to a terminal outcome label $y \in Y$ using the scenario rubric.

Count occurrences of each $y$ and normalize to obtain the empirical distribution $\hat p_M$.

If a reference distribution $q$ exists, compute TVD$(\hat p_M, q)$ for PAW‑Calibration.

Otherwise, compute coverage = $\frac{|\{y \in Y : \hat p_M(y) > 0\}|}{|Y|}$ for PAW‑Coverage.

**Figure 2.** **PAWBench scenario taxonomy.** PAWBench covers eight mechanism groups under fixed initial observations and actions. PAW-Calibration contains calibrated probability scenarios with analytically specified reference distributions, while PAW-Coverage contains complex stochastic interactions evaluated by valid-support coverage. Each scenario fixes the source image and action prompt, then scores repeated model rollouts by their terminal outcomes.

**Figure 9: Benchmark statistics of PAWBench.** PAWBench contains 50 manually curated scenarios, evenly divided between PAW-Calibration and PAW-Coverage, across eight stochastic mechanism groups. Each scenario fixes one source image and action prompt and uses a compact terminal-outcome set, allowing repeated rollouts to be aggregated into an empirical outcome distribution.

Benchmark Performance

Key findings on how current video generators fall short of probabilistic alignment.

Current video generation models fail to achieve probabilistically aligned world modeling on PAWBench.

Table 1 shows that no model simultaneously attains low Calibration TVD and high Coverage.

**Table 1.** Benchmark Results

Current models generally fail to match reference distributions, even when individual samples look plausible.

Prompt Engineering and Future Sampling

We evaluate Prompt Engineering by varying prompts, measuring distributional alignment with PAW‑Calibration and PAW‑Coverage.

Prompt Engineering lets the user steer a video generator by phrasing a desired future in natural language, turning the model into a conditional sampler that selects among possible outcomes. It is like giving a photographer a precise scene description so the camera captures exactly the intended shot.

How is Prompt Engineering different from simply describing a scene in a caption?

Describing a scene only names what is visible, while Prompt Engineering encodes the desired future as a conditioning signal that the generator uses to bias its stochastic sampling toward that outcome, affecting the underlying probability distribution.

Fix a single initial observation and action, then query each VLM multiple times under identical conditions.

Map each textual prediction to a PAWBench outcome and aggregate the results into an empirical distribution (direct VLM sampling).

For Prompt Engineering (PE), let GPT‑5.5 predict a target outcome and construct a generator prompt requesting that outcome for each rollout.

For Oracle PE, directly specify the target outcome in the generator prompt, arranging targets to match the reference distribution.

Run the video generator with the constructed prompts, then compute PAW‑Calibration TVD and PAW‑Coverage for each model.

GPT‑5.5 selects outcomes A, B, A, C, D based on its internal distribution.

Prompts such as “Generate a video where the ball rolls left” (A) are sent to the generator.

The generator produces videos; outcomes are evaluated against the target list.

Success count = 3/5 → $60\%$ realized outcomes; the empirical distribution deviates from the uniform reference.

Even when the prompt correctly specifies a target, the generator may fail to produce it, highlighting the second source of error.

**Table 1.** Evaluation of VLM future sampling on PAWBench, comparing base models, models with prompt engineering (PE), and models with Oracle PE across PAW-Calibration (TVD) and PAW-Coverage metrics.

**Table 3.** Predicted outcomes do not substitute for target outcomes. The first row scores the outcomes selected by GPT-5.5 PE before video synthesis. Matched Base, PE, and Oracle PE results for four generators follow; all conditions cover 25 scenes per PAWBench track at $K = 50$.

**Figure 7.** Oracle PE often misses requested outcomes.

Causal Interventions and Training

We examine how noise coupling and causal interventions affect model exploration and calibration.

Recall that PAWBench evaluates whether a model captures the full distribution of plausible futures, not just a single plausible outcome. In this section we probe how noise sampling and explicit causal interventions shape that distribution.

A causal intervention modifies the underlying physical transition in a scene, forcing the model to update its predicted outcome distribution accordingly.

Apply a causal intervention: tilt the ramp upward by 5°, which reverses the gravitational bias.

Re‑compute the outcome probabilities: the left side drops to 0.4, the right side rises to 0.6.

Observe the model’s output distribution shift from (0.7, 0.3) to (0.4, 0.6) if it correctly incorporates the intervention.

The shift isolates the model’s causal reasoning ability because the language prompt stays identical.

C2C introduces negative dependence among a batch of initial‑noise vectors, encouraging them to explore different modes while each retains a standard Gaussian marginal.

Repulsive Gaussian coupling for $K\!=\!50$ noise vectors.

How does C2C differ from simply drawing more independent samples?

Independent sampling increases the number of draws but leaves them statistically identical, so many samples can still fall into the same mode. C2C actively pushes the $K$ samples apart, guaranteeing that each sample explores a distinct region of the latent space while preserving the per‑sample Gaussian distribution.

**Figure 4.** Models underreact to physically causal interventions and overreact to non-causal cues. Upper and lower bars show outcome distributions before and after intervention; panels (b) and (d) show the paired scenes. The pencil tilt is causal because it changes the physical transition, whereas the Galton-board text is non-causal because it leaves the transition unchanged.

**Figure 5.** Larger rollout budgets increase coverage but leave calibration largely unchanged. Conditional TVD (left) and valid-support coverage (right) as the rollout budget increases from $K = 1$ to $100$.

**Table 4.** Tab. 4 reports results across all 25 scenes in each PAWBench track. Across the scenes that pass in each condition, C2C lowers mean Calibration TVD and raises mean Coverage for all three generators The gains vary across mechanisms and do not consistently raise SPR. C2C therefore helps the 50 rollouts explore the model's existing possibilities more broadly, rather than changing the distribution learned by the model. We therefore turn next to the model itself.

**Table 5.** **Training mixtures reshape outcome mass.** TVD $\times 100$ is reported for the unadapted Base and five LoRA models trained on increasing proportions of left-fall examples.

Related Work and Limitations

We situate PAWBench among prior video‑generation benchmarks and outline its current limitations.

Video generators have increasingly been treated as world models, enabling planning by predicting action consequences across reinforcement learning, latent dynamics, and decision‑centric frameworks.

PAWBench shifts the focus from single‑sample visual plausibility to the distribution of possible futures, asking whether repeated rollouts from the same observation and action place probability mass on the correct outcomes.

Existing video‑generation benchmarks primarily assess visual quality, temporal consistency, or text‑video alignment, while physics‑focused suites probe plausibility and action following, but they usually evaluate individual generations rather than the full outcome distribution.

Calibration‑and‑coverage studies separate accuracy from probability assignment, and recent generative‑video uncertainty work demonstrates feasible confidence estimation.

CaliBench evaluates stochastic scenes with known reference distributions, reporting conditional TVD and miscalibration metrics, making it the closest prior work to PAWBench’s repeated‑rollout, discrete‑outcome approach.

PAWBench distinguishes two regimes: PAW‑Calibration, where reference probabilities are available, and PAW‑Coverage, where only the valid support set can be specified.

Limitations of PAWBench include its reliance on terminal outcomes, which omits trajectory‑level dynamics and intermediate physical processes.

Because estimates derive from a finite number of rollouts, larger sampling budgets improve reliability but increase evaluation cost without fixing biased model distributions.

The benchmark currently uses controlled, visually parseable scenarios, leaving longer‑horizon, interactive, and embodied environments for future study.

Future work should extend probabilistic alignment from terminal labels to richer state trajectories, develop more efficient rollout‑based estimators, scale to interactive settings, and create training objectives that explicitly learn calibrated future distributions.

The PAWEval Protocol

We detail the PAWEval protocol, its metrics, and supporting analyses.

Recall that PAWBench measures whether a model captures the full distribution of physically plausible outcomes rather than a single “most realistic” video.

PAWEval turns many generated rollouts into a distributional test by first assigning each rollout a readable terminal label and then comparing the resulting label distribution to a reference.

How does PAWEval differ from a simple per‑frame visual similarity score?

PAWEval ignores frame‑wise pixel differences entirely; it only cares whether the final observable outcome matches a predefined category. This makes the evaluation robust to stylistic variation while directly testing distributional alignment of physically meaningful outcomes.

For a given scene, generate $K=50$ rollouts under the same initial observation and action.

Apply the frozen outcome‑readout rubric to each rollout, producing a label $y_i\in Y$ or failure $\bot$.

Count failures $m=\sum_i\mathbf{1}[y_i=\bot]$; the scene passes if $m\le30$ (i.e., at least 20 readable outcomes).

For passing scenes, compute the empirical distribution $\hat p_{M,\text{readout}}(y)=\frac{1}{n_{\text{readout}}}\sum_i\mathbf{1}[y_i=y]$ where $n_{\text{readout}}=K-m$.

Measure conditional TVD between $\hat p_{M,\text{readout}}$ and the reference distribution $q$ (Equation [F1] from §1).

Measure conditional coverage $\text{Cov}_{\text{readout}}=\frac{|\{y:\hat p(y)>0\}|}{|Y|}$.

Average the per‑scene metrics across all passing scenes in a mechanism group; compute the track‑level average over the full 25‑scene track.

Record the scene‑pass rate $n_{\text{pass}}/25$ alongside the conditional metrics.

Rollout 1 → rubric returns $\text{Head}$; Rollout 2 → $\text{Head}$; Rollout 3 → $\text{Tail}$; Rollout 4 → $\bot$ (unreadable); Rollout 5 → $\text{Head}$.

Failures $m=1$ ≤ 30, so the scene passes the gate.

Readable count $n_{\text{readout}}=4$; empirical distribution $\hat p(\text{Head})=3/4=0.75,\ \hat p(\text{Tail})=1/4=0.25$.

Conditional TVD against the reference $q(\text{Head})=q(\text{Tail})=0.5$ is $\frac12(|0.75-0.5|+|0.25-0.5|)=0.25$ (25 %).

This toy example shows how a single unreadable rollout does not bias the TVD calculation, because the failure label is omitted from the conditional distribution.

**Figure 3.** PAWEval turns repeated rollouts into a distributional test. In this PAW-Calibration example, a rubric-based judge maps each readable, in-schema coin-toss rollout to Head or Tail. Aggregating these labels yields the empirical outcome distribution, which is compared with the reference probabilities rather than matching generated videos frame by frame.

Table 7 lists the outcome‑readout criteria (endpoint readability and schema membership) and the three trustworthiness audit criteria (action completion, object continuity, physical‑process compliance), together with their failure conditions.

**Figure 10.** PAWEval prompt scaffold. The scene-specific outcome rubric assigns a terminal label or an outcome-readout failure. A separate trustworthiness rubric checks action execution, physical process, and object continuity without changing the PAWBench score. The placeholders {`evidence_summary`} and {`rubric_text`} are filled with sampled-frame evidence and the scene-specific rubric.

Observed TVD across the eleven video generators averages 31.2 % and exceeds the matched Monte‑Carlo baseline (average 8.33 %).

Figure 12 shows each model’s red‑diamond TVD lying to the right of the blue‑shaded 99th‑percentile interval; Table 8 confirms the gap under alternative aggregations.

**Figure 12.** **Observed TVD exceeds the matched finite-sample baseline.** Blue intervals show the 2.5th to 99th percentiles obtained by sampling from the reference distributions with each model’s passing scenes and readable sample counts. Circles mark the simulated means, and red diamonds mark the observed model TVDs. The aggregate row first averages passing scenes within each model and then weights the eleven models equally.

Table 8 repeats the matched analysis with pooled passing cells and with a stricter $n_{\text{readout}}\ge30$ threshold; the conclusions about calibration remain unchanged.

Full causal and non‑causal controls (Figures 13–16) compare a base scene to its causal variant (which changes the physical transition) and to a non‑causal variant (which preserves the transition). The paired bars show how the outcome distribution shifts only under causal interventions.

**Figure 13.** Causal-state controls across video generators and direct VLM future samplers. Each panel compares the outcome distribution for a base scene (upper bar) with that for a causal variant (lower bar), whose reference distribution changes. Bars report distributions conditional on readable, in-schema outcomes; gray bars marked / denote scenes that fail the outcome-readout gate.

Human Study and Alignment

PAWEval aligns with human judgments on 81.3% of comparable videos.

PAWEval matches the decisive human label on 81.3 % of comparable videos.

Table 9 shows 722 exact matches out of 888 comparable videos.

The analysis only includes videos with decisive human labels and in‑schema PAWEval predictions, holding all other factors constant.

**Table 9.** PAWEval–human agreement on comparable videos. Agreement is the exact canonical-label match when both PAWEval and the seven-vote human panel provide a clear physical outcome. Consensus rows are descriptive strata of the 888-video comparison set.

Extended Intervention Experiments

We intervene on language, noise, and training data to probe their effect on stochastic video generation.

We intervene at three points where a video generator’s future distribution can be shaped: the language prompt, the initial diffusion noise, and the training data composition.

D.1 Language‑side diagnostics and interventions examine how the textual controller influences outcome distributions.

PE asks a language model to predict a plausible outcome and then embeds that prediction in the video generator’s prompt; Oracle PE writes the target outcome directly into the prompt.

Why does PE increase Calibration TVD while Oracle PE decreases it?

PE relies on the language model’s own outcome distribution, which is misaligned with the physical reference; Oracle PE replaces that distribution with the ground‑truth targets, so the resulting rollout distribution matches the benchmark more closely.

D.2 Coupled noise sampling (C2C) tests whether correlating the initial diffusion noises improves coverage without altering any single rollout’s marginal distribution.

C2C replaces independent Gaussian noises with a set that sums to zero, preserving each noise’s marginal $\mathcal{N}(0, I)$ while enforcing negative correlation across the group.

How does C2C differ from the standard independent noise sampling?

Independent sampling draws each $\varepsilon_{g,i}$ independently, so the joint distribution is factorized. C2C forces the group to have zero mean and introduces negative covariance, which spreads the sampled futures more evenly across the finite budget while leaving each individual rollout’s marginal unchanged.

D.3 Training‑distribution intervention varies the left/right composition of the training set to see whether the generator internalizes the relative frequencies of futures.

We create five LoRA‑adapted models, each trained on a dataset with a different left/right outcome ratio, while keeping all other training factors constant.

Does the model recover the exact training ratio in its output distribution?

No. The observed left‑fall frequencies increase with the training ratio but do not match it precisely; the model learns a biased prior that moves in the same direction as the training mixture.

The left‑leaning pencil scene shows a similar monotonic increase, confirming that the intervention’s effect is not scene‑specific.

**Figure 14.** **Non-causal routing controls.** The paired scenes change an irrelevant cue while preserving the physical transition and reference distribution. Upper and lower bars show the base and cue-perturbed outcome distributions, respectively; gray bars marked / denote scenes that fail the outcome-readout gate.

**Figure 15.** **Non-causal draw and text controls.** Distractor appearance in the blind-draw scene and an outcome-suggestive sign in the Galton-board scene leave the reference distribution unchanged. Several systems nevertheless move substantial probability mass between the paired conditions.

**Figure 16.** Non-causal dispenser and coin controls. Outcome-suggestive text or an irrelevant appearance change leaves the underlying chance process and reference distribution unchanged. Upper and lower bars show the base and cue-perturbed distributions conditional on readable, in-schema outcomes.

Across all three intervention families, the experiments expose distinct failure modes: language prompts misrepresent outcome distributions, independent noise limits coverage, and training mixtures bias the generator toward learned directional priors.

Qualitative Examples and Failure Cases

Qualitative case cards illustrate PAWBench successes and typical failure modes.

This appendix adds 13 qualitative case cards that complement the aggregate PAWBench results. Each card presents a short movie strip from a single rollout together with its scene, model, generation condition, instruction, and observable behavior.

**Figure 17.** Prompts for the two language-side probes. The VLM scaffold requests one plausible future without exposing outcome labels or probabilities. The target-conditioned I2V scaffold instead adds one terminal outcome to the original scene prompt. Braced fields are filled at runtime.

These cards demonstrate how PAWEval parses terminal outcomes and why visually plausible videos can still fail to complete the intended physical process. The collection includes both successful physical trials and typical failure modes such as action‑execution errors, breaks in object continuity, inconsistent trajectories, apparatus instability, and clips that are readable yet questionable.

Benchmark Scenario Details

Details of the benchmark scenarios, task taxonomy, and quality‑review process.

Each PAWBench scenario fixes a source image, an atomic action prompt, and a finite set $Y$ of valid terminal outcomes. The scenario also defines how outcomes are read out and, when possible, a reference distribution $q$ over Y. These elements are finalized before any model is evaluated.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers