VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement

Wenzhuo Xu, Yuchen Zhu, Chongjian Ge, Xuan Shen, Jing Shi, Jason Kuen, Yongxin Chen, Molei Tao, Christopher McComb, Noelia Grande Gutiérrez, Jiuxiang Gu

VeriPhy is an auditable video-evaluation system that compiles prompts into typed physical obligations and traces every verdict to localized evidence.

How can we move beyond scalar quality scores to provide auditable, evidence-based physical verification of generated video?

Video generation models often produce visually coherent clips that violate basic physical laws, yet current evaluators provide only opaque scalar scores that fail to localize or explain specific failures. VeriPhy addresses this by compiling prompts into a statically validated plan of physical obligations, using frozen specialist tools to gather evidence, and composing those measurements into a three-valued verdict with full provenance. On a core benchmark of 149 clips, VeriPhy accounts for 228 of 304 human-annotated defects, outperforming a question-decomposition baseline while providing an auditable evidence chain for every decision.

Paper Primer

The system operates as a constrained reason-act-observe loop. A text-only planner first decomposes the prompt into a typed measurement plan, which then gates and scopes calls to frozen specialist instruments—such as segmentation, tracking, and depth estimation—to verify physical consistency.

VeriPhy achieves higher recall on physical flaw detection than standard question-decomposition evaluators.

Comparison against a published question-decomposition evaluator on a 149-clip core corpus with 304 annotated defects. VeriPhy identified 228 defects compared to 164 for the baseline.

Unlike monolithic models that output a single quality score, VeriPhy produces a localized contradiction packet for every failure. This packet links the verdict to the specific prompt span, the temporal window of the failure, and the raw evidence records from the specialist tools.

Why is a "plan-before-act" approach necessary for physical verification?

It ensures that every measurement is tied to a pre-declared physical obligation, preventing the system from "hallucinating" justifications after observing the video and ensuring that all verdicts are traceable to specific, scoped evidence.

How does VeriPhy handle cases where evidence is missing or ambiguous?

The system uses a three-valued logic (supported, contradicted, or unknown). If a specialist tool fails to return usable evidence or a window is unavailable, the claim defaults to "unknown" rather than forcing a binary pass/fail judgment.

VeriPhy shifts video evaluation from opaque scalar ratings to auditable evidence chains, providing a concrete interface for future closed-loop physical refinement of generative models.

Introduction: Beyond Scalar Quality

We expose the need for auditable, physics‑grounded video evaluation beyond opaque quality scores.

Current video generators produce visually appealing clips, yet visual plausibility offers no guarantee that the underlying physics are correct. Objects may float, collisions can be silent, or sounds may precede impacts, leading to subtle but critical failures when such videos are used for downstream embodied tasks.

Quality scores alone cannot reveal which physical claim a video violates or where the failure occurs; we need a traceable chain that ties each verdict to concrete, typed evidence.

VeriPhy operationalizes this motivation: a text‑only planner first compiles the prompt into typed obligations and a statically validated plan. During execution, observations gate only the calls declared by the plan, dispatching frozen specialists such as Segment Anything Model (SAM) 3 for grounding and tracking, identity‑based counting, eleven track‑based physical measurements, monocular depth, OCR, and audio‑event detection.

**Figure.** Overview of the VeriPhy framework. The top section contrasts a baseline approach with the VeriPhy auditable execution pipeline. The main central panel illustrates an evidence trace of a basketball bouncing, showing the alignment between visual observations, expected physical behavior, and audio waveforms. The right-hand panels provide zoomed-in details of specific events: (0) initial spin, (1) floor contact, and (2) wall impact/penetration analysis. The bottom text provides a sample prompt used for the generation.

The shift from aesthetic ranking to physical verification enables verifiable, failure‑aware video evaluation.

Physics-Guided Simulation Pipeline

The section details the refinement loop that validates physics‑driven controls and how they steer video generation.

Current video generators produce plausible motion but have no explicit physical target, making the resulting trajectory opaque and inseparable from appearance.

The loop repeatedly simulates a candidate scene, checks whether it satisfies the event’s physics predicates, and asks a language model to revise the specification until the predicates hold.

Step 1: Solve(S⁽⁰⁾) → $\tau$⁽⁰⁾ (missed by 0.4 m).

Step 2: Summarize($\tau$⁽⁰⁾) → “missed by 0.4 m”. Author revises S → S⁽¹⁾.

Step 3: Solve(S⁽¹⁾) → $\tau$⁽¹⁾ (missed by 0.1 m).

Step 4: Summarize($\tau$⁽¹⁾) → “missed by 0.1 m”. Author revises S → S⁽²⁾.

Step 5: Solve(S⁽²⁾) → $\tau$⁽²⁾ (all predicates satisfied).

The loop converges quickly because each revision incorporates the precise error signal from the validator, turning a blind simulation into a guided search.

How does this refinement loop differ from a naïve “run‑once” simulation?

In a naïve run‑once, the scene is simulated a single time and any predicate violation is ignored, so the resulting motion may be physically invalid. The refinement loop closes the gap by feeding back the failure report to the authoring model, guaranteeing that the final rendered control satisfies the prescribed physics.

Once a validated trajectory is obtained, the Render map converts it into a depth control video C that encodes per‑pixel distance as intensity.

If the percentile computation fails (e.g., all pixels are background or percentiles coincide), Render falls back to a binary silhouette instead of a depth map.

Encode the appearance prompt p with a frozen text encoder.

Encode the depth control video C and the all‑white mask M into control features.

At each transformer layer ℓ and diffusion step i, add the scaled control residual s $W^{(ℓ)}$ $z_{ctrl}$ to the hidden state.

Iteratively denoise the latent representation from high noise ($\sigma_{i}$ large) to low noise ($\sigma_{i}$ → 0) using the UniPC schedule.

Decode the final latent into a video clip \hat V.

Because the control is injected at every diffusion step, it may become redundant once the trajectory has settled; we therefore probe the point at which the control can be withdrawn without harming fidelity.

Control‑withdrawal probe: stop attending to control tokens after step i*

The corresponding noise level $\sigma_{r}$el = $\sigma_{i*}$ is reported; different schedules or shift values change $\sigma_{r}$el even for the same i*, highlighting the schedule‑dependence of the withdrawal point.

**Figure 2.** Physics-guided generation. A *physics request* becomes a scene that is simulated and validated, revising on failure (red loop); the validated motion is a depth control that, with an *appearance prompt*, conditions the generator. Right: depth control and generated clip at matched timestamps, tracing the ball's path.

The VeriPhy System

VeriPhy turns opaque video scores into a traceable, three‑valued verdict.

Opaque video‑quality scores give no insight into why a clip fails a physical claim; VeriPhy replaces them with an auditable critic that produces a three‑valued verdict anchored to concrete evidence.

VeriPhy first compiles the prompt into a typed measurement plan, then runs only the declared tools, finally aggregates deterministic evidence into a supported/contradicted/unknown verdict.

Specialist instrument returns $W_A$ with timestamps $[1.00,1.05]$ s (duration 0.05 s).

Specialist instrument returns $W_B$ with timestamps $[1.20,1.30]$ s (duration 0.10 s).

The deterministic compiler computes slack $\delta = \tfrac{1}{4}\times\min(0.05,0.10)=0.0125$ s.

It checks the order condition $A_{\text{end}} \le B_{\text{start}}+\delta$, i.e. $1.05 \le 1.20+0.0125$, which holds.

The sub‑check state is set to $S$ (supported) and propagates to the claim verdict.

The quarter‑window slack prevents both “before” and “after” from being true when windows overlap, ensuring a single directional decision.

How does VeriPhy differ from a conventional black‑box video scorer?

A black‑box scorer outputs a single scalar without explaining why; VeriPhy emits a structured plan, runs only pre‑declared tools, and returns a three‑valued verdict together with a full evidence trace that shows exactly which measurement caused each decision.

Plan: the text‑only planner parses the prompt, creates a surface plan $\Pi$, and validates it statically.

Act: each declared operation dispatches to a frozen backend (specialist instrument or semantic verifier).

Observe: the deterministic aggregator consumes the returned measurements, applying logical rules and slack thresholds.

Verdict: the three‑valued clip decision $Y$ is emitted, together with a localized contradiction packet for any failed claim.

Table 1 lists the nine surface operations (e.g., judge, before), while Table 2 enumerates the frozen backends (including SAM 3 for grounding subjects).

**Figure 13** The same loop scored by VeriPhy’s own physical metric. Every quantity here is self-assessed: the critic that triggers a rewrite is also the judge of its result, so these quantities show the loop is internally consistent—targeted (it acts on 87 of 589 clips, 14.8%), effective on its targets (63/87 raise $p_{plausible}$, 49 of the 65 acted clips first judged implausible flip to plausible, mean $\Delta p_{plausible}$ +0.524), and low-collateral (8/502, 1.6%)—and move the overall mean $p_{plausible}$ from 0.797 to 0.862. Each bar is drawn against its own denominator. They are not an independent confirmation and should be read against Table 6, where an independent rater registers no corresponding change, including on the physical dimension; every value is recomputed from the per-clip records in Table 11.

The figure illustrates a four-step process for verifying a temporal claim about dominoes. The top section is a flowchart with four steps: 1. Plan, 2. Gate, 3. Measure, and 4. Decide. The middle section is a timeline from 0 to 8 seconds, showing event windows for A [1.62, 1.67] s and B [2.33, 7.96] s, with corresponding video frames below. The bottom section provides the mathematical condition for the decision: $\delta = \frac{1}{4} \times \text{shorter window} = 0.0125$ s, and the claim is supported if $A_{end} \leq B_{start} + \delta$, which evaluates to $1.67 \leq 2.33 + 0.0125$.

Experimental Evaluation

We evaluate VeriPhy’s recall, training gains, and refinement impact across benchmarks.

VeriPhy recovers 75 % of human‑annotated flaws on the core benchmark.

Finds 228 of 304 flaws (191 whole, 37 partial) across 149 clips.

The refinement loop improves 72.4 % of flagged clips and flips 49 of 65 implausible verdicts, while regressing only 1.6 % of unflagged clips.

Measured on 589 Cosmos3‑Nano clips; VeriPhy’s own metric shows targeted gains.

**Figure 4.** Tracked ball path in nine clips at one seed, against the same depth control throughout. Each box is a single 848 × 480 frame, $x$ left to right and $y$ downward; each dot is one frame, sized by the tracked region's area and shaded from first frame to last. The pale band is the control video's ball, in frame for 63 of its 81 frames; the number in each box is that clip's sideways travel. The three prompts differ only in the closing sentence beside each row; the red row asks for the motion the control does not perform. Without the control the path follows the clause and the framing shifts with it; with the control throughout, all three clauses reproduce its arc; with the control over the first ten steps the path holds the arc and departs at the landing.

**Figure 5** The release-point sweep: one clip per point, three scenes (including a ball rolling off a ramp), scored separately over the airborne window and the window after the simulated object first reaches the ground. The horizontal axis is the noise level at which the control was switched off, high noise on the left. Colour marks the window (blue airborne, rose after contact) and line style the axis (solid across-frame, dashed up/down); retention is the correlation between the generated object’s tracked path and the simulation’s, defined in the text. The thrown ball’s after-contact agreement falls below the threshold in both directions at the earliest withdrawal, and the vertical component is the later of the two to recover. Guides mark the commitment band and the backbone’s expert switch; the grey band marks the region below the 0.7 threshold fixed before scoring.

**Figure 6** The same mechanism across two scenes with very different requested appearance—an indoor studio and an outdoor field—each shown as its depth control above and the resulting generated clip below. Timestamps are per scene, chosen from the turning points of its tracked vertical path, so each column falls within a moving interval. Appearance changes completely with the prompt while the motion stays the simulation's: for the thrown ball the generated and control paths agree to a correlation of 1.0000 horizontally and 0.9998 vertically, with a median disagreement of 0.7 pixels across and 1.1 pixels down, over the 63 frames in which the ball is measurable in both before it leaves the frame.

**Figure 9** Localized flaw recall vs. cost per evaluator, over the same 149 clips and 304 flaws (Section 5.1.2). Horizontal axis: mean model and tool calls per clip; vertical: recall. Up-and-left is better. Full comparison: Table 8 in Section A.6.

**Figure 10.** Six flaws identified through specialist measurement, one per row. Each row gives the specialist that answered, a five-frame filmstrip sampled uniformly across the clip, the human annotator's sentence verbatim, the specialist's own required-versus-measured pair with its evidence, and the resulting claim state. Displacements are normalized by the subject's radius, so no scene calibration is required. In the text case, the boxed banner region of the last frame is magnified in place, showing the render's own glyphs. Measurement text is copied from the run's records.

**Figure 11** Physion-Eval physical-implausibility flag rate for every evaluator of Table 9, sorted high to low, with the untrained-human reference (light) and this work’s evidence-conditioned rate (red). This work’s rate is measured on the first 601 release clips; the benchmark’s critic rates are transcribed from its Figure 4 over its full evaluation set, so the comparison is descriptive rather than common-protocol.

**Figure 12** Flaws found against the number of experience clips the lessons were distilled from, over the same 502 held-out flaws in every condition. The dotted line marks the untaught condition, 340 of 502 (67.7%). Whiskers are 95% intervals on the change from untaught, in flaws, computed from the flaws that switch status, since the conditions score the same flaws. Every taught condition exceeds the untaught one, while pairwise comparisons among the taught conditions return $p \geq 0.8$.

**Figure 14.** The refinement loop read two ways, on the same 589 Cosmos3-Nano clips. **Left:** by VeriPhy’s own physical metric—the quantity the rewrite optimizes—the loop is targeted and low-collateral: of the clips it flagged, 63/87 improve and 49/65 flip from implausible to plausible, against 8/502 unflagged clips that regress. **Right:** the same clips scored by VideoPhy’s independent AutoRater show no movement—each dumbbell from baseline (navy) to rewrite (red) collapses to a point at every threshold, on both the full set and the hard subset, in the semantic *and* the physical dimension. Every number is transcribed from Table 6 and the self-assessed counts of Figure 13. The improvement is thus visible on the critic’s own metric and not on the one rater independent of it. Clips in Figure 15.

Temporal Predicates and Benchmark

We formalize temporal predicates, noise schedules, trajectory metrics, and the critic dataflow used in our ablations.

This appendix supplies the precise definitions behind the temporal and noise components that our ablations manipulate.

Temporal predicates decide whether one interval occurs before, after, or contains another, allowing a tolerant “quarter‑window” margin.

Why use a quarter of the shorter window as slack instead of a fixed time?

Because intervals vary widely in length across clips; a fixed time would be too strict for long windows and too lax for short ones. Scaling the slack to the shorter interval preserves relative tolerance.

The withdrawal point is defined by a relative noise level $\sigma_{\text{rel}}$, which follows a smooth decay schedule rather than a discrete step index.

What would happen if we used the step index $i^{*}$ directly instead of the relative noise $\sigma_{\text{rel}}$?

Using the raw step index would make the withdrawal point depend on the absolute number of steps, which varies with dataset size and training schedule; the relative noise abstracts away those details and yields a comparable stopping criterion across runs.

These metrics quantify how closely a generated clip’s tracked trajectory follows the simulated ground‑truth, focusing on horizontal correlation before and after first ground contact.

Why is a Pearson correlation used instead of a simple Euclidean distance?

Correlation captures linear alignment of motion regardless of scale, which is important when the generated and simulated trajectories may differ in speed but still follow the same path; Euclidean distance would conflate speed differences with shape mismatches.

The critic parses a textual plan, executes typed checks on video frames, and aggregates the results into a deterministic verdict for each claim.

Algorithm 1 VeriPhy critic – deterministic compilation and roll‑up.

What would break if the fallback verification step were omitted?

Without the fallback, any claim that the dense pass marks as “supported” could remain falsely supported if the dense model missed subtle contradictions; the fallback ensures that contradictory evidence can overturn a spurious support, preserving soundness.

Related Work

We situate VeriPhy among prior approaches to physically‑aware video generation, evaluation, and execution harnesses.

Prior work splits into three strands: (1) physically‑aware generation pipelines that condition diffusion models on geometric cues, (2) structured evaluation methods that decompose prompts into explicit checks, and (3) execution harnesses that orchestrate planning, observation, and verification.

Methods condition diffusion generators on depth maps, segmentation masks, reference frames, motion fields, or other spatiotemporal signals to steer appearance synthesis.

Evaluation pipelines move beyond scalar quality scores, using learned vision‑language models, multidimensional ratings, and explicit prompt‑decomposition checks that produce evidence traces.

Harnesses separate orchestration from model judgment, making planning, routing, validation, and trace logging explicit, and managing persistent state across episodes.

VideoScore2 extends scalar video quality scoring by attaching an evidence trace that records which physical predicates were satisfied.

Framework and Roadmap

The section maps the auditable state, read/write maps, and roadmap phases that bound current capabilities.

This limits section enumerates the concrete objects that make the system auditable, explains the read‑write maps that bound current capability, and lays out a phased roadmap from the built components to future extensions.

$O$ is a catalog of frozen measurement functions—each takes a video and typed arguments and returns a physical metric together with an explicit cost node $c(o)$.

$M$ stores human‑readable priors distilled from past episode traces, separating recorded evidence from current belief.

$K$ is an open port that admits retrieved facts or social cues, but only after a candidate‑and‑pruning gate validates their utility.

The read operator consumes the current state $S$ and a prompt–video pair $(p,V)$ to produce a fully provenance‑rich trace $T$.

When $Y$ contains contradictions, the outward channel maps the packet $F$ to a localized edit $\Delta_{\text{ctrl}}$ of the generator’s control signal and regenerates the video.

All state components ($O$, $M$, $K$) and every trace $T$ are human‑readable diffs, so a reviewer can trace any verdict back to its origin.

The system can improve its own state $S$ without altering any neural weights, by distilling experience into $M$ and operators into $O$.

When a plan fails to capture a human‑recorded flaw, the failure is turned into a new entry in $M$ (e.g., a physics‑failure prior).

Human reviewers audit each trace $T$ and any proposed state edit before the system deploys a regenerated video or updates $S$.

**Table 10** Recall grouped by annotator severity, clip duration, number of annotated flaws in the clip, and difficulty stratum, on the run of Table 3. Severity is the annotator's own 1–5 judgement of how badly the clip fails.

**Table 11.** Every number in Figure 13 and Table 6 recomputed independently from the run's per-clip raw files, against the reported value. All reproduce exactly. Two definitions are pinned by the recompute: collateral is a verdict flip (plausible→implausible) on a non-flagged clip—8/502; scored instead as any decrease in $p_{plausible}$ it would be 31/502, so the stricter reading is the one reported. The AutoRater baseline column of Table 6 is not in this check: the released bundle synced only the rewrite-side per-clip scores and a 12-clip baseline subset, so the 600-clip baseline means are corroborated by the run log rather than recomputed here. Baseline $n = 600$ and rewrite $n = 589$ (the 11 absent clips did not complete regeneration or re-scoring), so the two columns of Table 6 rest on slightly different denominators.

**Table 12.** Each pillar is one component of the same loop. Only Phase 0 rows denote implemented artifacts or measured analyses; solid arrows in Figure 22 correspond to them, dashed arrows to the roadmap rows, which are unmeasured.

Conclusion

The conclusion reflects on achievements, current limits, and future extensions.

Physical faithfulness in generated video is tackled in three parts: steering what gets generated, measuring whether obligations are met, and providing a ground‑truth source for those measurements.

This report contributes one artifact for each part—a simulation‑driven controllable generator, a benchmark of human‑annotated defects, and the VeriPhy critic that compiles prompts into typed obligations before any frame is read.

The central design decision is to fix the plan before any pixel is observed, guaranteeing that every measurement can be traced back to the obligation that requested it and that each verdict carries its supporting evidence.

On a development core of 149 clips with 304 human‑marked defects, VeriPhy accounts for 228 findings, compared with 164 for a published question‑decomposition evaluator using the same clips and model.

The monolithic baseline reaches 222 findings, showing that recall alone does not explain VeriPhy’s advantage; the advantage lies in the auditable evidence trace.

Distilling lessons from experience clips improves recall from 340 / 502 to 375 / 502 (net + 35 at $p$ = 3 × 10⁻⁴) without any retraining, and the gain saturates after only 25 distilled clips.

The outward channel is less developed; feeding a verdict back as a prompt rewrite and resampling under the same control closes the critic‑to‑generator loop on Cosmos3‑Nano.

On the rewrite metric the loop fires on 87 / 589 clips (14.8 %), improves 63 of them, flips 49 implausible verdicts to plausible, and regresses only 8 of the 502 untouched clips (1.6 %).

An independent rater observes no corresponding movement, localizing the remaining open problem to the word‑only rewrite channel.

The next direction is to incorporate audio, which can resolve physical implausibility that vision alone cannot capture, enabling synchrony claims that bind visual contact windows with audio onsets.

The architecture already includes a sound‑event specialist, but this capability is not exercised in the current evaluation.

Primary limitations stem from the simulator‑derived control signal, which bounds scene diversity and leaves the mapping to appearance unverified; the corpus is recall‑only, with single‑annotator defects and machine‑derived tags.

Spatial‑relation predicates and negative‑event verification remain limited, and binding across heterogeneous tools is a central failure mode despite identifier consistency enforcement.

Four additional limits bound Section B.7, including an unevaluated acceptance gate, a rule without proxy history, a taught planner that reads 1.76 × as many claims, and non‑accumulated lesson induction.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers