Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

Runmao Yao, Kairui Hu, Yukang Cao, Ruisi Wang, Shulin Tian, Ziang Cao, Weichen Fan, Ziqi Huang, Yuhao Dong, Hao Li, Zhaoxi Chen, Zhongang Cai, Lei Yang, Ziwei Liu

Apple-$\pi$ is a diagnostic benchmark that evaluates whether video models reason through physical laws or merely mimic visual plausibility.

Can current video generation models actually reason about physical laws, or are they merely generating visually plausible but physically incoherent sequences?

Current video models are often called "world models," but existing benchmarks only check if their output looks plausible, not if they actually follow physical laws. When a model fails, it is impossible to tell if it misread the scene, chose the wrong law, or stumbled in the calculation. Apple-$\pi$ forces models to perform scientific reasoning by breaking tasks into three stages: Perception (identifying entities), Formulation (selecting the governing law), and Deduction (simulating the trajectory). It uses infographic-annotated first frames and chain-of-frames prompting to make the model's internal reasoning trace auditable. Even the best video models score only 0.473, revealing a "reasoning funnel" where performance drops sharply as tasks move from simple perception to complex, law-consistent temporal deduction.

Paper Primer

The benchmark uses a dataset called Orchard, which contains 400 cases across ten classical mechanics tasks. By standardizing objects to four primitive solids and providing infographic overlays, the authors isolate physical reasoning from semantic noise and reference-resolution errors.

The core mechanism is a stage-resolved protocol: models must pass Perception (text/graphic), Formulation (symbolic/state prediction), and Deduction (full trajectory). This structure acts as a diagnostic filter: if a model fails, the authors can pinpoint whether the breakdown occurred at the initial binding of variables or the final execution of the law.

Video models exhibit a progressive reasoning bottleneck where early success in perception does not guarantee law-consistent temporal dynamics.

Stage-resolved scores show a consistent decline from Perception to Formulation to Deduction across all tested video models. The best video model (Seedance 2.0) achieves an average score of 0.473, while unified understanding-generation models like GPT Image 2 reach 0.704.

Why use infographic-annotated frames instead of standard text prompts?

Text prompts make binding physical quantities to specific objects indirect and ambiguous. Infographics spatially ground parameters to their visual referents, ensuring the benchmark tests physical reasoning rather than the model's ability to parse complex natural language instructions.

Does this benchmark prove that video models are incapable of physical reasoning?

No; it demonstrates that current models rely on visual priors rather than explicit law-grounding. The authors find that while large-scale training provides useful physical intuition, reliable performance requires explicit understanding modules and reasoning-oriented post-training.

Researchers should shift from evaluating video models as "plausible generators" to "auditable reasoners." Future progress requires explicit mechanisms for physical understanding rather than just scaling video synthesis.

Introduction: The Physical Intelligence Gap

We expose the gap between visual plausibility and law‑grounded reasoning in video models.

Current video‑generation benchmarks measure only the visual plausibility of the output, leaving the model’s internal physical reasoning unexamined.

The history of physics moves from Aristotle’s intuitive description of a falling apple to Newton’s law‑grounded deduction of celestial motion, exemplifying the shift from intuition to explicit law.

Modern video models appear to emulate this trajectory at scale, but without a benchmark that audits the reasoning chain we cannot tell whether they truly invoke physical laws or merely produce plausible motion.

Physical Intelligence is the ability of a model to extract quantitative physical quantities from a scene, select the governing law, and generate dynamics that obey that law.

Apple‑$\pi$ is the first benchmark that anchors video‑model evaluation in explicit physical laws, exposing where models succeed or fail in the reasoning chain.

Its Orchard component provides 400 videos covering ten canonical mechanics tasks, organized into single‑law and multi‑law groups to enable confounder‑free diagnosis and compositional generalization.

The Benchmark Protocol asks models to follow a three‑stage chain‑of‑frames: Perception (identify quantities), Formulation (select the governing law), and Deduction (generate law‑consistent dynamics), using an infographic‑annotated first frame as a prompt.

The Evaluation Suite combines MLLM‑based subjective scoring with physics‑law‑grounded objective metrics (e.g., Masked PSNR, Spatial IoU) to pinpoint the exact stage where a model breaks down.

Benchmarking eleven state‑of‑the‑art video models yields a top score of 0.473, revealing a substantial gap to reliable law‑grounded simulation. Detailed analysis shows a Perception→Formulation→Deduction bottleneck, weak transfer across multiple laws, and a persistent Sim‑to‑Real gap.

**Figure 1.** **Apple-$\pi$ at a glance.** Apple-$\pi$ benchmarks law-grounded physical intelligence in video models through chain-of-frames traces over Perception, Formulation, and Deduction. Built on the 400-video Orchard mechanics dataset, it combines a hybrid evaluation suite of MLLM-based and physics-law-grounded metrics to diagnose where reasoning fails.

How does Physical Intelligence differ from generic physical reasoning in a model?

Physical Intelligence demands an auditable trace: the model must first extract measurable quantities, then explicitly map them to a known law, and finally generate dynamics that obey that law. Generic reasoning may produce correct outcomes without ever exposing these intermediate steps.

The key distinction is that visual plausibility alone does not guarantee law‑grounded reasoning; Apple‑$\pi$ forces models to demonstrate explicit physical intelligence.

Benchmark Landscape

Side‑by‑side comparison of existing physical‑intelligence benchmarks and the new Apple‑$\pi$ dataset.

The table below places Apple‑$\pi$ alongside prior physical‑intelligence benchmarks, exposing which reasoning components each supports.

**Table 1.** Comparison with representative physical intelligence benchmarks. Apple-$\pi$ is the first to explicitly anchor evaluation in physical laws and decompose it into diagnosable reasoning stages. Source: R = Real, S = Simulated. Legend: $\checkmark$ full support $\star$ partial $\times$ not supported.

Related Work

We position Apple‑$\pi$ among prior video benchmarks, highlighting its law‑grounded, stage‑resolved evaluation.

Learning world models from video has long been a goal in AI, and Sora framed large‑scale video generation as a path toward world simulation. Despite recent advances, it remains unclear whether models truly internalize governing physical laws or merely reproduce visual regularities, a question Apple‑$\pi$ explicitly addresses.

A growing body of benchmarks targets physical intelligence, ranging from synthetic intuitive‑physics probes to large‑scale QA suites for understanding, and from VideoPhy and Physics‑IQ to other generation‑focused tests for plausibility. These works largely evaluate visual plausibility without verifying the underlying reasoning process. Apple‑$\pi$ fills this gap with explicit physical‑law grounding, stage‑resolved evaluation over Perception, Formulation, and Deduction, and hybrid MLLM‑based subjective and objective metrics.

Chain‑of‑thought prompting has progressed from text to image and now to video, with Wiedemer et al. introducing the “chain‑of‑frames” analogy between frame‑by‑frame generation and step‑by‑step textual reasoning. Apple‑$\pi$ adopts this paradigm, using chain‑of‑frames prompts to elicit visible, frame‑level reasoning traces that make the model’s physical thought process auditable.

The Apple-π Benchmark and Orchard

Defines the Apple‑$\pi$ benchmark, its Orchard dataset, and the three reasoning stages.

The Apple‑$\pi$ benchmark probes whether video models can reason with explicit physical laws rather than relying on visual plausibility alone.

Apple‑$\pi$ treats a video model like a scientist: it first reads the scene, then picks the governing law, and finally predicts the dynamics that follow.

How does Apple‑$\pi$ differ from existing video‑prediction benchmarks?

Existing benchmarks score a model on how realistic the generated video looks, which can be satisfied by memorising visual patterns. Apple‑$\pi$ forces the model to expose its internal physical reasoning by requiring it to read quantities, select the correct law, and generate dynamics that obey that law.

Orchard is a curated collection of 400 short videos, each built around a single classical‑mechanics law and annotated with exact physical parameters.

Why keep the object vocabulary limited to four primitives?

Limiting objects removes semantic priors (e.g., “a car is heavy”) that could let a model cheat by guessing the law from object type alone. The benchmark then measures genuine physical reasoning rather than object‑recognition shortcuts.

The three stages mirror the scientific method: observe the scene, hypothesize the governing law, then predict the outcome.

What would happen if a model succeeded at Perception but failed at Formulation?

It would correctly read the quantities and segment objects but would choose the wrong governing equation, leading to downstream trajectory errors that the Deduction stage would amplify.

Provide an infographic‑style first frame that overlays all physical quantities next to their visual referents.

Prompt the model with a chain‑of‑frames instruction that specifies the required subtrack (e.g., Perception‑Text).

Collect the model’s output in the prescribed format (final‑frame artifact with or without fade‑to‑white, or full video).

Score the output using MLLM‑based subjective metrics and, when possible, physics‑law‑grounded objective measures.

The model reads the numeric overlays and outputs a white‑background frame containing “mass = 2 kg” and “height = 5 m” at the original spatial locations.

Because the task is Perception‑Text, no object segmentation is required; the fade‑to‑white removes the background.

The output is then fed to the Formulation‑Text subtrack for law selection.

This subtrack isolates pure OCR‑style extraction from any physical inference, revealing whether the model can bind visual symbols to their numeric values.

**Figure 2.** Overview of Orchard. Left: Per-task data-source breakdown across simulated, self-recorded, and Internet-sourced videos. Right: Two-level task taxonomy organizing 400 cases into three single-law pillars and a multi-law composition branch.

**Figure 3.** **Apple-$\pi$ benchmark protocol.** An infographic-annotated first frame and chain-of-frames prompt elicit five reasoning subtracks: Perception-Text for reading quantities, Perception-Graphic for segmenting objects, Formulation-Text for selecting laws, Formulation-Graphic for predicting target states, and Deduction for generating full dynamics.

Experimental Results

Unified models outperform video models on Apple‑$\pi$, with a modest average gain.

Unified models achieve higher average Apple‑$\pi$ score than video models

Table 2 shows average scores 0.43 for unified models versus 0.32 for video models

Diagnostic Analysis

Stage‑wise and pillar‑wise analysis reveals progressive bottlenecks and limited generalization.

Recall that Apple‑$\pi$ evaluates video models on explicit physical laws rather than mere visual plausibility.

Unified models GPT Image 2 and Nano Banana 2 achieve the highest overall scores, 0.704 and 0.699 respectively, far surpassing video‑model averages.

Video models average 0.473, while the best unified scores exceed 0.70.

**Figure 4.** Stage-, pillar-, and source-resolved analysis. (a,b) Stage-wise results show a decline from Perception to Formulation to Deduction. (c) Pillar-wise results compare single- and multi-law cases. (d) Source-wise results show the Sim-to-Real gap.

Failure Analysis and Conclusion

Qualitative failures and protocol ablations reveal why current video models fall short of law‑grounded reasoning.

Qualitative Failure Analysis visualizes representative breakdowns of video models beyond the aggregate scores.

One failure class concerns annotation semantics: the model may copy a text label or an arrow but bind it to the wrong physical variable, e.g., an initial‑velocity arrow is recognized yet assigned the opposite direction, corrupting both the Formulation‑Graphic target state and the subsequent Deduction trajectory.

A second class appears before any physics reasoning: models ignore the requested output format, producing blurred annotation text, failing to isolate target objects, emitting unreadable formulas, or omitting velocity arrows and labels—symptoms of OCR, segmentation, rendering, and instruction‑following limits.

**Figure 5.** Qualitative failure analysis. The model preserves annotations and objects, but misbinds the initial-velocity cue, yielding a wrong target state and trajectory.

Protocol ablations show that providing physical quantities in a structured text prompt (Text‑Parameter) has a near‑neutral effect—average deltas stay within ±0.05—so the benchmark’s infographic annotations are not a performance shortcut but a compact, spatially grounded audit trail. By contrast, a concise prompt reduces the average score, mainly hurting graphic‑output tracks (P‑G, F‑G) while slightly improving formula‑selection (F‑T).

Conclusion: Apple‑$\pi$ anchors each case in explicit classical‑mechanics laws and evaluates Perception, Formulation, and Deduction stages. Experiments reveal that today’s video models learn useful physical priors from large‑scale data yet remain far from reliable law‑grounded simulators, especially when they must formulate laws, execute dynamics, compose law transitions, or transfer to real‑world domains. Future progress will require explicit reasoning modules, richer physics‑oriented data, and post‑training techniques that target reasoning competence.

Orchard Design Principles

Appendix details Orchard design principles, dataset composition, and frame‑rate metadata.

A.1 Design Principles outlines the “Law‑first rather than video‑first” philosophy: each case starts from an explicit governing law in classical mechanics, an initial condition, and a law‑predicted trajectory.

This approach lets Apple‑$\pi$ pinpoint failures to Perception, Formulation, or Deduction stages instead of treating the generated video as an undifferentiated output.

The dataset splits into a single‑law branch, which isolates one dominant physical principle, and a multi‑law branch, which composes two or more principles (e.g., inclined‑plane followed by projectile motion) to test state‑variable transfer.

Object vocabulary is limited to four primitive solids—sphere, cube, cylinder, and cone—ensuring consistent geometry, masks, and size parameters across simulated, self‑recorded, and Internet‑sourced cases.

Physical quantities are attached to visual referents: mass to an object, friction coefficient to a contact surface, angle to a ramp, and velocity vector to a moving body. An infographic‑style annotated first frame lists these initial conditions without revealing the answer.

A.2 Dataset Composition and Native Frame Rates reports that Orchard comprises 400 cases covering ten canonical mechanics tasks, drawn from three sources with differing ground‑truth construction pipelines.

Simulator cases provide programmatic ground truth from the physics engine, while real‑world cases supply measurement‑backed, human‑reviewed annotations, preserving a meaningful distinction between synthetic and real data.

**Table 4.** Data sources and native ground-truth frame rates. Frame rate is stored per case and used by the evaluation pipeline; Apple-$\pi$ does not assume a single global GT fps.

Benchmark Protocol Details

Defines the subtrack protocol, prompt templates, and evaluation standardization for Apple‑$\pi$ benchmarks.

This appendix details the benchmark protocol that governs all five Apple‑$\pi$ subtracks, the prompt families used for different model families, and the evaluation pipeline that normalizes heterogeneous outputs.

The protocol specifies, for each subtrack, which annotated first frame is provided, how the model is prompted, and what artifact constitutes the answer.

Provide the same annotated first frame to every subtrack.

Append a subtrack‑specific chain‑of‑frames prompt (Table 11 defines the expected output).

Run the model: unified models produce PNG/keyframe artifacts; video models produce an MP4.

Canonicalize the output according to Table 12 (resolution, fps, timestamps).

Score the canonical packet with the appropriate MLLM‑judge rubric (Table 13).

Aggregate three rollout scores per case to obtain the final subtrack score.

**Table 5.** Task taxonomy in Orchard. The first nine tasks are single-law diagnostic tasks; the last row contains multi-law compositions.

**Table 6.** Canonical case-level metadata and artifacts.

**Table 7.** Real-world physical-parameter measurement. The velocity procedure uses the recorded fps: 30 fps for self-recorded clips and the native fps for Internet-sourced clips.

**Table 8.** Examples of issues caught and corrected during simulator QA.

The table presents Cohen's $\kappa$ values for five different fields: Object naming convention compliance, Task-type classification, Formula correct-answer choice, Camera-framing acceptability, and Velocity-label legibility.

The table outlines five subtracks: P-T, P-G, F-T, F-G, and Deduction. Each row specifies the associated question, the expected output format, and the main evaluation target for that subtrack.

**Table 11.** Input-output specification for the five Apple-$\pi$ subtracks.

**Table 12.** Canonical evaluation packets for different model families.

**Table 13.** MLLM-judge rubric dimensions. P-G and F-G additionally include objective IoU as described in Section D.5.

Evaluation Metrics and Rubrics

Evaluation metrics, scoring formulas, and benchmark limitations are detailed.

Three low‑level Perception‑Text metrics are defined: `velocity_position_match` checks exact arrow start locations, `no_extra_annotations` penalises any hallucinated labels, and `white_background` requires a pure white canvas.

The Perception‑Text score aggregates three groups—content (50 % weight), layout (30 %), and style (20%)—by averaging the respective metric averages.

Perception‑Graphic evaluation uses four groups: background purity, annotation removal, object preservation, and spatial consistency, weighted 0.10, 0.30, 0.30, and 0.30 respectively.

Formulation‑Text judging focuses on three lines of text: the correct option letter, the symbolic formula, and the fully instantiated numeric version, with four weighted groups (0.20, 0.30, 0.40, 0.10).

Formulation‑Graphic scoring compares the generated target‑time scene against ground truth and the initial annotated frame, using annotation removal, object match, velocity‑arrow quality, and velocity‑text groups, each weighted 0.30 except background (0.10).

Deduction evaluation for video models combines subjective MLLM judgments with objective metrics (PSNR, masked PSNR, IoU variants, velocity accuracy) into integrity (20 %), fidelity (20 %), and physics (60 %) groups.

Objective metrics are computed on programmatically extracted motion masks: normalized PSNR, masked PSNR, spatial IoU, spatiotemporal IoU, weighted spatial IoU, and velocity‑accuracy score.

Final Deduction scores are aggregated hierarchically across rollouts, cases, subtracks, and models, following the formulas in Sections D.6 and D.7.

Judge reliability is ensured by using Gemini 3 Flash as the primary MLLM judge and cross‑checking a subset with Qwen3‑VL‑30B‑A3B‑Instruct‑FP8; Table 14 shows strong Pearson $r$ and Spearman $\rho$ agreement.

**Table 14.** Agreement between Gemini 3 Flash and Qwen3-VL on per-model mean scores.

Limitations are explicitly listed: the benchmark covers only classical rigid‑body mechanics, a restricted set of geometric primitives, a single fixed camera, and relies on MLLM judges supplemented by objective metrics.

Real‑world ground‑truth noise arises from measurement and mask refinement; mitigation includes controlled recording and inter‑annotator agreement analysis, yet residual error remains.

Temporal normalization assumes the generated clip encodes the requested physical duration, a necessary convention given variable provider output lengths.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers