PROWBench: Do Video Models Render What the Program Specifies?

Zheng-Hui Huang, Guixu Lin, Yu-Ju Tsai, Jian-Kai Zhu, Fengbo Lan, Yu-Lun Liu, Yung-Yu Chuang, Kaipeng Zhang, Zhixiang Wang

PROWBench evaluates whether video models faithfully render prescribed world states and interaction events rather than just plausible visuals.

How well do current video generation models adhere to explicit, programmable world rules and spatial constraints rather than just generating plausible-looking motion?

Video generation models can produce visually realistic scenes, but they often fail to adhere to the specific rules, entity trajectories, and interaction timelines defined by a program. PROWBench provides a programmatic framework that logs replayable world states and renders them into synchronized proxy representations, allowing researchers to check generated videos against the actual engine-recorded events. General-purpose video models perform surprisingly well on short clips, often matching task-specific renderers in entity control and logic adherence, though all models struggle with long-horizon consistency and multi-view identity.

Paper Primer

The core challenge is that visual plausibility does not imply logical fidelity. A model might generate a realistic-looking video of a person walking, but if the program specified they should grasp an object at a precise timestamp, the model frequently fails to realize that interaction or maintain the correct entity state.

PROWBench acts as a diagnostic engine: it takes a structured world record—containing entity transforms, articulated poses, and interaction anchors—and maps it into standardized proxy inputs like coarse 3D shapes or colored bounding boxes. By feeding these proxies to a model, the benchmark isolates the model's ability to translate explicit logic into visual output, independent of the rendering style.

General-purpose video models are competitive with task-specific renderers for short-clip control.

In the Unverified-FF setting, general-purpose models like MiniMax-H3 achieved entity control (BBox IoU) and camera motion accuracy comparable to or better than specialized world models, while also leading in Interaction Success Rate. General-purpose models outperformed camera-conditioned world models in entity placement in 87 of 89 scenes, with a median IoU improvement of +0.14.

Long-horizon consistency and multi-view identity remain significant failure points.

Across 30-second evaluation clips, entity reappearance accuracy (Reappearance IoU) dropped to 0.05–0.13, and models frequently swapped entity identities or appearances in multi-view settings. Most methods saw BBox IoU fall below 0.3 after ten seconds of generation.

Why is a "proxy representation" necessary instead of just using the raw engine output?

Raw engine output is often too specific to a single renderer. Proxies like colored bounding boxes or coarse 3D shapes provide a standardized, lightweight interface that forces the model to interpret spatial and temporal logic without being distracted by high-fidelity textures or lighting.

How does the benchmark distinguish between a "plausible" video and a "correct" one?

It uses two VLM-based metrics: Logic-Render Alignment, which checks if prescribed actions appear in their assigned time windows, and Interaction Success Rate, which verifies if an interaction (e.g., grasping an object) reaches its required end state as recorded by the engine.

Researchers should shift focus from visual quality metrics to logic-adherence benchmarks; the current generation of models is spatially capable but lacks the persistent memory and identity binding required for reliable interactive world simulation.

Introduction: The Programmable World Gap

We define PROWBench and programmable world models, exposing the gap between visual plausibility and rule adherence.

Recent video world models produce impressive visuals but often ignore explicit, programmable constraints, making it unclear whether generated videos truly follow the underlying rules. Existing benchmarks measure aesthetics, controllability, and physical consistency, yet they lack replayable world records that capture fine‑grained entity states and timestamped events, so violations of program‑specified logic can go unnoticed.

PROWBench is a benchmark that records the exact state and events of a programmable world and supplies proxy videos, enabling direct comparison between generated outputs and the ground‑truth execution.

A programmable world model separates the underlying simulation dynamics from visual generation, letting a program dictate exact entity states while a video generator produces the observed frames.

The shift from visual plausibility to rule‑based adherence reveals that current video models still fall short of faithfully rendering programmable world constraints.

Related Work

We situate PROWBench among prior video world models, structured rendering, and benchmarks.

Video world models such as Genie, GameNGen, and GameFactory predict future frames from visual context and actions, but they keep state implicit. Programmable world models like Code World Model and the Programmable World Model instead expose explicit, executable state representations. PROWBench uses these explicit states to assess how faithfully generated videos follow the recorded world dynamics.

Structured conditions—depth maps, trajectories, bounding boxes, masks, and camera motion—provide direct spatial control for visual generation. Methods such as Coarse‑to‑Real, VideoFrom3D, Diffusion Renderer, AlayaRenderer, and LiVER differ in how densely they encode geometry. PROWBench isolates the effect of these representations by deriving paired proxy views from the same engine‑recorded trajectory.

Existing benchmarks (VBench, EvalCrafter, WorldSimBench, WorldModelBench, WorldScore, MIND, WorldMark, Omni‑WorldBench, WorldArena, iWorld‑Bench, WBench) evaluate video quality, physical plausibility, and interactive consistency, often conflating errors across pipelines. PROWBench separates rendering fidelity by fixing the world‑state trajectory and measuring Logic‑Renderer Alignment and Interaction against engine‑recorded ground truth.

The PROWBench Generation Pipeline

The pipeline builds programmable worlds, compiles synchronized conditioning inputs, and exports records for video generation evaluation.

The world generation pipeline turns high‑level scene specifications into fully replayable episodes, then compiles those episodes into conditioning frames for generative video models. It isolates world dynamics from rendering, enabling controlled experiments across multiple visual representations.

Synthesize a dynamic world from a configuration that lists environment layout, entity definitions, random seeds, and execution duration.

Run specialized controllers (interaction, activity/locomotion, driving) to produce articulated poses, object trajectories, and interaction events.

Record a structured episode containing shared scene geometry, entity hierarchies, initial conditions, and per‑tick transforms, visibility, and appearance changes.

Compile recorded world states and camera trajectories into synchronized conditioning frames for each view.

Render each frame using one of three adapters: Coarse 3D, Semantic proxy (CWM proxy), or Colored‑OBB, preserving the same camera parameters across representations.

Export three output categories: (1) scene and motion records, (2) rendered observations, and (3) text and interaction labels.

Validate the episode with task/timeline checks, geometric consistency checks, replay/alignment verification, and visual/media quality inspections.

Verified‑FF forces the first generated frame to match the ground‑truth pose of the target entity, whereas Unverified‑FF leaves the first frame unconstrained.

The recorded first‑frame transform for entity 1 is $T_1 = \begin{bmatrix}I & (0.5,0.0,0.2)^\top \\ 0 & 1\end{bmatrix}$.

The rendering adapter generates the first‑frame image $I_{v,r_1}$ using $i = R_r(G, S_1, C_v)$.

Verification compares the rendered pose extracted from $I_{v,r_1}$ to $T_1$; because they match exactly, the episode is marked Verified‑FF.

Verified‑FF guarantees that the model’s initial visual input is physically consistent, preventing early‑stage drift that would otherwise corrupt the entire generated video.

How does Verified‑FF differ from simply initializing a model with the correct pose?

Verified‑FF not only initializes the pose but also enforces that the rendered first frame exactly reproduces the ground‑truth geometry and camera parameters. This extra check catches mismatches between the underlying world record and the visual rendering, which a naïve initialization would miss.

**Figure 1. Overview of our extensible framework and benchmark.** Our extensible framework pairs visual proxies with ground-truth world records, enabling reliable evaluation of how faithfully generated videos realize prescribed states, rules, and interactions. This assessment would otherwise be difficult. Here, we compare the CLIP score with our proposed interaction metric. CLIP assigns a higher score to the clearly inferior case, whereas one of our metric correctly reflects whether the prescribed interaction is faithfully executed.

Rendering adapter that maps a world state and camera to a conditioning frame.

Evaluation Suite and Metrics

We detail the evaluation protocol, metrics, and computation steps for assessing controllability and consistency.

To quantify how faithfully generated videos follow the engine‑recorded world, we define a suite of metrics covering control, interaction, logic, and long‑term consistency.

Collect engine‑recorded trajectories, events, and states as ground‑truth references.

Localize generated entities with SAM 3 from semantic‑label prompts.

Match detections to references using label‑compatible Hungarian matching.

Compute BBox IoU and Center Error for each visible entity over its trajectory.

For each registered event, evaluate interaction success and compute ISR.

Weight ISR by clip‑level BBox IoU to obtain gISR.

Assess each timeline segment for prescribed action presence to obtain LRA.

Weight LRA by BBox IoU to obtain gLRA.

Identify reappearance events and compute Reappearance IoU and State Persistence.

Average each metric first within clips, then across samples for reporting.

Event $e_1$ satisfies all three conditions → indicator = 1.

Event $e_2$ fails the end‑state check → indicator = 0.

Event $e_3$ is invalid → indicator = 0.

Sum of indicators = 1; divide by $|E_s|=3$ → $ISR_s = 1/3 \approx 0.33$.

ISR captures both semantic correctness (action) and outcome verification (end state), penalizing any missing component.

Evaluated Model Families

We describe the 11 models evaluated, the variables we vary, and the metrics we record.

Proxy‑conditioned models receive a coarse 3‑D proxy (depth, segmentation, or colored OBB) that encodes scene geometry but no explicit camera pose, while Camera‑conditioned models are given the exact recorded camera trajectory and the first RGB frame but no entity trajectories.

How does proxy conditioning differ from camera conditioning in terms of control over entity motion?

Proxy conditioning gives only a coarse geometric scaffold (depth, segmentation, or colored OBB), so the model must infer both where objects are and how the camera moves. Camera conditioning supplies the precise camera pose, removing that inference burden but providing no explicit object trajectories.

Select one of the 11 models listed in the section.

Provide the common inputs: the first RGB frame, the text prompt, and either the recorded camera trajectory (Camera‑conditioned) or the proxy representation (Proxy‑conditioned).

Run the model to generate a five‑second video for a given scene‑camera pair.

Compute the full suite of metrics (BBox IoU, CErr, ISR, LRA, gISR, gLRA, Rot, Trans, CamMC, Img, Aes, Warp, Smooth, CLIP) by comparing the generated video against the engine‑recorded ground truth.

Record results separately for the Verified‑FF setting (first frame matches ground truth) and the Unverified‑FF setting (first frame taken from MiniMax‑H3 output).

Feed $F_0$, the prompt, and $P_{cam}$ into EchoWM.

EchoWM generates $V_{gen}$.

Compare $V_{gen}$ to the ground‑truth video $V_{gt}$, obtaining BBox IoU = 0.351, CErr = 0.055, ISR = 0.506, LRA = 0.685, Rot = 0.061°, Trans = 0.064 m, CamMC = 0.040, Img = 0.099, Aes = 0.075, Warp = 0.083 × 10⁻³, Smooth = 0.101, CLIP = 0.758.

This concrete run shows that even with perfect camera control, the model’s IoU remains modest, highlighting the challenge of accurate object placement.

The evaluated models fall into two families. Camera‑conditioned world models—EchoWM [60], SANA‑WM [63], LingBot‑World 2.0 [43], and AlayaWorld v1.1 [46]—receive the first RGB frame, the text prompt, and the recorded camera trajectory. Proxy‑video‑to‑RGB models—Cosmos‑Transfer2.5 [1], C2R [15], CWM [9], LynnReal‑Omni [37], Seedance 2.5 [4], and MiniMax‑H3 [39]—receive various proxy representations (depth, segmentation, colored OBB, etc.) together with the first frame and the prompt.

Main Track Performance

Key performance shifts on the main benchmark, with numbers that quantify the change.

Proxy‑enabled methods gain a noticeable edge in entity‑control IoU when the first frame is verified.

Across the Verified‑FF setting the top proxy method (◐) reaches an IoU of 0.34 versus 0.22 in the Unverified‑FF setting.

**Table 1.** Main track. Results on the Verified-FF setting (top, 40 five-second scene-camera pairs) and the Unverified-FF setting (bottom, 90 pairs); the two settings use different scenes, so best and second-best values are marked within each setting in **bold** and <u>underlined</u>. Input: 🖼️ = first frame checked against the recorded state (Verified-FF), 🎞️ = unchecked first frame, i.e., frame 0 of the MiniMax-H3 output for the same scene and camera (Unverified-FF), 🎥 = camera pose, 💿 = depth, 🖌️ = class-level segmentation, ◐ = CWM proxy, ⬢ = coarse 3D proxy, ⬜ = colored-OBB proxy. All methods also receive the text prompt. ↑/↓: higher/lower is better. CErr: normalized box-center error; Rot/Trans: camera rotation (degrees)/translation error; Img/Aes: imaging/aesthetic quality; Warp: warping error (×10⁻³); Smooth: motion smoothness; CLIP: CLIP-Score; ISR: Interaction Success Rate; LRA: Logic-Render Alignment; gISR/gLRA: ISR/LRA multiplied by BBox IoU per sample and then averaged, i.e., events realized by correctly placed entities.

Challenge Track: Long-Horizon and Multi-View

Challenge results measure how faithfully models follow explicit control proxies while generating video.

The challenge track tests whether video generators respect the explicit control signals supplied by proxies, using the same engine‑recorded ground truth as in the main track.

Proxy Leakage happens when visual elements of the input control representation (e.g., colored bounding boxes) unintentionally appear in the generated frames, showing that the model has not fully abstracted away the control signal.

**Figure 6. Proxy leakage.** The top row shows proxy inputs, and the bottom row shows corresponding generated frames. In these representative failure cases, colored bounding boxes or wireframe outlines from the proxies remain visible in the outputs, revealing unintended leakage of control representations into the rendered appearance.

Discussion and Takeaways

We compare proxy‑conditioned, task‑specific, and general‑purpose video models across entity, logic, and camera metrics.

Recall that PROWBench measures the gap between video generation and the explicit, programmable world constraints defined by the engine.

Entity control favors proxy‑conditioned models: in Verified‑FF the top three methods (LynnReal‑Omni, MiniMax‑H3, CWM) reach BBox $IoU$ of 0.518, 0.494, 0.454, while camera‑conditioned world models stay between 0.303 and 0.351; the advantage persists in Unverified‑FF (0.457–0.409 vs. 0.204–0.255).

Logic and state metrics reveal a mixed picture: LingBot 2.0 attains the highest Interaction Success (0.733) in Verified‑FF despite a modest BBox $IoU$ (0.327), whereas Cosmos‑Transfer2.5, despite richer depth input, scores lowest on both Interaction Success and Logic‑Render Alignment. LynnReal‑Omni and Seedance 2.5 dominate the weighted scores (gISR, gLRA) in both Verified and Unverified settings.

Camera control gaps are smaller: most methods achieve rotation errors of 1.7–3.4°, close to the pose estimator’s 1.1–2.7°. LynnReal‑Omni records the lowest CamMC in both Verified‑FF and Unverified‑FF, and general‑purpose models tie with camera‑conditioned models on CamMC.

**Figure 3.** Method classes at a glance. Main-track results per method class in Verified-FF (light) and Unverified-FF (dark). Bars show class means, dots the individual methods. Camera-conditioned: EchoWM, SANA-WM, LingBot 2.0, AlayaWorld v1.1; task-specific renderers: Cosmos-Transfer2.5, C2R, CWM, LynnReal-Omni; general-purpose video generation: MiniMax-H3, Seedance 2.5.

Challenge‑track long‑horizon results (Table 2) show modest BBox $IoU$ (0.175–0.280) over 30 s and very low Reappearance $IoU$ (0.053–0.131). State Persistence varies, with CWM achieving the lowest Reappearance $IoU$ but a high persistence score (0.886), highlighting divergent memory behaviours across methods.

In the multi‑view setting (Table 3), C2R attains the lowest MEt3R (0.397) yet suffers poor Appearance Compliance (32.2 %) and Cross‑view Consistency (25.5 %). Methods that explicitly separate participants achieve 82–94 % Appearance Compliance and 81–90 % Cross‑view Consistency, underscoring the importance of explicit multi‑view cues.

**Figure 5. Proxy composition and camera control.** LynnReal-Omni on 20 scene-camera pairs, from entity boxes only to adding ground, sky and 50% of the background objects. Top: scene 043 at 4.25 s; the overlay gives the mean rotation error of that video. Bottom: means over the 20 pairs (lower is better). The 100% BG reference uses the original prompts and first-frame setting and is not a controlled step.

**Figure 4** Proxy only vs. camera input. Per-sample CamMC in Unverified-FF of three models without camera trajectory input against LingBot 2.0; points below the diagonal favor the proxy-only model.

Takeaway T1: Rendering from a proxy consistently outperforms camera conditioning for following entity trajectories and event timelines, as reflected by higher class‑mean BBox $IoU$ in the majority of scenes (38/40 Verified‑FF, 87/89 Unverified‑FF). Takeaway T2: Because the proxy video is rendered from the prescribed camera, it already carries the camera motion, making proxy‑only and camera‑input models comparable on CamMC.

Conclusion

We recap PROWBench findings, highlight remaining challenges, and outline future directions.

General‑purpose video generation models already render short clips well. On the short, single‑camera clips of the main track they are competitive with task‑specific renderers, achieving comparable $BBox\ IoU$ (23/40 and 43/89 scenes), lower $CamMC$ errors (30/40 and 76/90 scenes, $p\\leq0.003$), and higher Interaction Success.

In the Unverified‑FF setting (53/67 and 61/79 scenes, $p\\approx10^{-6}$) they also attain the lowest warping error, with class‑mean errors of 0.38 and 0.33 versus 0.72 and 0.83 for task‑specific renderers.

Fine‑tuning gives mixed gains. CWM, built on MiniMax‑H3, has lower $BBox\ IoU$ in 71 % of Unverified‑FF pairs ($p=10^{-4}$) and lower imaging quality in ≈ 90 % of pairs, yet lower warping error in ≈ 80 % of pairs. LynnReal‑Omni, also based on MiniMax‑H3, achieves higher $BBox\ IoU$ in 62 % of pairs ($p=0.03$) and higher Logic‑Render Alignment in 81 % of Verified‑FF pairs ($p=0.02$), but higher warping error in every pair.

Despite strong short‑clip performance, general‑purpose models still struggle with prescribed events. On a 20‑scene gun‑fight benchmark, MiniMax‑H3 records $BBox\ IoU$ 0.526 and Interaction Success 0.600, while the domain‑specific PWM reaches 0.602 and 0.850, highlighting that spatial alignment alone does not guarantee event fidelity.

**Table.** Comparison of Verified-FF and Unverified-FF methods across various parameters including References, Task, Appearance source, Style source, Control rule, and Structural skeleton.

The proxy leaves a key ambiguity unresolved: appearance assignment. A manual audit of 62 multi‑person scenes found MiniMax‑H3 swaps the appearances of two people in 12 cases. Orientation can also be ambiguous, with some entities rendered facing the opposite direction.

Open problems persist. Over long horizons (30 s) BBox IoU drops to 0.175–0.280 and Reappearance IoU to 0.053–0.131. For re‑entering pedestrians, IoU after return falls to 0.34 (CWM), 0.30 (LynnReal‑Omni), 0.17 (Seedance 2.5), and 0.01 (MiniMax‑H3). Multi‑view identity suffers without explicit cues: methods with identity‑colored boxes or a first frame achieve 82–94 % Appearance Compliance and 81–90 % Cross‑view Consistency, whereas those without such cues perform markedly worse, often rendering a differently dressed player in 10–19 % of view pairs.

**Figure 7** BBox IoU over the 30-s recordings. Averaged in 5-s windows over the ten recordings and all entities with a reference box: every method is best in the first five seconds and loses the prescribed layout as the sequence continues (the visible entities differ between windows, so the curves show a trend).

**Figure 8** A returning pedestrian. Park scene, first-person camera, the recording with the largest drop: the dashed box is the engine reference of a pedestrian who leaves the view at 3.9 s and returns at 17.25 s. All methods localize it before it leaves (IoU about 0.6); after the return, the IoU is 0.34 (CWM), 0.30 (LynnReal-Omni), 0.17 (Seedance 2.5) and 0.01 (MiniMax-H3).

**Table 4.** Comparison with a domain-specific model on armed-confrontation scenes. The 20-scene set includes first-frame conditioning and is designed for comparison with PWM, which is trained only on gunfight scenes (third-person; an armed confrontation in which one person is shot and collapses). Metrics use native model outputs; best and second-best values are **bold** and <u>underlined</u>. Input: 🖼️ = first frame, 🛢️ = depth, ◐ = CWM proxy, 📦 = coarse 3D proxy, ◻ = colored-OBB proxy. All methods also receive the text prompt. CWM = Code World Model. Metrics as in Table 1.

**Table 6.** Open problems remain in the challenge track: long horizons, re-entering entities and multi-view identity.

Metric analysis reveals that Interaction Success and Logic‑Render Alignment are largely independent of $BBox\ IoU$ (Spearman $\\approx0.05$–$0.06$). CLIP‑Score varies little (LynnReal‑Omni records the lowest at 24.2) despite large differences in geometric and logical metrics, underscoring the need for multi‑facet evaluation. The benchmark’s reliance on a VLM judge that is not yet calibrated to human annotation is a noted limitation.

In sum, PROWBench provides a systematic framework for evaluating programmable‑world video models, exposing the gap between visual plausibility and faithful world realization and charting a path toward more controllable, consistent, and long‑term coherent generation.

Appendix: Benchmark Details

Appendix details dataset composition, prompt construction, evaluation metrics, and implementation specifics.

PROWBench contains $170$ evaluation episodes, $200$ selected camera views, and $600$ proxy input videos across four collections. The videos are rendered at $1280\times720$ for $570$ five‑second clips ($150$ frames each) and $30$ thirty‑second clips ($900$ frames each), yielding $62.5$ minutes of aggregate proxy video.

In the Verified‑FF setting we generate a single RGB first frame per view using MiniMax‑H3, chosen for its strong alignment with proxy geometry. Each candidate frame is automatically verified against the engine record for person count, box IoU (minimum $0.30$), absence of proxy artifacts, and correct heading of vehicles and mounts.

The base prompt for every method consists of five components: a fixed control block, subject definitions, a one‑sentence task summary, a retention analysis, and a detailed description of style, environment, appearance, and timeline. Appearance text is produced by Qwen3.6‑27B from the first‑frame image and the CWM proxy layout, while the proxy video supplies exact geometry and motion.

Evaluation metrics include $BBox\ IoU$ (Bounding Box Intersection over Union) and $CamMC$ (Camera Motion Consistency) for entity and camera control, respectively. Visual quality is measured with MUSIQ, LAION aesthetic predictor, Warp Error, Motion Smoothness, and CLIP‑Score. Appearance compliance and cross‑view consistency are assessed by Qwen3.6‑27B on five attributes, while geometric multi‑view consistency uses MEt3R.

Implementation details for the evaluated models are listed in Table 7, covering conditioning inputs, spatial resolutions, frame rates, and output frame counts. Qualitative examples (Figures C.3–C.10) illustrate entity control, interaction success, and the effect of providing or omitting a verified first frame.

**Figure A.1** Gallery of all 170 PROWBench cases. Each tile shows the MiniMax-H3 output at the middle of the clip (at 15 s for the 30-second long-horizon recordings); the inset shows the colored-OBB proxy input at the same instant. MiniMax-H3 is used only because it has outputs for every collection; its frames are generated results, not ground truth. Cases are grouped by collection: the Verified-FF (40) and Unverified-FF (90) settings of the main track, the long-horizon (10) and multi-view (10, view A shown) settings of the challenge track, and the domain-specific gunfight set (20). FPP marks first-person views.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers