Orbia: A Benchmark for 3D Consistency in Interactive World Models

Dongyue Lu, Guian Fang, Hur Lim, Yonggang Wu, Yanrong Wang, Hongshan Chen, Shengju Qian, Yue Liao, Wei Chow, Lingdong Kong, Tianxin Huang, Yu Yang, Hongji Yang, Junchao Huang, Ao Liang, Yihao Wang, Zihan Wang, Rong Li, Hanlin Chen, Xin Wang, Wei Tsang Ooi, Mike Zheng Shou, Shuicheng Yan

iWorld-Bench is the first systematic benchmark for evaluating 3D consistency in interactive video world models.

How can we systematically measure whether interactive video world models maintain 3D spatial consistency during long-term exploration?

Interactive video models can generate visually convincing frames, but they often fail to maintain a coherent 3D environment as a user explores or revisits previously seen locations. The authors introduce iWorld-Bench, a benchmark that evaluates 3D consistency across three dimensions: input preservation, generated-content persistence, and 3D self-consistency. The benchmark reveals that while leading world models approach the visual quality of text-to-video generators, they significantly outperform them in camera control and temporal stability, though all models struggle to maintain consistency during long-horizon exploration.

Paper Primer

The paper addresses the "3D consistency gap" in world modeling: a model might generate a high-quality frame, but if the camera moves or returns to a previous spot, the scene often drifts, forgets its own generated content, or violates basic geometric constraints. iWorld-Bench formalizes this by testing whether a model can preserve the initial input, retain content it created during exploration, and maintain a globally coherent 3D structure.

The benchmark uses a pipeline of 720 evaluation instances—spanning real-world datasets, self-captured scenes, and simulation—to test models across six exploration-and-revisit protocols. It evaluates 27 models by mapping their native interfaces (text, action, or camera) to standardized trajectories, then measuring performance via cross-view depth, feature correspondence, and held-out Gaussian rendering.

World models significantly outperform text-to-video (TI2V) models in camera control, but TI2V models often achieve higher 3D self-consistency.

World models average 57.29 in control compared to 32.48 for TI2V models, while TI2V models average 69.88 in 3D self-consistency, exceeding world models by 10.14 points. A 24.80-point control gap exists between the two groups, highlighting that internal geometric coherence does not guarantee trajectory adherence.

Visual quality is a poor proxy for 3D consistency.

Visual quality shows a low rank correlation (0.33 for world models) with the mean of the three consistency tiers, and models with high visual quality often fail to preserve scene content or maintain geometric agreement. Leading models in visual quality often rank outside the top tier in consistency metrics, proving that "convincing" frames do not imply a reliable 3D world.

Why is this benchmark needed if existing video benchmarks already measure temporal consistency?

Existing benchmarks focus on perceptual quality or local geometric coherence, but they lack systematic evaluation of long-term 3D persistence. iWorld-Bench specifically tests whether a model can "remember" content it generated itself when the camera returns to a previously visited region.

Does this benchmark apply to all video generators, or only those designed for 3D navigation?

It evaluates both camera/action-conditioned world models and standard text-to-video (TI2V) models. By using adapters to translate trajectories into native model interfaces, the authors compare how different architectures handle the same spatial exploration tasks.

Researchers should stop using visual quality as the primary metric for world models; iWorld-Bench demonstrates that control and 3D consistency are independent capabilities that require explicit geometric memory to solve.

Introduction

Introducing iWorld‑Bench to expose the gap between visual plausibility and 3D spatial coherence.

Video generators now produce visually impressive frames, yet those frames often fail to describe a single coherent 3D world. As interactive agents revisit scenes, inconsistencies in geometry and content become apparent, exposing a gap between visual plausibility and true 3D spatial coherence.

How well the generated frames jointly describe a single, stable 3D scene as the camera moves and time progresses.

**Figure 1. Overview of Orbia.** This work introduces Orbia, a benchmark for evaluating 3D consistency along three dimensions: $^1$input preservation, $^2$generated-content persistence, and $^3\$3D self-consistency. Six exploration-and-revisit templates support short and minute-long evaluation across public datasets, self-captured scenes, and UE simulation. We benchmark world models and video generators through text, action, and camera-pose interfaces.

The primary gap is that visually plausible frames do not guarantee 3D spatial coherence.

Related Work

Brief survey of interactive video world models and evaluation benchmarks.

Interactive video world models extend large‑scale video generators by conditioning future frames on user‑controlled inputs such as actions, language prompts, or explicit camera trajectories. This enables exploration of generated environments where each choice influences subsequent observations.

World models are predictive systems that simulate future observations given the current state and a sequence of actions.

TI2V (Text‑to‑Image‑to‑Video) models first synthesize a still image from a text prompt and then extend that image into a temporally consistent video.

Table 1 summarizes how existing benchmarks differ in input modalities, data provenance, and the suite of consistency tests they provide, highlighting that Orbia uniquely covers all five consistency dimensions.

Benchmark Overview

Defines the experimental protocol and data pipeline for measuring 3D consistency of interactive video world models.

The Orbia Benchmark formalizes how to probe an interactive video world model’s 3‑D consistency by varying camera trajectories while keeping all other factors fixed.

The benchmark supplies a set of controlled exploration‑and‑revisit trajectories and a suite of metrics that together quantify input preservation, generated‑content persistence, and 3‑D self‑consistency.

How does the Orbia Benchmark differ from a generic video‑generation test that simply samples random camera poses?

Random sampling does not guarantee that a model sees the same scene twice; the Orbia Benchmark forces a return to the exact pose $P_r = P_0$, so any inconsistency can be attributed to the model rather than to unseen content.

Six hand‑crafted motion templates (forward‑backward, lateral return, yaw return, arc return, orbit, peek‑and‑return) define how the camera moves away from and back to the initial view.

Why not just use a single straight‑line forward‑and‑backward trajectory instead of six templates?

Different motions stress different aspects of the underlying 3‑D representation. A straight line only tests depth consistency under translation; the other templates add rotation, curvature, and occlusion dynamics that reveal subtler inconsistencies.

Select a case from the dataset and load its registered RGB‑D pair $(I_0, D_0)$ at pose $P_0$.

Choose one of the six exploration‑and‑revisit templates and instantiate concrete poses $P_q$, $P_{first}$, $P_{revisit}$, $P_r$.

Run the interactive model $M$ with initial image $I_0$, scene description $s$, and user inputs $U$ (text $T$, actions $A$, or camera poses $P$) to generate video $\hat V$.

Recover depth and camera poses from $\hat V$ using the DA3 estimator.

Compute the three consistency dimensions: (i) input preservation (compare $P_r$ to $P_0$), (ii) generated‑content persistence (compare $P_{revisit}$ to $P_{first}$), (iii) 3‑D self‑consistency (cross‑view depth/feature agreement across $\hat V$).

Aggregate scores according to the benchmark’s weighting scheme (30 % short videos, 70 % minute‑long extensions).

At $P_0$ the model receives the initial RGB image $I_0$ and depth $D_0$.

The model is instructed to move forward 0.5 m; it generates frame $F_1$ at $P_q$.

The model is then instructed to move backward 0.5 m; it generates frame $F_2$ at $P_r$.

Depth and pose are recovered for $F_2$; the return view is compared to $I_0$ using DINO similarity and PSNR.

This concrete trial shows how a simple translation isolates input preservation: any drop in similarity between $F_2$ and $I_0$ directly signals a breach of 3‑D consistency.

**Figure 2.** Data construction and annotation pipeline. $^1$Reference selection uses geometry and template constraints. $^2$Anchor verification combines VLM proposals, segmentation, tracking, and cross-view surface checks. $^3$Trajectory synthesis defines geometrically valid exploration and revisits. $^4$Quality review includes ranking, human curation, and captioning.

**Figure 12** Exploration-and-revisit templates. Blue arrows show outward motion and rose arrows show return motion. In yaw return, the camera stays at a fixed center and the symbols along the arc indicate viewing directions.

Overall Performance

SolarWM‑H3 tops the leaderboard, revealing key gaps between world and TI2V models.

SolarWM‑H3 leads overall, achieving the highest overall score among all evaluated models.

Table 2 shows SolarWM‑H3 at $76.32$, the top overall score.

Explicit 3D memory appears in three of the top five world models (Lyra2, EVOKE, Gen3C), linking past observations to 3D locations and boosting control and content‑preservation scores. Camera control remains the biggest shortfall for TI2V models, while their visual quality and 3D self‑consistency surpass world models despite weaker control.

**Table 2.** Comparison across six evaluation tiers. Gen. Persist. denotes generated-content persistence. Temp. Stab. denotes temporal stability. 3D Repr. indicates scene geometry used by the generator ($\checkmark$: present, $\times$: absent, $-$: unconfirmed). Models are sorted by Overall within each group. 1st, 2nd, 3rd mark the top three scores per column and group.

**Table 11.** Tier 5: Detailed 3D self-consistency results. Scores use a 0–100 scale (↑). Models are ordered by Tier Score within each group. 1st, 2nd, 3rd mark the top three scores per column and group.

World models excel in control and consistency, while TI2V models lead in visual quality but lag in camera execution.

Consistency Metrics

We quantify input preservation and generated-content persistence, revealing complementary strengths across models.

Gen3C leads input preservation and Return, while SolarWM‑H3 leads Query and Cosmos3‑Nano leads Identity.

Table 3 shows Gen3C with the highest Tier Score (75.23) and the other models topping their respective columns.

How faithfully a model keeps the original scene when it revisits the same viewpoint or returns after exploring elsewhere.

How consistently a model maintains the visual content of generated frames across different camera poses.

All models were evaluated under the same exploration‑and‑revisit protocol, so differences stem from the models themselves rather than data or camera variations.

Quality and Control Relationships

Quality and control metrics correlate weakly with 3D consistency, revealing distinct failure modes.

Visual quality correlates only weakly with 3D consistency across models.

Rank correlation is 0.33 for world models and 0.21 for TI2V models, indicating that high‑quality frames often lack spatial coherence.

All models were evaluated under the same Orbia benchmark settings, using identical exploration‑and‑revisit templates and the same evaluation metrics.

**Table 4.** Tier 4: Detailed generated-content persistence results. Scores use a 0–100 scale ($\uparrow$). Models are ordered by Tier Score within each group. 1st, 2nd, 3rd mark the top three scores per column and group. Same and Diff. denote direct comparisons at identical and different requested poses. Reproj. denotes geometry-aligned comparison.

Trajectory and Long Exploration

Ablation analysis reveals TI2V models struggle with camera control and long‑term consistency across trajectories.

Across the six trajectory templates we observe systematic weaknesses: TI2V models consistently trail world models in camera control, and all models lose performance as exploration length grows.

TI2V models fall behind world models in camera control on every trajectory template.

On short forward–backward videos TI2V scores 44.05 versus 62.41 for world models; on orbit trajectories the gap widens to 12.95 versus 53.15. Short world‑model orbit scores are 53.15 (control), 61.83 (input‑preservation), 60.97 (generated‑content‑persistence). Long world‑model peek‑and‑return achieves the highest control (55.09) but the lowest generated‑content‑persistence (37.32).

Control performance degrades sharply during long‑duration exploration.

Across eight leading models the average control drop is 26.98 points; for TI2V models the average drop is 23.25 points. SolarWM‑H3 falls from 91.91 to 64.50, Cosmos3‑Super from 60.26 to 34.94. Local geometry declines by 7.13 points on average while visual quality drops only 2.08 points. Wan2.7 loses 16.26 geometry points but only 3.11 quality points; Cosmos3‑Super’s quality improves slightly while geometry falls 5.34. Forward‑backward scores fall from 66.21 to 40.96 (world) and 67.74 to 44.58 (TI2V); peek‑and‑return drops from 64.29 to 37.32 (world) and 69.52 to 42.83 (TI2V).

**Figure 5.** Cross-tier ranking correlations. Top: world models. Bottom: TI2V models. Left: short videos. Right: long videos. Cells show Spearman correlations between model-level tier scores.

**Figure 6** Performance across trajectory templates. Top: *world models*. Bottom: *TI2V models*. Left: *short videos*. Right: *long videos*. Cells show **mean tier scores** with equal model weights within each evaluated model group.

Scene Variation and Human Alignment

Analyzes how scene source affects model performance and how automatic scores align with human judgments.

We first examine how different scene sources influence model scores. Figure 9 visualizes the mean tier scores for each source, separating short and long videos. The heatmap reveals distinct strengths for each source.

UE‑generated videos achieve the highest short‑video control scores among all sources.

World‑model control = 63.6 and TI2V control = 37.1, both exceeding scores from ScanNet++, public videos, and self‑captured scenes.

Public‑video scenes yield the highest visual‑quality scores for both model families.

World models score 74.3 (short) and 72.2 (long); TI2V models score 79.1 (short) and 78.9 (long), surpassing all other sources.

To validate whether automatic metrics reflect human perception, we use the WorldLens benchmark. Ninety cases covering six models are rated by twenty participants, who rank videos on realism, plausibility, consistency, and safety. Aggregated human preferences correlate with automatic scores between 0.859 and 0.921 (Spearman), with temporal stability showing the highest agreement.

**Figure 9** Performance across data sources. Top: *world models*. Bottom: *TI2V models*. Left: *short videos*. Right: *long videos*. Cells show mean tier scores with equal model weights. Dashes indicate unavailable groups.

Conclusion and Contributions

We synthesize benchmark insights, outline contributions, and preview temporal performance trends.

Our benchmark, comprising 720 exploration‑and‑revisit cases, probes three core aspects: input preservation, generated‑content persistence, and 3D self‑consistency. Across 27 diverse models we find that high visual quality and precise control do not imply spatial consistency, and that good cross‑view agreement can mask poor scene reconstruction. Moreover, preserving the initial scene does not predict whether newly generated content survives a later revisit, especially as exploration extends to minute‑long durations where control degrades and geometry deteriorates.

Project leadership was provided by Dongyue Lu and Shengju Qian. Benchmark and metric design involved Dongyue Lu, Hongshan Chen, Hur Lim, Yanrong Wang, Yonggang Wu, Guian Fang, and Shengju Qian. Data collection and curation were performed by the same team. Simulation data were constructed in Unreal Engine by Hongshan Chen, Hur Lim, Yanrong Wang, and Yonggang Wu. Model integration and experiments were carried out by Dongyue Lu, Guian Fang, Yue Liao, Wei Chow, and Shengju Qian. Writing and editing involved Dongyue Lu, Guian Fang, Yue Liao, Lingdong Kong, Tianxin Huang, and Yu Yang. Discussions and insights were contributed by a broader group of collaborators, and advising was provided by senior researchers.

The appendix details the dataset and evaluation pipeline. Appendix A presents dataset statistics, exploration‑and‑revisit templates, and annotated examples. Appendix B describes construction from public videos, ScanNet++, UE simulations, and self‑capture, plus extensions to one‑minute explorations and quality ranking. Appendix C enumerates the six evaluation tiers—control alignment, visual quality, input preservation, generated‑content persistence, 3D self‑consistency, and temporal stability—followed by the aggregation protocol.

**Figure 8.** Temporal performance for TI2V models. Top: short videos. Bottom: long videos. Rows follow the model order in Table 2. Time increases from left to right. Cells show mean window scores for quality, control, and local geometry on a shared 0–100 scale, rounded to integers. Darker colors indicate higher scores within each evaluated time window.

Appendix: Metric Details

Metrics and dataset details for evaluating video world models.

Section D introduces three families of model adapters—camera‑conditioned, action‑conditioned, and TI2V (Text‑to‑Image‑to‑Video)—that define how generation settings are tied to external signals.

Section E reports tiered evaluation results, from low‑level control alignment up to high‑level temporal stability, each corresponding to a specific metric defined earlier in the paper.

Sections A and B describe the dataset construction pipeline, the exploration‑and‑revisit templates, and the annotation workflow that enable the metrics above.

**Figure 10.** Dataset gallery. Selected input images from the Orbia benchmark illustrate diverse indoor and outdoor environments.

**Figure 11** Dataset gallery. Selected input images from the Orbia benchmark illustrate diverse indoor and outdoor environments.

No caption provided in the image.

**Table 6.** Short and long protocols. Median interval measures time between first visits and their paired revisits.

**Table 7.** Median input-query motion. Translation is camera-center distance in the original reference-pose units.

The figure displays a multi-panel visualization of a living room scene. The top row shows an "Input image" and a "Query image," both featuring colored overlays (red for the television, cyan for the framed portrait). The middle row presents the corresponding "Input depth" and "Query depth" maps, rendered in a purple-to-yellow color scale, alongside a "Trajectory" plot showing a vertical path with markers. The bottom text provides a descriptive caption of the scene.

**Figure 13.** Examples of the forward–backward (top) and lateral return (bottom) templates from ScanNet++.

The figure displays a multi-panel visualization of a street scene. The top row shows an "Input image" and a "Query image," both featuring a narrow cobblestone street with a highlighted white SUV. The middle row presents the corresponding "Input depth" and "Query depth" maps, using a color gradient from dark purple (near) to yellow (far), alongside a "Trajectory" plot showing a path with directional arrows. The bottom section provides a descriptive text summary of the scene.

**Figure 14.** Examples of the forward–backward template from Sekai (top) and the lateral return template from MiraData (bottom).

A sunlit urban alleyway features a paved walkway bordered by a white building and a green metal archway. A vibrant pink tree stands centrally, casting soft shadows on the ground. Yellow protective railings line the base of the building wall, while a simple bench sits in the corner. The architecture includes industrial-style doors and windows, and the overall atmosphere is calm and still, with no visible movement or people. The scene is characterized by clean lines and a mix of natural and man-made elements, creating a serene environment.

A weathered wooden cabin with peeling siding and ivy-covered walls stands adjacent to a muddy dirt road, partially obscured by overgrown grass and white wildflowers. A rustic wooden fence runs along the property line, separating the yard from the path. Stacks of firewood, metal barrels, and green recycling bins are positioned near the house. A yellow diamond-shaped road sign with a winding arrow is mounted on a metal post in front of the cabin. Tall evergreen trees line the under a bright blue sky with scattered clouds. Sunlight illuminates the scene, casting soft shadows across the terrain and highlighting the textures of the wood, foliage, and wet ground. The road surface appears uneven and damp, reflecting the sky in small puddles. The overall atmosphere is serene and slightly overgrown, suggesting a remote or rural setting.

**Figure.** A modern, well-lit interior space features a row of black, geometric lounge chairs arranged on a grey patterned carpet. Adjacent to the seating is a long, light-grey cabinet topped with a white surface, displaying potted plants and a framed piece of calligraphy. A hallway extends towards a bright window, flanked by a vertical green wall and additional white seating furniture. The scene is characterized by clean lines, neutral tones, and a calm, professional atmosphere.

**Figure 18** Anchor selection in the UE editor. A cart is selected as a bounded static landmark, with its scene Actor highlighted in yellow. Its location and bounds guide the initial camera framing and trajectory design.

**Figure 19.** Manual review of a MiraData forward-backward case. The interface combines scene review, annotation scores, per-anchor decisions, reference overlays, sampled frames, and camera trajectories.

Appendix: Camera-Conditioned Models

Overview of data acquisition, quality ranking, case packaging, evaluation tiers, and model adapters.

In Self‑Capture, 85 outdoor videos are recorded around static landmarks; after review 60 cases remain. Geometry is recovered with DA3, preserving timestamp‑RGB‑depth alignment, and anchors are segmented independently in input and query views using SAM3.

Scenes are rejected if any frame crosses a shot boundary, contains prominent moving subjects, suffers severe visibility issues, or includes overlaid watermarks. Peripheral people, parked vehicles, or background vegetation do not trigger rejection.

After selecting sharp reference frames, reviewers verify that segmentation follows the same physical landmark in both views, ensuring mask consistency across the source recording.

For longer cases, the short case’s input image, anchors, and reference views are reused, and a smooth 60‑second trajectory (961 poses at 16 FPS) is synthesized, preserving the final return while regularizing handheld motion.

Quality ranking prioritizes candidates using an automatic score that combines image sharpness, anchor quality, trajectory quality, and revisit quality; Table 8 details the 70‑point component weighting.

This table outlines the scoring criteria for candidate selection, categorized by Image, Anchor, Trajectory, and Revisit, with associated point values and descriptions of the factors considered.

Human reviewers then inspect the remaining candidates, checking for shot continuity, moving subjects, visibility, weather, and watermarks, while recording indoor/outdoor classification. The pass/reject decision is stored separately from the numerical ranking.

Annotations may be corrected: reviewers can replace anchors, adjust queries, or reject cases, with the final choices recorded alongside frame indices, timestamps, and comments for evaluation.

Each released case separates model‑facing inputs (initial RGB, caption, trajectory) from evaluator‑only references (query RGB, depth, masks, calibration). Short cases contain 357 poses; long cases contain 961 poses, both at 16 FPS.

The evaluation protocol defines six tiers, each with explicit metric formulas and aggregation rules, converting all component scores to a 0–100 scale before weighting them into the overall benchmark score.

Tier 1 measures control alignment via translation and rotation errors after global‑scale fitting, and discrete action accuracy for move and look directions, weighting pose and action components equally.

Tier 2 assesses visual quality using five predictors (aesthetic, MUSIQ, flicker, motion smoothness, human preference), each normalized to 0–100 and equally weighted.

Tier 3 (Input Preservation) evaluates scene and anchor fidelity in appearance (PSNR, SSIM, LPIPS, DINO) and geometry (scale‑aligned depth error), plus anchor identity consistency via DINO descriptors across four timestamps.

Tier 4 (Generated‑Content Persistence) compares first‑visit and revisit frames for appearance, depth, and surface agreement, averaging direct and geometric scores.

Tier 5 (3D Self‑Consistency) checks cross‑view depth and feature consistency, regional scale agreement, and reconstruction quality using DA3‑predicted Gaussian scenes.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers