Holi-World: Building Holistic World Intelligence from Human-Ego Experience
Hao Li, Xiyuan Zheng, Yuanyuan Gao, Yalun Dai, Xinhao Ji, Xuan-Li Chen, Jing Zhao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu
An integrated ecosystem that automatically converts raw egocentric video into structured 3D world records for training spatial intelligence.
How can we automatically convert raw, uncurated egocentric video into high-quality, spatially and temporally aligned 3D supervision for training world models?
Learning how the physical world is organized requires connecting scene geometry, object states, and interaction histories across time, but scaling this process currently demands expensive manual annotation for every new recording. HoliWorld automates this by reconstructing 3D scene geometry and tracking objects through egocentric video, then converting these records into grounded supervision for training vision-language models. This approach eliminates the need for manual labeling, achieving state-of-the-art performance in 3D perception and substantially improving spatial reasoning capabilities in downstream models.
Paper Primer
The core challenge is the "annotation bottleneck": while egocentric video captures rich, evolving world states, turning these into usable training data for spatial intelligence usually requires human-in-the-loop curation. HoliWorld solves this by building an automated pipeline that treats raw video as a source of structured 3D records rather than just pixels.
HoliWorld-Foundry acts as a data-construction engine: it reconstructs scene geometry via visual-inertial SLAM, then uses vision-language models to route objects into either static 3D detection or dynamic 4D tracking pipelines. The system is a digital archivist: it reads the raw video stream, identifies persistent objects and their interactions, and writes them into a unified 3D representation that links geometry, semantics, and temporal history.
HoliWorld-Foundry significantly outperforms existing baselines in both static 3D object detection and dynamic 3D tracking.
On ScanNet, the system achieves an AP25 of 0.68 compared to 0.40 for the strongest baseline; on Aria Digital Twin sequences, it reaches a HOTA of 54.31% versus 36.00%. The system provides a 70% relative improvement in static detection accuracy and a 50% relative gain in dynamic tracking consistency over previous methods.
Training on automatically constructed HoliWorld-Data substantially boosts the spatial and temporal reasoning of vision-language models.
Fine-tuning Qwen3.5-4B on this data increases Token F1 from 16.35% to 78.02% on HoliWorld-Bench. The model's reasoning performance on spatial benchmarks more than quadruples, demonstrating that automatically generated supervision is highly effective for downstream tasks.
Why is this approach better than simply using existing large-scale egocentric datasets?
Existing datasets like Ego4D often require massive manual annotation to link actions to 3D geometry; HoliWorld automates this recovery, allowing the system to scale to new, unannotated recordings without additional human labor.
What is the scope of the "dynamic" tracking capability?
The current pipeline assumes at most one moving object is being manipulated at any given time, which simplifies the association of identities across interaction segments.
Introduction and Motivation
Introducing HOLIWORLD: scaling 3D annotation from egocentric video via automated reconstruction and verification.
Building robust spatial‑temporal models requires dense 3D supervision, but manual annotation of egocentric recordings does not scale. The core obstacle is a scalability bottleneck: each new video demands costly geometry reconstruction, semantic labeling, and interaction annotation. HOLIWORLD tackles this by automating the end‑to‑end conversion of raw egocentric video into structured 3D supervision.
Learning from a first‑person view that simultaneously captures scene geometry, object identities, and interaction histories, providing a unified signal for spatial and temporal reasoning.
**Figure 1.** **Overview of HOLIWORLD.** HOLIWORLD connects in-the-wild egocentric capture, automated static and dynamic annotation, and vision-language learning. Scene geometry, object semantics, and interaction histories form aligned multimodal supervision for grounding and spatial and temporal reasoning. Representative annotations and results illustrate data coverage, annotation quality, and HOLIWORLD-VL performance.
**Figure 2.** The HOLIWORLD ecosystem. It connects egocentric capture, Foundry, Data, and VL. Human-annotated Bench evaluates annotation quality and spatial understanding.
The primary obstacle to scaling 3D annotation is the manual effort required to turn raw egocentric video into structured supervision.
The HoliWorld-Foundry Pipeline
HoliWorld‑Foundry turns raw egocentric video into structured 3D supervision for static and dynamic objects.
Raw egocentric recordings give dense visual data but no explicit 3‑D object labels; without a pipeline to align geometry, semantics, and motion, training spatial‑reasoning models is infeasible.
Think of the pipeline as a factory line: first a rough scaffold of the scene is built, then parts are routed to specialized stations that either label static furniture or track moving items.
Visual‑inertial SLAM estimates poses $T_1 = I$ and $T_2 = \text{translate}(\Delta p)\,\text{rotate}(5^\circ)$.
DA3 predicts depth maps $D_1$ and $D_2$ with average values 1.2 m and 1.15 m.
Using $K_1 = K_2$ (focal length 800 px, principal point (320,240)), we back‑project each pixel to 3‑D points, forming a sparse cloud $P$.
We pack the observations into $O = \{(I_1,D_1,K_1,T_1), (I_2,D_2,K_2,T_2)\}$.
The VLM reads the two RGB frames and emits the category “chair” with a motion cue “static”, routing the pair to the static perception branch.
Even a tiny translation between frames yields enough parallax for SLAM to anchor the scene, and the VLM’s early categorisation prevents the static detector from chasing the moving camera.
How does HoliWorld‑Foundry differ from a conventional SLAM‑only pipeline?
Standard SLAM stops after producing a point cloud; HoliWorld‑Foundry adds a VLM‑driven router that splits the cloud into static‑object detection and dynamic‑track pipelines, and it augments the geometry with a 3‑D Gaussian representation for high‑quality rendering.
**Figure 3.** **HoliWorld-Foundry overview.** Shared preparation reconstructs geometry and routes objects to static or dynamic perception. Static perception (03) detects, merges, and verifies persistent 3D objects; dynamic perception (04) associates identities and recovers 3D tracks. Both outputs are fused into a unified world state.
Static objects are detected by fusing two complementary sources—image‑level detections lifted into 3‑D and point‑cloud proposals—then filtered and verified.
Why not rely solely on the image‑based detections instead of the dual‑path approach?
Image‑only detections miss large objects that extend beyond a single view and lack depth precision; the point‑cloud path captures those extents, and merging them yields more complete and accurate 3‑D boxes.
**Figure 4.** Static object perception. (A) Complementary image and point-cloud paths generate 3D candidates. (B) Geometric merging and filtering retain a current candidate and one preselected fallback. (C) VLM validation uses each candidate's own evidence, with at most one fallback retry. Synthetic illustrations depict inputs and intermediate states.
Dynamic objects are tracked by first grounding the interaction (hand region + VLM description) to a target identity, then using that identity to drive mask selection and propagation.
How does identity‑guided mask tracking differ from generic mask propagation?
Generic propagation blindly follows the previous mask, which can drift when the object disappears; identity guidance re‑initialises the mask from the VLM‑derived target each segment, keeping the track anchored to the correct object.
**Figure 5.** Dynamic object perception. Cross-segment identity association guides mask selection and within-segment propagation. Valid masked depth yields filtered 3D points and per-frame 3D oriented bounding boxes (OBBs). Linking boxes by identity produces 4D tracks; tracks and interaction evidence jointly support motion cues. Synthetic illustrations depict inputs and intermediate states.
The VLM acts as a quality‑control inspector, checking each candidate box against rendered evidence and either keeping, refining, relabelling, or rejecting it.
Why not trust the initial detection without VLM validation?
Initial detections can suffer from inaccurate geometry (e.g., depth noise) or category confusion; the VLM cross‑checks against the full 3‑D scene, catching errors that single‑view detectors miss.
Static 3D Object Perception
Static 3D perception results on ScanNet, ScanNet++, and HoliWorld‑Bench.
HOLIWORLD‑FOUNDRY achieves 0.55 AP50 on ScanNet (an indoor RGB‑D benchmark), a 0.43 improvement over the strongest baseline.
The next‑best AP50 on ScanNet is 0.12 (SpatialLM); our method reaches 0.55.
ScanNet is a widely used indoor scene dataset that provides RGB‑D scans together with 3D object annotations.
The table compares our method against a range of model‑based, 3DGS‑based, and spatial‑pipeline baselines. Across all three datasets, we lead on the stricter AP50 metric and show consistent gains on mIoU, indicating both better detection and tighter box alignment.
**Table 1.** Static 3D perception on ScanNet, ScanNet++, and HoliWorld-Bench ([0, 1] scale). AP uses 3D IoU thresholds of 0.25/0.50; mIoU uses pairs matched at IoU 0.25. Best and second-best results are highlighted in deep green and light green.
Foundry’s performance advantage on standard benchmarks demonstrates that automated egocentric‑to‑3D supervision can surpass existing static perception pipelines.
Dynamic 3D Tracking Performance
Dynamic tracking results show large HOTA gains over baselines.
HOLIWORLD‑FOUNDRY achieves Higher Order Tracking Accuracy (HOTA) 54.31 % on ADT20 and 43.14 % on HoliWorld‑Bench, outperforming the strongest baselines by +18.3 % and +28.0 % respectively.
Table 2
AssA and PCL metrics are higher for HOLIWORLD‑FOUNDRY, indicating more consistent identities and broader trajectory coverage. Box metrics (IoU, GTC, PP) provide a complementary view of spatial extent accuracy on matched objects, confirming that the predicted 3D boxes align closely with ground‑truth geometry.
ADT20 is a dataset that measures how well a system can continuously locate and track 3D objects over time, requiring both spatial precision and temporal consistency.
**Table 3.** (Note: The table does not have an explicit label in the provided image, but is referenced as Table 3 in the text.)
Downstream Spatial Reasoning
Generated supervision dramatically lifts spatial reasoning performance.
HoliWorld converts egocentric video into structured 3‑D supervision, enabling scalable training of spatial‑reasoning models.
Fine‑tuning Qwen3.5‑4B on HoliWorld‑Data raises Token F1 by over 60 points on HoliWorld‑Bench and by nearly 48 points on EgoDynamic4D, and the resulting HoliWorld‑VL‑4B outperforms zero‑shot Qwen3.8‑27B on all four metrics across both benchmarks.
Table 3 shows the detailed numbers for Token F1, Y/N, MCQ, and Num metrics.
Foundry‑generated data provides strong supervision for training VLMs, yielding large gains in spatial and temporal reasoning.
Ablation Studies
Ablation results reveal each static and dynamic component’s contribution to detection and tracking performance.
Static ablations isolate the contribution of each perception module. Removing the image‑based lifting step drops detection of small objects, while omitting point‑cloud detection harms recovery of large extents. Stereo fusion bridges the two, and caption guidance rescues entities missed by geometry alone. VLM verification adds only modest metric shifts but corrects semantic labels.
Dynamic ablations focus on trajectory consistency. Dropping cross‑interaction label continuity fragments object identities across frames, while removing SAM association leads to temporally inconsistent masks and larger trajectory deviations.
Quantitative ablations are reported in Table 5 (Appendix A.3). Removing any static component lowers AP25 and mIoU, while the dynamic removals decrease HOTA and PCL scores, confirming that each module contributes measurably to overall performance.
**Figure 9.** Qualitative ablation results for the static and dynamic components.
Additional Ablation Results
Component ablations and the full suite of static‑and‑dynamic QA tasks are evaluated.
Table 5 reports component‑wise ablations: static AP25/mIoU ranges from 0 to 1, while dynamic HOTA/PCL is measured at 0.1 m resolution. Check‑marks indicate enabled components, crosses disabled, and dashes denote unavailable settings; rows S6 and D3 correspond to the full system.
We evaluate 30 fine‑grained QA types, divided into 11 static questions (camera motion, object spatial relations) and 19 dynamic questions (trajectory reasoning, caption‑based action understanding).
**Figure 13.** Examples of the 30 QA types in Holi-World, covering static mapping, dynamic trajectory, and dynamic caption reasoning.
The benchmark probes both static scene geometry (e.g., distances, depths) and dynamic motion understanding (e.g., trajectories, action phases) through targeted questions.
The reproducibility appendix (Section 3) details the full HoliWorld pipeline, dataset composition, and ablation configurations to enable exact replication of these results.
Responsible‑use considerations emphasize that HoliWorld should not be deployed for non‑consensual surveillance or high‑stakes decisions without additional safeguards.
Evaluation Metrics and Details
Details of metrics, baselines, and evaluation protocol for dynamic tracking experiments.
The evaluation suite uses several percent‑based metrics. $HOTA_{0.1}$, $DetA_{0.1}$, and $AssA_{0.1}$ follow Luiten et al. [25] with a 0.1 m world‑coordinate center‑distance tolerance, measuring joint detection‑association quality, detection accuracy, and identity‑association consistency respectively. $PCL_{0.1}$ counts correctly localized ground‑truth instance‑frames under a global one‑to‑one assignment, while $R_{0.1}$, $P_{0.1}$, and $TP\!-\!L2$ derive from $HOTA$’s frame‑level matches to report recall, precision, and mean center error (in meters). Box geometry is evaluated separately with IoU, ground‑truth coverage (GTC), and prediction precision (PP).
Four baseline pipelines are compared against our method. EgoSeg3D† receives sequence‑level ground‑truth category queries and uses a GroundingDINO–SAM–DEVA frontend to generate masks before 3D lifting; IT3DEgo∗ is given a predefined target set and converts the first‑visible ground‑truth mask to a bounding box for its native SVOE interface; SAM+Whareformer∗ and SAM+LMK∗ share a SAM2 propagation frontend initialized with each target’s ground‑truth mask at its first visible frame, then propagate masks for association. Our approach uses no annotation‑derived test‑time inputs.
Evaluation runs on 20 sequences from ADT and HoliWorld‑Bench, covering the full RGB timeline and including out‑of‑view periods whenever a valid ground‑truth pose exists within 20 ms. Missing predictions count as failures; sequence scores are averaged equally. Pipelines are grouped by whether they output 3D boxes, and binding uses nearest world‑coordinate centers, with an additional IoU > 0.5 requirement for box‑producing pipelines. All methods are then scored with the distance‑based metrics described above.
**Figure 10.** Qualitative comparison of static 3D object perception across methods and scenes. Panels (a) and (b) show complementary subsets of the evaluated methods.