EgoXperience Learning World-Grounded Egocentric Representations from Human Experience

Xinhao Ji, Hui En Pang, Hao Li, Kairui Hu, Jiner Wang, Mingze Sun, Ziang Cao, Yalun Dai, Thang Luu, Qin Ma, Hongyi Chen, Yukang Cao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu

EgoX jointly reconstructs camera, scene, hands, and objects in a shared metric world frame from monocular video.

How can we jointly reconstruct camera motion, scene geometry, hand pose, and object interactions in a consistent metric world frame from egocentric video?

Egocentric video captures rich human interaction, but existing methods typically recover only isolated components like hands or scene depth, leaving them disconnected in scale and coordinate space. EgoX is a unified feed-forward Transformer that jointly predicts camera intrinsics, scene depth, hand meshes, and rigid object poses in a single forward pass. It uses a shared metric scale and camera-conditioned transformations to tie these entities into a consistent world coordinate system. On standard benchmarks, EgoX outperforms specialized baselines, reducing camera error by up to 68.9% and hand-pose error by 65.6% compared to the strongest competing methods.

Paper Primer

The core challenge in egocentric reconstruction is that camera motion, scene geometry, and hand-object interactions are tightly coupled; a change in the image can be caused by any of these factors simultaneously. EgoX addresses this by learning a joint representation where camera, hand, and object tokens exchange information across views, then explicitly composing them through predicted camera poses to ensure geometric consistency.

EgoX achieves state-of-the-art performance in joint world-space reconstruction.

Evaluated against specialized baselines on four benchmarks (HOT3D, EgoX-Bench, H2O, HOI4D), EgoX consistently ranks first in depth, hand, and object metrics. Reduces camera error by 68.9% (HOI4D), hand W-MPJPE by 65.6% (EgoX-Bench), and improves object AUC-S50 by 28.0% (HOT3D).

To enable this joint learning, the authors introduce EGOX-Engine, an automatic annotation pipeline that uses SLAM-based world alignment and geometrically verified object tracking to produce EGOX-Data-6M, a 50-hour dataset of task-driven egocentric recordings.

Why is a unified model necessary instead of just combining existing specialized tools?

Combining outputs from different methods often results in inconsistent scales and coordinate frames, which complicates their use in downstream embodied AI applications that require a coherent world model.

What is the scope of the objects EgoX can reconstruct?

EgoX currently reconstructs rigid objects and estimates their 6-DoF poses; it does not yet predict object meshes or handle articulated objects.

EgoX demonstrates that joint, feed-forward reconstruction of complex egocentric scenes is feasible, shifting the paradigm from modular, multi-stage pipelines to unified, world-grounded estimation.

Introduction

We expose the inconsistency of isolated egocentric reconstructions and propose a unified joint‑reconstruction paradigm.

Current egocentric pipelines recover only a subset of the tightly coupled components—camera pose, scene geometry, hand pose, and object pose—so their outputs must be manually reconciled, which breaks spatial consistency.

EGOXPERIENCE tackles this gap by providing a unified feed‑forward framework that jointly predicts all four components in a single forward pass, backed by a large‑scale, jointly annotated dataset.

It is the task of estimating camera pose, scene depth, hand pose, and object pose simultaneously, all expressed in a common metric world frame.

EGOX‑Engine automatically tracks objects, verifies their poses geometrically, and aligns them with SLAM‑estimated camera trajectories, producing the jointly annotated EGOX‑Data‑6M. From this pool we extract a one‑hour held‑out benchmark (EGOX‑Bench) for comprehensive evaluation.

The shift from isolated task reconstruction to a unified world‑grounded representation enables consistent, metric‑scale egocentric perception.

Related Work

Survey of prior egocentric reconstruction methods and datasets compared to EgoX.

We position EgoX among prior egocentric reconstruction approaches and datasets, highlighting where earlier work stops short of joint hand‑object‑camera reconstruction.

**Table 1.** Comparison of Egocentric Datasets. We compare data scale, capture settings, and annotation coverage. Obj. counts objects with 3D models.

The EgoXperience Paradigm

EgoXperience unifies capture, annotation, and learning for joint metric reconstruction from egocentric video.

Existing egocentric pipelines treat camera motion, scene geometry, and hand‑object interactions as separate problems, which leads to spatial inconsistency when the observer and objects move together.

The paradigm bundles data capture, annotation, and model learning into a single joint‑reconstruction pipeline that predicts camera pose, depth, hands, and objects in a common metric world frame.

EgoX‑Engine resolves the ambiguity between camera motion and object motion by anchoring all annotations to a SLAM‑derived session frame and by recovering metric scale from calibrated depth and AprilTag references.

EgoX‑Data‑6M is a task‑driven dataset that records 50 hours of egocentric video across manipulation, cross‑room, and outdoor locomotion, each annotated in the shared metric frame produced by EgoX‑Engine.

We embed camera, scene, hand, and object tokens into a shared Transformer so that every prediction is conditioned on the same global context and can be composed through the estimated camera pose.

Form the homogeneous coordinate $\tilde{u} = (100, 50, 1)^\top$.

Compute $\hat{K}_t^{-1}\tilde{u} = (0.2, 0.1, 1)^\top$.

Scale by depth: $z \hat{K}_t^{-1}\tilde{u} = (0.4, 0.2, 2)^\top$.

Apply the inverse rotation (identity) and subtract translation (zero): $U_t = (0.4, 0.2, 2)^\top$.

After the ScaleHead predicts $\hat{s}=1.5$, the final world‑frame point is $\hat{s}\,U_t = (0.6, 0.3, 3)$.

This toy calculation shows how a single pixel is lifted into metric space and then uniformly scaled, illustrating the tight coupling between depth, camera pose, and the global scale factor.

How does this joint representation differ from a standard multi‑view reconstruction pipeline?

Standard pipelines first recover geometry per view and then fuse them, treating camera pose as a separate output. Here the Transformer jointly reasons about camera, scene, hands, and objects in a single pass, so the pose, depth, and entity predictions are mutually constrained during inference rather than being post‑hoc aligned.

**Figure 1.** Overview of the EgoXperience Paradigm. Task-driven capture and EgoX-Engine produce EgoX-Data-6M, supporting joint reconstruction with EgoX.

**Figure 2.** Overview of EgoX-Data-6M. Examples and statistics summarize capture settings, task coverage, and aligned multimodal annotations.

**Figure 3.** **Overview of EgoX-Engine.** The pipeline converts egocentric recordings into structured multimodal annotations, including aligned geometry and temporally localized task captions.

**Figure 4.** Overview of EgoX. Shared scene and entity features support joint reconstruction, while predicted cameras and scale place all outputs in a common metric world frame.

Experimental Setup

We evaluate EgoX and baselines on four datasets using uniform 150‑frame clips.

We evaluate camera, depth, hand, and object reconstruction on HOT3D, EgoX‑Bench, H2O, and HOI4D using identical 150‑observation clips for every method. Evaluation sequences are held out from EgoX training, and EgoX receives only RGB and a first‑frame object mask, while baselines may also require estimated depth, intrinsics, per‑frame masks, or metric CAD assets.

Results and Ablations

Quantitative and qualitative results show EgoX outperforms all baselines across hand, object, camera, and depth metrics.

The paper’s core premise is that treating camera motion, scene geometry, and hand‑object interactions as a single joint‑reconstruction problem yields spatially consistent egocentric results.

HaWoR is a baseline that fuses adaptive DROID‑SLAM with Metric3D‑based scale to reconstruct hands in a world frame.

How does HaWoR differ from EgoX’s joint‑reconstruction approach?

HaWoR treats camera motion, scene geometry, and hand pose as separate modules and aligns them only after each is estimated, whereas EgoX learns a unified representation that enforces consistency during inference, reducing accumulated error.

EgoX attains the lowest W‑MPJPE of 38.9 mm on all four benchmarks, outperforming the strongest baseline HaWoR (56.3 mm) and WiLoR + SLAM (328.8 mm).

Table 3 reports EgoX at 38.89 mm, HaWoR at 56.30 mm, and WiLoR + SLAM at 328.81 mm.

**Figure 5.** World-Space Hand and Camera Reconstruction. Qualitative comparisons on HOT3D and EgoX-Bench. Predictions and ground truth are shown in a common world frame.

**Figure 6.** **Camera-Frame Object Pose Reconstruction.** Qualitative comparisons of object 6-DoF predictions with ground truth on HOT3D and EgoX-Bench.

**Figure 7 | Depth Reconstruction.** GT: refined official depth. VGGT/VGGT-$\Omega$: one scale fitted to DA3 Nested depth per sequence; others: predicted metric scale.

Appendix and Supplementary Material

Appendix provides extended related work, dataset details, evaluation protocol, runtime analysis, and extra qualitative results.

The extended related work surveys prior joint geometry, hand‑object, and camera pipelines. It highlights how earlier methods treat camera motion, scene geometry, and hand‑object interaction as separate problems, unlike EgoX which unifies them.

Public dataset cleaning removes erroneous hand annotations and aligns depth, MANO meshes, and object meshes into a common 3D coordinate system. This reduces label noise and ensures consistent metric supervision across tasks.

The evaluation protocol uses four disjoint benchmarks (HOT3D, EgoX‑Bench, H2O, HOI4D) and identical 150‑frame clips for all methods. Hand metrics are PA‑, W‑, and WA‑MPJPE; cameras use nATE and RRE; depth uses AbsRel and $\delta$₁.₂₅; objects use ADD, ADD‑S, and AUC‑S₅₀.

Additional qualitative results extend the main figures with more HOI4D and H2O sequences. Figure 8 shows hand‑camera reconstructions, Figure 9 shows camera‑frame object poses, and Figure 10 shows depth maps, all confirming EgoX’s superior alignment to ground truth.

Training combines the EgoX‑Data‑6M split with six public interaction datasets and several synthetic sources, totaling ~280 hours. The model samples 1–24 observations per input and optimizes all modules jointly with AdamW, cosine decay, and module‑wise gradient clipping.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers