Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning

Zishuo Li, Bowen Yang, Changtao Miao, Kai Zhu, Hao Chen, Qingze Guan, Zhengxing Wu, Wanke Zhan, Yang Sun, Zhiyi Huang, Zitong Shan, Zhenchao Jin, Jiadong Hong, Taowen Wang, Yushi Feng, You Liu, Yibo Wang, Yifan Yang, Zhaowen Zhou, Man Luo, Hao Cheng, Bo Zhang, Jianshu Li, Jiansheng Cai, Guocai Yao, Jize Zhang, Chenhao Lin, Renjing Xu, Lequan Yu, Chao Shen, Chunhua Shen, Zhe Li

Open-AoE provides a 2,000-hour egocentric manipulation dataset and end-to-end toolchain for embodied AI training.

How can we build a scalable, high-quality egocentric manipulation dataset using consumer smartphones and a unified toolchain for robot learning?

Embodied foundation models require large-scale, structured interaction data, but existing egocentric datasets are often fragmented, rely on specialized hardware, or lack the geometric and temporal alignment needed for robot learning. The authors introduce Open-AoE, an open infrastructure that uses a smartphone-based capture pipeline to produce structured training samples—including hand poses, camera trajectories, and action segments—from raw video. The dataset covers 2,000 hours of manipulation across 400+ smartphone models, providing a unified toolchain for visualization, 4D reconstruction, and cross-embodiment retargeting.

Paper Primer

Open-AoE functions as a complete data-to-model loop: it gates raw smartphone capture with on-device quality checks, processes clips into semantically labeled and geometrically reconstructed samples, and provides downstream adapters for Vision-Language-Action (VLA) policies, World Action Models (WAMs), and World Models. The core move is the creation of a "data contract" that maintains temporal and geometric alignment from raw capture through to model-ready representations.

Open-AoE achieves superior visual and semantic diversity compared to existing egocentric datasets.

Evaluated across 3,000 balanced trials using CLIP embeddings, Open-AoE consistently ranks first in metrics including Effective Rank (97.43) and k-NN Domain Mixing (0.0775). It maintains higher local connectivity to other visual domains and spans a larger number of active feature directions than OpenEgo, EgoDex, or EgoXtreme.

Why is a unified toolchain necessary for this dataset?

Different embodied models—such as VLA policies, WAMs, and world models—require different invariants, such as dense hand articulation versus end-effector targets. The toolchain allows researchers to select the appropriate representation level from the same synchronized source signals without rebuilding the processing stack.

How does this dataset handle the "embodiment gap" between human and robot?

The toolchain supports cross-embodiment retargeting by mapping human MANO hand poses and camera-local trajectories to robot-executable joint or end-effector targets, while also providing "robotized" videos that composite simulated robots into the original egocentric scenes.

Open-AoE shifts the focus from static video collection to a reusable data-production ecosystem, enabling researchers to train embodied models on diverse, smartphone-captured human manipulation data without needing specialized hardware or private processing pipelines.

Overview of Open-AoE

We introduce Open‑AoE, a smartphone pipeline delivering robot‑ready egocentric manipulation data.

Embodied intelligence needs massive, high‑quality interaction data, yet existing egocentric collections fall short of robot‑ready formats. Open‑AoE fills this gap by delivering a complete pipeline from smartphone capture to model‑training assets.

Robots require large, structured interaction datasets, but current egocentric sources lack both scalable capture and manipulation‑level annotations needed for training.

**Figure 1.** Overview of Open-AoE. Open-AoE provides independent modules for smartphone capture, structured data processing, and downstream robot learning, supporting 2,000 hours of data across 500+ contributors, 400+ device types, 400+ scenes, and 8,000+ tasks.

The first release offers roughly 2,000 hours of manipulation video collected by over 500 contributors using more than 400 distinct smartphones, covering 8,000+ atomic tasks.

Each recording is enriched with textual descriptions, MANO‑based hand poses, camera trajectories, and temporally localized atomic actions.

The processing pipeline runs four stages: (1) temporal action segmentation, (2) semantic annotation, (3) hand reconstruction, and (4) camera‑trajectory reconstruction.

The downstream toolchain adds visualization, cross‑embodiment retargeting, model‑specific data conversion, and ready‑to‑run training recipes for VLA policies, whole‑body agents, and world models.

Open‑AoE provides a complete path from smartphone capture to robot‑ready training data.

The Embodied Data Gap

Identifying the data infrastructure gap that hampers embodied model scaling from egocentric video.

Embodied foundation models depend on large, high‑quality interaction data, unlike language models that can learn from web text. Robot learning requires demonstrations that capture perception, hand motion, object contact, and temporally extended tasks. Scaling studies show that diversity across environments and objects can outweigh sheer quantity of demonstrations.

Egocentric video offers a natural, scalable source for such data because it records from the actor’s viewpoint, capturing hands, objects, scenes, and action progress in a single stream. Compared with third‑person video it aligns better with robot execution, and compared with teleoperation it can be collected continuously and cheaply in everyday settings. Recent datasets (e.g., Ego4D, EPIC‑KITCHENS, EgoDex, EgoLive, OpenEgo, EgoScale) have expanded scale, hand‑pose annotation, and manipulation scenarios.

Nevertheless, the community lacks an open data infrastructure that can continuously collect, systematically process, and directly support model training. Existing pipelines are fragmented, hardware‑dependent, or produce raw video without the hand motion, camera trajectory, and action boundaries needed for robot training. This gap motivates Open‑AoE, a smartphone‑based capture pipeline and toolchain that delivers structured, retargetable training assets.

Existing Egocentric Datasets

We situate Open‑AoE among existing egocentric datasets and outline related application categories.

Section 2.1 surveys publicly released egocentric corpora that differ mainly in the supervision they expose beyond raw video.

Ego4D is a massive collection of first‑person video spanning diverse daily activities, intended as a general‑purpose foundation for egocentric perception research.

EPIC‑KITCHENS‑100 captures kitchen‑centric activities with synchronized audio, dense verb‑noun action labels, and transcribed narrations, enabling language‑grounded manipulation learning.

OpenEgo aggregates over a thousand hours from six existing egocentric datasets, normalizing hand‑pose representations and aligning action primitives to a common schema.

Section 2.2 reviews how these corpora have been turned into actionable supervision for embodied learning.

Reconstruction pipelines (e.g., HaWoR, EgoInfinity) convert raw video into world‑space hand‑object trajectories that can be retargeted to robots; vision‑language‑action methods (Being‑H0, EgoVLA, VITRA) learn action‑aware representations directly from human video; latent‑action world models (LaWAM, DreamZero) infer compact motion surrogates for zero‑shot policies; and large‑scale predictive world models (DreamDojo, iVideoGPT, Ctrl‑World) exploit long egocentric streams for generalist dynamics.

The Open-AoE Capture Pipeline

Describes the end‑to‑end capture pipeline that turns raw egocentric video into robot‑ready training assets.

The dataset is built around a single end‑to‑end pipeline that starts on a smartphone and ends with a fully anonymized, robot‑training‑ready corpus.

**Figure 2.** Open-AoE data processing pipeline. Starting from edge-side on-device detection and control, raw egocentric videos pass through offline quality checking and scene labeling (image-based detection, video slicing, and large-model detection), followed by pose reconstruction, atomic-action annotation, and three-gate quality inspection to produce an anonymized training corpus. The released subset contains approximately 2,000 hours of egocentric manipulation video captured with consumer smartphones, together with English annotations, MANO hand poses, and camera trajectories.

The pipeline works like an assembly line: a lightweight on‑device filter keeps only useful hand‑object clips, then successive offline stations clean, label, reconstruct, and finally inspect the product before shipping a robot‑ready dataset.

How does this pipeline differ from simply recording video and annotating it later?

Instead of a raw dump, the on‑device gate discards irrelevant footage before it ever leaves the phone, and the offline stages enforce a strict privacy‑first contract (face pixelation, metadata scrubbing) while automatically generating pose and action annotations. The result is a ready‑to‑train corpus rather than a collection that still needs massive manual cleaning.

The segment is split into two Parts because a brief hand‑occlusion at second 4 creates an unqualified sub‑span.

Each Part is passed to the large‑model; Part A receives the label “kitchen → counter → cutting → knife”, Part B receives “kitchen → counter → cutting → vegetable”.

Hand pose reconstruction yields MANO parameters for 120 frames in Part A and 95 frames in Part B.

All three quality gates pass, so both Parts are added to the released corpus.

The pipeline turns a single raw video into multiple, fully labeled training examples while guaranteeing that every retained frame meets privacy and quality standards.

Edge‑side online detection runs lightweight models that trigger recording when both hands are visible, pause when they leave the frame, and give voice prompts if a hand touches the image border.

Offline quality check begins with image‑based detection (rule‑based frame filters, hand re‑check, security erasure, calibration validation), then slices the video into qualified Parts, and finally runs a large‑model (Qwen 3.7‑Plus) to assign a four‑level hierarchical label tree.

Reconstruction and Annotation

We audit a random 100‑hour sample to reveal the dataset’s multi‑level diversity.

Existing egocentric datasets are dominated by a handful of scenes, actions, and a single camera family, which limits a robot’s ability to generalize across real‑world manipulation contexts.

The audit treats the 100‑hour sample as a micro‑cosm and extracts orthogonal statistics—semantic, contributor, device, and optical—to quantify how broadly the data span the real world.

How does this audit differ from a simple “total number of videos” count?

Simple counts ignore how the data are distributed across dimensions that matter for robot learning—scene context, hand‑object interactions, camera optics, and contributor habits. By normalizing (e.g., density) and reporting long‑tails, the audit reveals whether the dataset truly offers varied experiences or merely a large volume of redundant clips.

Convert minutes to total minutes: 3 min (already in minutes).

Apply the density formula: $45 / 3 = 15$ segments per minute.

The resulting density of 15 segments/min indicates a very fine‑grained annotation compared with the full dataset’s 13.97 segments/min.

Density captures annotation granularity independent of raw video length, exposing whether a dataset provides enough temporal detail for learning fine‑grained manipulation.

Open‑AoE’s strength lies in three coupled diversity axes—what is manipulated, where it happens, and the consumer‑camera domain that records it.

All reported distributions are derived from the same random 100‑hour sample; the hypothesized benefit of camera‑domain diversity remains a training hypothesis that must be validated by downstream ablations.

The Toolchain for Robot Assets

Turn egocentric video into 4D assets, robot motion, and robot‑augmented video.

Egocentric recordings give rich visual cues but lack a unified, robot‑ready representation. Existing pipelines either discard the underlying physical signals or force a single action vector that cannot serve both human‑motion modeling and robot control. The Open‑AoE toolchain resolves this tension by preserving the raw multimodal evidence and branching it into task‑specific formats.

The toolchain takes a synchronized egocentric capture and simultaneously produces (i) a 4‑D hand‑object digital twin, (ii) robot‑executable motion, and (iii) a robot‑augmented video, all while keeping the original physical measurements intact.

How does this toolchain differ from a conventional motion‑capture retargeting pipeline that simply records a single joint trajectory?

Conventional pipelines collapse the multimodal egocentric signal into one trajectory, discarding camera motion, hand geometry, and action semantics. The Open‑AoE toolchain keeps all those signals separate, allowing each downstream task to consume exactly the subset it requires—e.g., 4‑D reconstruction uses full hand‑object geometry, while retargeting uses only wrist and finger synergies, and robotized video re‑uses the original background.

Align the two frames using the provided camera trajectory, yielding a pixel‑to‑meter scale of 0.001 m/px.

Run MANO on each frame to obtain hand pose matrices; compute the 3‑D wrist positions (0.12 m, 0.15 m) in world coordinates.

Feed the wrist trajectory and object 6‑DoF estimate (derived from the hand‑object contact) into the 4‑D reconstruction module, producing a time‑varying hand‑object mesh.

Pass the same wrist trajectory to the Phantom retargeting module, which solves an inverse‑kinematics problem for a Unitree G1 arm, yielding joint angles ($\theta$₁=0.3 rad, $\theta$₂=‑0.1 rad, …).

Use the MANO keypoints as masks to inpaint the human arm in the original frames, then render the G1 arm with MuJoCo and composite it back, producing robotized video.

This tiny example shows how a single egocentric clip can be turned into three perfectly aligned assets without losing any of the original physical measurements.

Ingest synchronized RGB video, camera intrinsics, trajectory, bilateral MANO meshes, validity masks, and atomic‑action timestamps.

Construct a unified evidence layer that timestamps all modalities on a common timeline.

Branch 1 – 4‑D reconstruction: combine calibrated intrinsics, metric‑scale motion, and MANO priors to generate time‑varying hand‑object meshes and contact cues.

Branch 2 – Cross‑embodiment retargeting: extract wrist and finger trajectories, feed them to robot‑specific IK or dex‑retargeting modules (Phantom, Galbot, Retarget Lab) to produce joint‑space commands.

Branch 3 – Robotized video synthesis: use MANO keypoints as segmentation prompts, apply video inpainting to remove the human arm, render the target robot with MuJoCo, and composite the result onto the original frames.

Export the three aligned assets for downstream learning or evaluation.

**Figure 4.** AoE-Reconstruct-Retarget. A smartphone-captured egocentric video is projected into 4D hand-object assets, robot-usable motion, and robotized video. The repository integrates reconstruction interfaces for EgoInfinity and Do-as-I-Do, multiple retargeting backends for G1, Galbot/Galaxea, Sharpa, and XHand, and simulation and egoview synthesis for result validation.

Adapting Data for Downstream Models

We define a spectrum of training‑ready action representations that map synchronized egocentric data to robot models.

The downstream models need different invariants—hand articulation for human‑motion prediction, end‑effector targets for cross‑embodiment control, and camera‑relative dynamics for video prediction—so a single egocentric stream must be projected into several complementary representations.

Instead of forcing every robot learner to consume the raw egocentric video, we expose a small set of purpose‑built vectors that capture exactly the quantities each downstream task cares about.

Form the per‑hand 55‑D block by stacking the fields in the order defined by the spec.

Repeat the same construction for the opposite hand (identical numbers for illustration).

Concatenate the two 55‑D blocks → a 110‑D vector: [1, 0.1, 0.0, 0.2, 0, 0, 0, 0.01, 0, 0, 0.2, 0.2, 0.2, 0.2, 0.2] × 2.

The resulting vector lives in the current‑camera frame; any downstream model that expects 110‑D dense MANO can ingest it directly.

This toy construction shows that the “dense” interface is just a deterministic packing of primitive signals—no learning is required to obtain it, and the same code can generate it for every frame of a segment.

**Figure 5.** AoE-Training-Ready. A synchronized AoE segment is projected into action representations with different supervision densities and semantic roles. Integrated recipes connect dense MANO, robot-facing, hand-plus-camera, and latent or weak action representations to Vision-Language-Action policies, World Action Models, and World Models.

Because the same egocentric evidence is reused at geometric, action, and visual‑dynamics levels, researchers can experiment with any downstream recipe—policy learning, video prediction, or latent‑action discovery—by simply selecting the appropriate vector column from the table.

Dataset Diversity and CLIP Analysis

Open‑AoE shows superior visual diversity across six CLIP‑based metrics.

Open‑AoE achieves the highest mean on all six CLIP‑based visual‑diversity metrics.

Effective Rank 97.43, Participation Ratio 40.47, Meaningful Coverage 19.89, Normalized Cluster Entropy 0.712, Effective Clusters 16.29, and k‑NN Domain Mixing 0.0775, averaged over 3,000 balanced trials.

**Figure 6.** High-dimensional CLIP feature analysis. Bars report the mean over 3,000 balanced random subsamples, with 2,000 embeddings drawn from each dataset in every trial. Open-AoE has the highest mean in all six metrics.

**Figure 3.** Open-AoE collection, sensor, and semantic diversity in a random 100-hour sample. The nested donut summarizes the consumer-phone brand and market-model composition, and the lower-left panel shows the horizontal-FOV distribution. The center panels report collection-context composition and anonymous collector-ID coverage, while the word clouds summarize atomic-action verbs, manipulated objects, and task-language terms. All panels use the same randomly sampled 100-hour subset; wedge area, bar height, and word size encode relative prevalence or frequency.

Semantic Breadth and Consistency

Open‑AoE delivers near‑perfect temporal coverage and the highest visual‑annotation consistency.

Open‑AoE achieves the highest image‑annotation consistency among evaluated egocentric datasets.

Mean Idefics2 consistency score of $4.58$ / $5$, compared to $3.03$ for OpenEgo, $2.92$ for EgoDex, and $2.02$ for EgoXtreme (Fig. 8).

Open‑AoE provides near‑complete temporal coverage of the video timeline.

Temporal coverage of 99.99 % versus 50.1 % for OpenEgo (Fig. 7).

**Figure 7.** Semantic breadth and temporal annotation characteristics across datasets. (a) Number of unique entries in the primary released action-annotation field versus temporal annotation coverage, with marker area proportional to annotation density, measured as the number of annotated segments per minute. (b) Mean segment duration and annotation density for each dataset. Because the datasets use different native annotation schemas, vocabulary size reflects the breadth of the released semantic supervision rather than a controlled vocabulary comparison under a shared ontology.

**Figure 8.** Sequence-level image-annotation consistency across datasets. (a) Sequence-macro means, sequence-level medians, and 95% bootstrap confidence intervals for the mean Idefics2 consistency score. (b) Fractions of sequences in the low ($\le 2$), intermediate ($2 < \cdot < 4$), and high ($\ge 4$) score ranges.

Training Sample Yield

Open‑AoE retains almost all possible training windows and supplies dense multimodal supervision.

Open‑AoE retains 97.8 % of the theoretical training‑window ceiling, delivering 1,760 windows per hour while offering near‑universal hand‑signal and multimodal supervision.

Measured on Open‑AoE versus Open‑Ego, EgoDex, and EgoXtreme using the 2 s history / 2 s future / 2 s stride protocol.

Compared to Open‑Ego (≈1,200 windows /hr) and older egocentric datasets (≈800–1,000 windows /hr), Open‑AoE’s yield is within 2 % of the ideal limit. Hand‑signal coverage is also superior, with 98.93 % of entries containing at least one valid hand and 98.02 % containing both hands, far above Open‑Ego’s 55–66 % range.

**Figure 9.** Training-sample yield and multimodal supervision availability across datasets. (a) Candidate history-future windows obtained per hour using 2 s of history, 2 s of future context, and a 2 s stride, together with the retention rate relative to the no-boundary ceiling of 1,800 windows per hour. (b) Availability of native hand signals, distinguishing entries with at least one valid hand from those with valid signals for both hands. (c) Sequence-level availability of action annotations, bounding boxes, annotation confidence, hand pose, and camera pose; "-" indicates that the corresponding field is not released or unavailable.

Conclusion

Acknowledging the individuals and institutions that built Open‑AoE.

Dataset and Visualization: Zishuo Li∗,1, Qingze Guan∗,5, Zhengxing Wu∗,7, Wanke Zhan∗,1, Yang Sun∗,1, Zhiyi Huang∗,1, Zitong Shan∗,1.

Toolchain and Experiments: Bowen Yang∗,1, Changtao Miao∗,1, Zhenchao Jin∗,3, Jiadong Hong∗,2, Taowen Wang∗,4, Yushi Feng∗,3, You Liu∗,6, Yibo Wang∗,6, Yifan Yang∗,7, Zhaowen Zhou∗,1, Man Luo∗,1, Hao Cheng∗,8.

Project Leaders: Zishuo Li1, Bowen Yang1, Changtao Miao‡,1, Kai Zhu‡,1, Hao Chen‡,2.

Supervisors: Bo Zhang1, Jianshu Li1, Jiansheng Cai1, Guocai Yao6, Jize Zhang8, Chenhao Lin9, Renjing Xu4, Lequan Yu3, Chao Shen9, Chunhua Shen2, Zhe Li1.

Organizations: 1 Ant Digital Technology, Ant Group; 2 Zhejiang University; 3 The University of Hong Kong; 4 The Hong Kong University of Science and Technology (Guangzhou); 5 National University of Singapore; 6 Beijing Academy of Artificial Intelligence; 7 University of Chinese Academy of Sciences; 8 The Hong Kong University of Science and Technology; 9 Xi’an Jiaotong University.

∗Equal contribution. ‡Corresponding authors.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers