Robotok: An Internet-Scale Data Engine for Human Demonstration Retrieval and Dexterous Manipulation Learning

Howard Qian, Yiting Chen, Yunfei Xie, Kejia Ren, Podshara Chanrungmaneekul, Gaotian Wang, Bowen Wen, Chen Wei, Kaiyu Hang

RoboTok retrieves human manipulation demonstrations from internet video by matching egocentric 3D hand trajectories.

How can we efficiently retrieve relevant robot manipulation demonstrations from massive, uncurated internet video datasets?

Robot learning is bottlenecked by the high cost of collecting diverse, high-quality demonstration data for dexterous manipulation. While internet video contains vast amounts of human activity, it is unstructured and visually heterogeneous, making it difficult to identify clips that share the same underlying manipulation behavior. RoboTok solves this by canonicalizing 3D hand keypoints into an actor-centered reference frame, allowing the system to compare motion trajectories regardless of camera viewpoint or scene appearance. It trains a lightweight encoder to map these trajectories into an embedding space that preserves motion similarity. On retrieval benchmarks, RoboTok significantly outperforms existing motion-based retrieval methods. Downstream, policies guided by RoboTok-retrieved demonstrations show substantial gains in success rates on complex manipulation tasks compared to those guided by random or baseline-retrieved data.

Paper Primer

The core challenge is that visual or semantic similarity in web videos does not guarantee behavioral similarity. RoboTok addresses this by treating manipulation as a 3D trajectory problem: it canonicalizes hand poses into an actor-relative frame and uses Dynamic Time Warping (DTW) as a supervision oracle to train a retrieval embedding.

The retrieval engine is a two-stage system: an offline indexing stage that embeds trajectories into a unit hypersphere, and an online retrieval stage that performs efficient cosine-similarity search. The encoder is trained using anchor-centered batches that emphasize relevant neighbors while pushing boundary negatives away, ensuring the embedding space respects the motion-based ranking defined by the DTW oracle.

RoboTok achieves superior retrieval quality compared to existing motion-based baselines.

On the RoboTok evaluation corpus, RoboTok achieves an MRR@20 of 0.8576, compared to 0.0824 for the next-best baseline (STRAP). A ~10x improvement in Mean Reciprocal Rank over the strongest baseline.

Demonstrations retrieved by RoboTok significantly improve downstream policy success rates in dexterous manipulation.

On the harder VTDexManip benchmark, RoboTok-guided policies reached 79.3% success on Lever Sliding, outperforming the next-best method (HAND) by 59.8%. Drastic performance gains in tasks where dense reward shaping is removed.

Why is an actor-centered reference frame necessary for this task?

Internet videos vary wildly in camera viewpoint and actor occlusion. By transforming hand poses into a torso-centered frame, the system can compare manipulation behaviors across different scenes and viewpoints, which would be impossible if using raw camera-space coordinates.

How does this approach differ from simply using a vision-language model to search for tasks?

Vision-language models often rely on semantic labels or visual appearance, which can be misleading for manipulation. RoboTok focuses on the kinematic structure of the hand motion itself, ensuring that retrieved clips are physically relevant for a robot to imitate, regardless of the objects or environment shown.

RoboTok demonstrates that internet-scale video can serve as a continuously growing, non-parametric source of supervision for robot learning, provided the retrieval is grounded in motion trajectories rather than visual semantics.

Introduction

We expose the need for motion‑trajectory retrieval to overcome data scarcity in robot learning.

Robot learning suffers from a chronic shortage of diverse demonstration data, and scaling manual data collection to cover the long tail of real‑world tasks is prohibitively expensive.

Robots need a continuously growing source of varied manipulation examples, but existing datasets are limited and costly to expand.

**Figure 1.** RoboTok is an internet-scale data engine that indexes human demonstration clips from the web by their 3D egocentric hand trajectories. The figure shows a t-SNE visualization of RoboTok embeddings, learned using a spatiotemporal motion-alignment metric to organize manipulation trajectories. For visualization, we display 2D camera-space hand poses, while retrieval operates on the underlying 3D egocentric representation. This latent motion space enables efficient retrieval of demonstrations for downstream dexterous manipulation policy training. Although semantic labels are not used for curation or retrieval, the learned embeddings still organize into coherent categories.

The shift from visual‑semantic retrieval to motion‑trajectory retrieval unlocks scalable, behavior‑focused supervision for dexterous robot learning.

Related Work

We situate RoboTok among prior large‑scale robot demonstration and retrieval approaches.

Large‑scale robot demonstration datasets such as Octo, OpenVLA, SpatialVLA, and CLIP‑RT have driven recent progress by training on diverse tasks, environments, and embodiments. Collections like BridgeData V2, DROID, and teleoperation systems (e.g., ALOHA) further increase the quantity of directly recorded robot demos, yet expanding coverage still demands costly data collection or complex generation pipelines.

Human video offers a natural source of supervision for dexterous hands because the morphology matches that of humanoid robots. Datasets such as HO‑Cap and EgoVerse provide structured human demonstrations, while methods like VideoDex, EgoMimic, K‑VIL, YODO, SPOT, and CHORD extract motion or contact cues to supervise manipulation. However, these purpose‑built collections remain limited in scale compared with web‑scale video corpora like HowTo100M and Action100M.

Retrieval‑based approaches aim to reduce data acquisition cost by selecting relevant experience from existing corpora. Prior work includes FlowRetrieval (optical‑flow matching), HAND (2‑D hand‑path matching), STRAP (dynamic‑time‑warping of visual features), SiMDex (semantic‑motion mining), and RfV (language‑guided video search). These methods typically rely on fixed robot datasets, image‑space motion representations, or semantic cues, limiting their ability to exploit the vast, heterogeneous internet video pool.

FlowRetrieval uses dense optical‑flow fields to find prior robot demos whose motion patterns resemble a target task.

HAND first filters robot play data by visual similarity, then aligns 2‑D hand trajectories with robot end‑effector paths.

STRAP temporally aligns visual foundation‑model features with query sub‑trajectories using dynamic‑time‑warping (DTW).

**Table 1.** Comparison of demonstration retrieval methods for robot learning.

Problem Formulation

We describe how canonicalized hand trajectories and a DTW‑supervised embedding enable fast, motion‑focused retrieval.

Existing baselines rely on raw visual cues, making retrieval brittle to camera angle and background clutter. Our method sidesteps this by indexing motion directly, using a compact representation of hand motion that is invariant to viewpoint.

Instead of raw pixels we describe a clip by the 3‑D positions of the hand joints expressed in a coordinate frame attached to the actor—so the same gesture looks identical regardless of where the camera sits.

Frame 1: wrist $(0,0,0)$, fingertip $(0.1,0,0)$.

Frame 2: wrist $(0,0,0)$, fingertip $(0.12,0.02,0)$.

Frame 3: wrist $(0,0,0)$, fingertip $(0.15,0.04,0)$.

Transforming to the wrist‑relative frame subtracts the wrist position, leaving fingertip vectors $(0.1,0,0)$, $(0.12,0.02,0)$, $(0.15,0.04,0)$.

Stacking these vectors yields the canonicalized trajectory $[(0.1,0,0), (0.12,0.02,0), (0.15,0.04,0)]$.

Canonicalization removes any effect of camera translation or rotation, making two clips of the same hand motion directly comparable.

Why does anchoring the trajectory to the wrist make it invariant to viewpoint?

Because the wrist defines a consistent origin for the hand; any global camera rotation or translation adds the same offset to both wrist and fingertip, which cancels out when we subtract the wrist position.

DTW works like stretching two rubber strings of hand poses until matching peaks line up—allowing one sequence to speed up or slow down locally so that corresponding gestures line up despite timing differences.

Compute pairwise Euclidean distances: $d_{11}=0$, $d_{12}=0.14$, $d_{13}=2.24$, $d_{21}=1.41$, $d_{22}=0$, $d_{23}=1.41$, $d_{31}=2$, $d_{32}=1.41$, $d_{33}=0.14$.

DTW selects the path $(1,1)arrow(2,2)arrow(3,3)$ with total cost $0+0+0.14=0.14$.

Lengths are $L_A=L_B=3$, so $s(A,B) = -0.14 / 3 \approx -0.047$.

Even though the second trajectory is slightly shifted, DTW aligns the corresponding poses and yields a high similarity.

How does DTW differ from simply averaging Euclidean distances between aligned frames?

Simple averaging forces a one‑to‑one correspondence, which fails when one clip is slower or faster. DTW relaxes this by allowing many‑to‑one or one‑to‑many matches, effectively warping time to achieve the best alignment.

Think of $\Gamma$ as a compressor that turns a long hand‑trajectory into a short fingerprint on the unit sphere, while guaranteeing that two fingerprints that are close together correspond to trajectories that DTW would rank as similar.

Compute inner products: $\langle\Gamma(q),\Gamma(A)\rangle = 0.9\cdot0.88 + 0.1\cdot0.12 = 0.804$.

$\langle\Gamma(q),\Gamma(B)\rangle = 0.85\cdot0.88 + 0.15\cdot0.12 = 0.782$.

$\langle\Gamma(q),\Gamma(C)\rangle = 0.2\cdot0.88 + 0.98\cdot0.12 = 0.304$.

Ranking by inner product yields $A$ then $B$ then $C$, matching the DTW ranking where $A$ and $B$ are more similar to $q$ than $C$.

The inner‑product ordering reproduces the DTW‑based similarity without ever computing DTW at query time.

Why does preserving the inner‑product order guarantee that the retrieved clips are the ones DTW would deem most similar?

Because the training objective explicitly enforces that any pair with higher DTW similarity must have a larger inner product; thus the learned geometry mirrors the DTW ranking, making inner‑product search a proxy for DTW.

**Figure 2.** Internet videos are filtered to retain clips with visible hands and near-static cameras. Hand trajectories are extracted from each clip and reconstructed in metric 3D. To enable view-invariant motion comparison, trajectories are represented in an egocentric coordinate frame. We train a lightweight model to estimate a static torso frame directly from the observed hand wrist frames without requiring visibility of the human body.

The RoboTok Retrieval Engine

RoboTok indexes human hand motions and retrieves nearest‑neighbor demos by motion similarity.

RoboTok solves the data‑scarcity bottleneck by turning internet videos into a searchable motion library. Instead of matching visual appearance, it retrieves demonstrations whose hand trajectories are close in a learned latent space.

The engine builds a compact motion embedding for every egocentric hand trajectory, stores all embeddings in a cosine‑similarity index, and at query time returns the nearest neighbors in that space.

How does RoboTok differ from a visual‑similarity retrieval system?

Visual systems compare raw pixels or appearance features, which vary wildly with camera angle and background. RoboTok first canonicalizes hand poses into an egocentric frame and then learns a motion‑only embedding; similarity is judged purely on hand‑trajectory dynamics, making it robust to viewpoint and scene changes.

Compute inner products: $q\!\cdot\!e_1=0.99$, $q\!\cdot\!e_2=0.98$, $q\!\cdot\!e_3=0.30$, $q\!\cdot\!e_4=0.28$.

Sort scores descending → nearest neighbors are $e_1$, $e_2$, then $e_3$, $e_4$.

Return the top‑2 trajectories ($e_1$, $e_2$) as the retrieved demos.

Because all vectors are ℓ₂‑normalized, the inner product directly reflects cosine similarity, so the retrieval reduces to a cheap dot‑product ranking without any DTW computation.

Pick an anchor trajectory $a$.

Retrieve its $K=20$ nearest neighbors under the DTW oracle, forming the relevant set $R(a)$.

Select two positives $p_1,p_2\in R(a)$ and one boundary negative $n$ just outside $R(a)$.

Form a group $G(a)=\{a,p_1,p_2,n\}$.

Repeat for 49 anchors to obtain a batch of $b=196$ trajectories.

Online retrieval – cosine‑similarity nearest‑neighbor search.

**Figure 3.** Every egocentric trajectory is encoded once into the RoboTok learned motion embedding space. The RoboTok encoder is trained using DTW-derived positives and hard negatives under a combined set and rank objective. At query time, a clip is encoded into the same space and its nearest neighbors are efficiently retrieved via cosine-similarity search.

Retrieval Evaluation Protocol

Evaluating RoboTok retrieval on large‑scale and cross‑dataset corpora.

RoboTok retrieval is evaluated on a total of 100,831 clips spanning in‑domain and cross‑dataset scenarios.

100,000 Action100M clips from the RoboTok corpus plus 831 AssemblyHands clips.

For each query we compute pairwise DTW distances to all candidates, treat the R nearest neighbors (R=20 for RoboTok, R=5 for AssemblyHands) as the relevant set, and compare RoboTok’s trajectory encoder against Random, FlowRetrieval, HAND, and STRAP baselines; the DTW oracle ranking serves as an upper bound.

DTW measures similarity between two temporal sequences by aligning them while allowing local speed variations, yielding a cost that reflects how much one sequence must be stretched or compressed to match the other.

**Figure 4.** Given a query demonstration video of knife cutting, we show the corresponding top-3 retrieved videos from each baseline method compared with RoboTok. Next to each retrieved video, we display its hand trajectories along with the estimated human torso frame. RoboTok retrieves based on the egocentric representation of hand trajectories and retrieves more relevant demonstrations than the baselines according to manipulation behavior.

Retrieval Performance

RoboTok’s retrieval performance eclipses all baselines on the AssemblyHands benchmark.

Recall that RoboTok indexes internet‑scale human videos as 3‑D hand‑motion embeddings, enabling robots to retrieve the most relevant motion sequence for any task query.

RoboTok dominates across all six retrieval metrics on the AssemblyHands corpus, beating every baseline.

Table 2 shows RoboTok achieving the highest scores on mAP@20, nDCG@20, CKNN@20, MRR@20, Kendall $\tau$ and the lowest DTW cost@20, while the GT upper bound provides the theoretical ceiling.

**Figure 5.** Per-$k$ plots of unseen retrieval quality on the RoboTok evaluation corpus: Recall@$k$ (hit rate), mean DTW cost@$k$, and CKNNA@$k$ (DTW alignment). Despite explicitly optimizing for local k=20 neighborhoods, RoboTok also generalizes to global neighborhoods.

The table compares the performance of various methods (Random, Flow, HAND, STRAP, and RoboTok) against a Ground Truth (GT) upper bound across six metrics: mAP@20, nDCG@20, CKNN@20, MRR@20, Kendall $\tau$, and DTW cost@20.

**Table 3.** Retrieval quality measured on the AssemblyHands corpus of two-hand assembly clips against the GT DTW metric at $k = 5$, the size of the relevant set.

Downstream Policy Evaluation

RoboTok dramatically boosts dexterous manipulation success across original and harder benchmarks.

We evaluate policies on the original VTDexManip benchmark and on a harder formulation that restores full 3‑D hand control and removes dense reward shaping, reporting success rates for seen and unseen object splits over eight random seeds per method.

VTDexManip measures a robot hand’s ability to complete dexterous manipulation tasks by reporting success rates on a set of simulated objects.

RoboTok‑guided policies outperform all baselines on the original VTDexManip benchmark.

Table 4 shows RoboTok achieves the highest average success rates, beating the next best baseline by 7.45 % on seen objects.

On the modified harder VTDexManip tasks, RoboTok‑guided policies achieve the highest success rates, with up to 79.3 % on Lever Sliding.

Table 5 shows RoboTok outperforms the next best method (HAND) by 59.8 % on Lever Sliding and by large margins on the other tasks.

**Figure 6.** The modified harder VTDexManip simulation-based robotic evaluation tasks. Visualized are successful, natural hand poses achieved by PPO guided by RoboTok retrieved demonstrations.

**Figure.** Success rate (%) over training iterations for three tasks: (a) BottleCap Turning, (b) Faucet Screwing, and (c) Lever Sliding. The legend indicates the methods compared: Base (red), Random (teal), Flow (purple), HAND (orange), STRAB (green), and RoboTok (blue).

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers