DONIC Generalist Dexterous Bimanual Tracking Control from Human Demonstrations

Mingze Sun, Ziang Cao, Yuan Fei, Qiankuan Tao, Zihan Wang, Tianqi Liu, Junhao Chen, Haiwen Diao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu

DONIC is a generalist bimanual tracking policy trained on a new multi-view dataset to coordinate two robot hands for complex object manipulation.

How can we train a single, generalist policy to track complex, bimanual dexterous human motions across diverse objects without per-task fine-tuning?

Existing bimanual trackers struggle to coordinate two hands for sustained contact, often failing because they lack diverse training data or cannot handle the shifting roles of each hand during a task. DONIC addresses this by jointly training a single policy across multiple operation categories using an object-centered reward that supervises each hand's contact role independently, paired with a technique that removes reference-specific difficulty offsets from the learning signal. This approach allows one checkpoint to track unseen trajectories and held-out objects without fine-tuning, significantly outperforming state-of-the-art baselines in success rate and completion.

Paper Primer

The core mechanism hinges on two innovations: an object-centered per-hand reward and Clip-Centered Advantages (CCA). The reward function treats the object as the primary anchor, rewarding hands only when they are required to contact the object, while CCA subtracts the mean advantage of each specific reference clip from the training updates to ensure the policy learns from relative performance within a task rather than being biased by the inherent difficulty of the reference itself.

DONIC achieves high success rates on unseen bimanual tasks without task-specific fine-tuning.

Evaluation on held-out trajectories and objects from public datasets. 86.79% success on unseen trajectories and 57.32% on held-out objects, outperforming the strongest baseline by 77.36 and 45.73 percentage points respectively.

The authors also introduce BiHOI, a multi-view dataset of 1,575 sequences emphasizing sustained bimanual interaction. By incorporating this data, the model improves performance across all metrics, demonstrating that the policy benefits directly from the increased diversity of coordinated contact patterns.

Why is bimanual tracking significantly harder than single-hand manipulation?

Bimanual tasks require asymmetric coordination where hands often switch roles—one stabilizing while the other manipulates—and contact timing must be precise. Standard trackers often fail because they cannot balance these complementary roles or handle the increased complexity of maintaining two-hand contact simultaneously.

What is the primary advantage of the Clip-Centered Advantages (CCA) technique?

CCA removes reference-dependent advantage offsets that occur when training on a mix of easy and hard clips. By subtracting the mean advantage of a specific clip, the model preserves the relative differences between actions within that clip, preventing the policy from being penalized for "difficult" tasks where the baseline advantage might be consistently lower.

Introduction to Bimanual Tracking

We expose the gap in dexterous tracking: single‑hand focus leaves bimanual coordination unaddressed.

Recent dexterous trackers excel at single‑hand manipulation but ignore the coordinated, contact‑rich motions that real‑world tasks demand. The scarcity of bimanual datasets and the difficulty of synchronizing two hands further impede progress. To bridge this gap we propose a unified solution that learns from diverse bimanual demonstrations.

DONIC (Dexterous bimanual reference tracker) is a policy that simultaneously follows reference trajectories for both hands, learning to coordinate contacts and object motion without task‑specific fine‑tuning.

**Figure 1.** Overview of DONIC. BiHOI provides multi-view bimanual demonstrations that are reconstructed and retargeted to robot hands. DONIC learns a shared policy for diverse bimanual reference-tracking tasks.

The key shift is moving from single‑hand tracking to a unified bimanual dexterous tracker that generalizes across objects and tasks.

Contributions and Related Work

We outline BiHOI, our dataset, and position our work among prior bimanual tracking research.

We present our contributions and then situate them within three strands of prior work: (1) using human demonstrations as motion references, (2) generalist manipulation policies, and (3) existing bimanual hand‑object interaction datasets.

BiHOI is a multi‑view collection of coordinated bimanual hand‑object interactions, designed to provide robot‑ready reference motions.

Our contributions are threefold: (1) DONIC becomes the first generalist bimanual reference tracker trained across operation categories and human motion datasets, evaluated without fine‑tuning; (2) we release the BiHOI dataset and its robot‑ready reference pool; (3) we introduce an object‑centered per‑hand reward together with Clip‑Centered Advantages to enable joint PPO training across diverse bimanual references.

Prior work on human references to executable manipulation typically learns a separate policy per clip or augments retargeted motions with task‑specific residuals (e.g., ManipTrans, DexMachina, CHORD, ConTrack). In contrast, DONIC learns a single policy that tracks diverse references without per‑reference training. Generalist manipulation policies such as UniDexGrasp++, UniCross, and ArtManip have demonstrated cross‑object or cross‑skill transfer, but they remain limited to single‑hand scenarios. Existing bimanual hand‑object interaction datasets (GRAB, ARCTIC, OakInk2, TACO, GigaHands, H2O, HOI4D, HOT3D, HO‑Cap) provide rich motion recordings, yet they differ in annotation conventions and do not guarantee robot‑executable references. Our BiHOI dataset harmonizes these recordings, screens for zero‑residual feasibility, and forms a unified reference pool for joint training.

The BiHOI Dataset

We present BiHOI, a multi‑view bimanual hand‑object dataset with extensive contact data.

BiHOI is captured with a 32‑camera multi‑view rig that records at 30 fps, providing dense visual coverage for robust hand‑object reconstruction.

BiHOI achieves the highest bimanual contact rate (65.91 %) among the surveyed datasets, indicating substantially richer two‑hand interaction data for training.

Methodology Overview

DONIC learns a bimanual tracker by aligning reconstructed human motion with robot control using object‑centered rewards and Clip‑Centered Advantages.

Existing dexterous trackers assume a single hand and cannot cope with the asynchronous contacts that arise when two hands manipulate a shared object. This mismatch creates unstable learning signals and poor generalization to new objects.

Each hand receives a reward that measures how well it follows the object’s motion, so the hands can act independently while still keeping the object on track.

How does this object‑centered reward differ from a traditional hand‑centric tracking loss?

Hand‑centric losses only compare the hand’s pose to a reference pose, so they punish a hand whenever the object moves independently. The object‑centered reward instead compares each hand to the object’s trajectory, allowing one hand to move the object while the other hand remains idle or prepares for the next contact.

After computing generalized advantages, CCA subtracts the average advantage of the whole demonstration clip, preserving only the relative advantage of each timestep.

Why can’t standard GAE be used directly for imitation learning from clips?

Standard GAE retains the absolute magnitude of the advantage, which includes a large offset caused by the fixed reference trajectory. This offset skews the policy gradient toward reproducing the exact reference rather than learning the relative timing cues that actually drive successful bimanual coordination. CCA removes that offset, letting the policy focus on the useful temporal structure.

Obtain synchronized RGB frames and depth maps from multiple calibrated cameras.

Use SAM3 masks and Depth Anything 3 to generate per‑view depth aligned to metric scale.

Reconstruct a canonical object mesh from the first‑frame views and masks; back‑project depth to recover physical dimensions.

Apply FoundationPose to estimate the object’s 6‑DoF trajectory via multi‑view pose consensus.

Triangulate hand keypoints across views, fit a participant‑specific MANO hand model, and refine joint positions against metric depth.

Retarget the reconstructed hand motion to the Wuji robot hands by aligning wrist frames, fitting hand keypoints to robot links under joint limits, and smoothing before retargeting.

Pair the resulting robot wrist and finger references with the object trajectory to form the final training clip.

Triangulate each corner using the two views → 3‑D positions with millimetre accuracy.

Fit a unit‑cube mesh to the 3‑D corners → side length 0.10 m (scaled from the back‑projected depth).

Estimate the cube’s pose at frame 0 as identity; at frame 1 the cube translates 0.02 m along X and rotates 5° about Z.

Fit the MANO hand model to the hand keypoints → wrist position (0.12, 0.03, 0.08) m, finger joints within joint limits.

Retarget to the robot hand: align the robot wrist to the MANO wrist, then map each fingertip to the nearest robot link respecting joint limits.

This tiny example shows how multi‑view geometry yields a metric object mesh and hand pose, which can then be directly transferred to a robot hand without any additional scaling.

**Figure 2.** **Overview of DONIC.** Top: BiHOI captures are reconstructed and retargeted, then combined with public-data references to form the training pool. Middle: A shared bimanual policy learns through simulated interaction and verified demonstrations in an iterative PPO+BC loop. Bottom: An object-centered per-hand reward, Clip-Centered Advantages (CCA), and behavioral cloning provide complementary learning signals.

By combining metric‑scale reconstruction, object‑centered rewards, and Clip‑Centered Advantages, DONIC acquires a robust bimanual tracking policy that generalizes across objects and hand‑object interaction patterns.

Policy Architecture and Observation Space

Object-centered reward aligns object pose and motion while gating hand contact supervision.

Hand‑pose accuracy alone does not guarantee proper contact or coherent object motion; without a dedicated signal the policy can drift into asynchronous or stationary behaviors.

Position term: $e^{-κ_p}\|p_{obj}^t-\hat{p}_o^t\|^{2}=e^{-1}\times(0.1)^{2}=0.3679\times0.01=0.00368$.

Rotation term: $e^{-κ_R}\angle(R_o^t,\hat{R}_o^t)=e^{-0.5}\times0.087\approx0.6065\times0.087=0.0528$.

Velocity term: $e^{-κ_v}\|v_o^t-\hat{v}_o^t\|^{2}=e^{-2}\times(0.2)^{2}=0.1353\times0.04=0.00541$.

Overall reward: $r_t = 0.00368 \times 0.0528 \times 0.00541 \approx 1.05\times10^{-6}$ (a very low value, indicating poor alignment).

The multiplicative structure makes the reward extremely sensitive to any single large error, forcing the policy to reduce all three errors together.

How does this object‑centered reward differ from a conventional per‑hand pose error reward?

Standard pose rewards penalize each hand’s joint error independently, ignoring whether the hands move the object correctly. The object‑centered reward instead evaluates the global object state (position, orientation, velocity) and only activates hand‑contact penalties when the reference explicitly requires contact, ensuring coordinated manipulation rather than isolated hand accuracy.

Reward Function Design

Defines the reward that supervises hand‑object contact and motion.

Object‑tracking loss alone does not tell the policy how the hands should support the motion. We therefore add a contact‑and‑hold reward that explicitly supervises fingertip forces and proximity.

Instead of only rewarding the hand pose, we also reward each required fingertip for actually touching the object, encouraging the policy to keep the hands in contact while moving.

How does this contact reward differ from simply penalizing fingertip distance to the object?

Because it multiplies a distance‑based exponential term with a hard contact indicator, the reward still requires a non‑zero contact force. A pure distance penalty would ignore whether the hand is actually touching, allowing the hand to hover near the object without establishing a stable grasp.

Compute binary indicators $\alpha_{h,i,t}$: left hand fingertip 1 → $1$, fingertip 2 → $0$ (force below $f_{\min}$); right hand fingertip 1 → $1$, fingertip 2 → $1$.

Evaluate the exponential term $e^{-\kappa_c \delta_{h,i,t}}$: left 1 → $e^{-0.1}=0.90$, left 2 → $e^{-0.15}=0.86$; right 1 → $e^{-0.05}=0.95$, right 2 → $e^{-0.12}=0.89$.

Form the per‑finger contribution $\zeta e^{-\kappa_c \delta}+ (1-\zeta)\alpha$: left 1 → $0.5 \cdot 0.90+0.5 \cdot 1=0.95$, left 2 → $0.5 \cdot 0.86+0.5 \cdot 0=0.43$, right 1 → $0.5 \cdot 0.95+0.5 \cdot 1=0.98$, right 2 → $0.5 \cdot 0.89+0.5 \cdot 1=0.94$.

Average over the two required fingertips for each hand: $r_{\text{cont},L}^t = (0.95+0.43)/2 = 0.69$, $r_{\text{cont},R}^t = (0.98+0.94)/2 = 0.96$.

Aggregate across hands with weights $w_R=1$, $w_L=0.8$: $r_{\text{cont}}^t = (1 \cdot 0.96 + 0.8 \cdot 0.69)/(1+0.8) \approx 0.84$.

This toy calculation shows how the reward blends proximity (the exponential term) with a hard contact check; even a fingertip with low force can still contribute via proximity, but the overall reward favours hands that actually touch the object.

Training the Generalist Policy

Training combines PPO with behavior cloning and a clip‑centered advantage correction to learn a bimanual tracker.

The paper’s goal is to train a single policy that can track coordinated bimanual hand‑object motions; we achieve this by alternating on‑policy PPO updates with behavior‑cloning from verified demonstrations.

Compute the mean for Clip A: $(0.8+0.6+0.4)/3 = 0.6$.

Center each Clip A advantage: $0.8-0.6=0.2$, $0.6-0.6=0.0$, $0.4-0.6=-0.2$.

Compute the mean for Clip B: $(-0.2+0.0+0.2)/3 = 0.0$.

Center each Clip B advantage (no change): $-0.2$, $0.0$, $0.2$.

The final centered advantage vector is $[0.2, 0.0, -0.2, -0.2, 0.0, 0.2]$, preserving the relative ordering inside each clip while eliminating the per‑clip offset.

CCA ensures that a clip with uniformly low raw advantages does not unfairly penalize its transitions; only the relative quality within the clip drives the PPO update.

Pure PPO can struggle to discover rare but essential hand‑object contact actions; by injecting behavior‑cloning loss on verified demonstration pairs, the policy receives a direct signal toward those successful actions while still learning from on‑policy interaction.

Compute the squared error: $\|(0.3, -0.1) - (0.5, -0.2)\|^2 = (0.2)^2 + (0.1)^2 = 0.05$.

Scale by $\lambda$: $0.1 \times 0.05 = 0.005$.

Add this BC contribution to the PPO loss for the current update (e.g., if $L_{\text{PPO}} = 0.12$, the joint loss becomes $0.125$).

The BC term provides a small but consistent pull toward the demonstrated action without overwhelming the on‑policy gradient signal.

Experimental Setup

DONIC is trained on 3,420 clips, evaluated on two held‑out splits with bimanual metrics.

DONIC is trained on a combined pool of 3,420 clips, merging public data and BiHOI sequences.

1,845 public‑data clips plus 1,575 BiHOI sequences.

We curate bimanual manipulation clips from GRAB, GigaHands, OakInk2, TACO, and H2O, retaining only those where both hands and the object are tracked and each hand contacts the object at some point. The resulting public‑data set provides 1,845 clips, and together with the 1,575 BiHOI sequences we form the 3,420‑clip training pool; evaluation uses two held‑out splits, UNSEEN‑TRAJ with 53 trajectories and UNSEEN‑OBJ with 164 clips, none of which appear in training. Metrics follow the AdaDexTrack protocol, reporting object translation $T_o$ and rotation $R_o$, wrist translation $T_h$ and rotation $R_h$, and finger‑joint error $E_{finger}$; success requires $(T_o+T_h)\le10\text{ cm}$, $(R_o+R_h)\le\pi/3$, and $E_{finger}\le2\pi/9$, while SR and Comp measure per‑hand and combined frame‑wise compliance.

We compare against DexTrack‑bi and Dexplore‑bi, adapting their code to the dual‑hand Wuji setup while sharing the same public‑data pool, Isaac Gym simulator, robot model, and held‑out splits. DexTrack‑bi follows the specialist‑distillation pipeline, whereas Dexplore‑bi retains its original reward and reference‑scoped exploration strategy.

Baselines and Limitations

DONIC extends single‑hand tracking to coordinated bimanual motions using the BiHOI dataset.

The central premise—single‑hand trackers struggle with coordinated hand‑object motions—was addressed by training a generalist bimanual policy (DONIC) on the large‑scale BiHOI dataset.

DexTrack‑bi reuses DexTrack’s specialist‑distillation pipeline, now feeding two Wuji hands instead of one.

Dexplore‑bi runs the published Dexplore code with a two‑hand interface, preserving its original reward and exploration strategy.

All baselines share the Isaac Gym simulator, two Wuji hands, the public training clips, and the held‑out split identities. Tracking errors are computed with the bimanual metrics from Section D.2 on 164 unseen‑object and 53 unseen‑trajectory clips.

**Figure S2.** Qualitative bimanual tracking comparisons. Rows in each group show the reference, Dexplore-bi, DexTrack-bi, and DONIC (Ours); time runs left to right.

Main Results

DONIC extends tracking to coordinated bimanual motions trained on BiHOI.

DONIC extends tracking from single‑hand manipulation to coordinated bimanual motions, leveraging the large‑scale BiHOI dataset.

DONIC raises the success‑rate by up to 9.4 % over the DexTrack‑bi baseline on the BiHOI test set.

SR↑ 0.00 9.43 (baseline → DONIC)

**Figure 3.** Qualitative comparisons on two held-out object trajectories. Rows show the reference, Dexplore-bi, DexTrack-bi, and DONIC; time runs left to right.

Appendix Overview

Provides a roadmap of the appendix’s sections and highlights key supplemental results.

This appendix expands the paper’s core contributions: it surveys related work, details the BiHOI dataset pipeline, specifies the DONIC controller and training regime, and provides exhaustive experimental protocols.

**Table 2.** Reference tracking without fine-tuning, using one checkpoint per method across both splits. Black and blue bold mark the best and second-best results within each split; ties share highlighting.

Ablation studies (Appendix D.4) reveal that removing the CCA signal or the contact‑ownership gate each reduces SR by roughly 5 % and increases tracking errors, underscoring the importance of both learning signals for joint hand‑object success.

Qualitative comparisons (Appendix D.5, Figure 3) illustrate that DONIC faithfully reproduces the reference object motion, whereas Dexplore‑bi and DexTrack‑bi either limit object movement or lose hand‑object alignment, highlighting the practical impact of the learned policy.

The final appendix section (Appendix D.6) discusses current limitations: evaluation is confined to simulated single‑object interactions, relies on reconstructed motion references, and future work will target longer‑horizon, multi‑object scenarios and real‑world transfer.

Dataset Construction Details

Describes the BiHOI capture setup, dataset composition, and contact statistics.

Reference‑based trackers close the gap between human demonstrations and executable robot actions by using closed‑loop control. Prior systems such as DexTrack, Dexplore, and AdaDexTrack generate trajectory‑specific demonstrations, expand exploration around references, or distill specialist policies, respectively. Our loop builds on this lineage with an object‑centered per‑hand reward and Clip‑Centered Advantages for shared bimanual tracking.

BiHOI was captured with a calibrated array of 32 RGB cameras (1920 × 1232 px) at 30 fps, recording a tabletop workspace. The dataset comprises 1,575 interaction sequences, 229 distinct objects, and 14.77 hours of unique interaction time, featuring symmetric manipulation, asymmetric coordination, and coordinated multi‑object tasks. Each clip enters the bimanual pool only if both hands and the reference object are tracked and both hands contact the object at some point.

We measure contact as either simultaneous (both hands touching the object) or sequential (only one hand). In BiHOI, simultaneous bimanual contact occupies 65.91 % of contact‑active frames, far exceeding the public subsets where the highest is H2O at 38.57 % (a 27.34‑pp gap). Table S1 details the training inventory and contact percentages for each source.

**Table S1.** Selected public training inventory and measured contact proportions.

Reconstruction and Annotation Pipeline

Describes how multi‑view data are turned into synchronized hand‑object trajectories.

In the public pool each clip contains both hands and the manipulated object; every hand must contact the object at some point, but continuous or simultaneous contact is not required.

MANO provides a parametric hand mesh where a low‑dimensional pose vector drives the full 3‑D joint positions, enabling precise hand‑object geometry without explicit modeling of each bone.

Extract consistent masks with SAM3 and compute metric depth maps using Depth Anything 3, aligning depth to calibrated camera extrinsics.

Back‑project masked depth into a common world frame, fuse point clouds, and estimate isotropic object scale $s_{\text{obj}}$ from principal‑axis extents.

Track the object across views by triangulating mask rectangles, refine poses with Foundation‑Pose, and enforce multi‑view compatibility (rotation $<30^{\circ}$, translation $<5$ cm).

Calibrate each participant’s hand by fitting a flat‑hand MANO shape $\beta_0$ to metric‑triangulated bone lengths, yielding hand scale $s_{\text{hand}}$.

Reconstruct hand motion by triangulating WiLoR keypoints, filtering with a constant‑velocity state, and fitting MANO pose while keeping shape fixed.

Jointly refine hand and object trajectories against fused metric depth using a combined loss $L$ that includes SDF alignment, regularization, and smoothness terms.

Contact activity is measured per frame by checking whether any fingertip slot (3, 6, 9, 12, 15) registers a non‑zero contact indicator $c_{h}^{m,t,i}$; the overall contact‑active fraction across all clips is 31.40 %.

**Figure S1 | Reconstruction and annotation pipeline.** Synchronized multi-view RGB observations are processed to obtain SAM3 object masks and metric depth maps. The object branch performs metric reconstruction followed by robust multi-view 6D tracking, while the hand branch performs participant-specific hand calibration and multi-view hand-pose reconstruction. The two branches are finally refined jointly using depth-guided hand-object optimization.

Method and Training Details

We detail the controller interface, reward formulation, training objectives, and simulation setup.

Before retargeting we clean up the trajectory; the low‑pass filter coefficient is set to 1, so no extra smoothing is performed.

The policy receives a reference clip and produces bounded corrections that are applied to a PD target for the next timestep.

The reward aggregates hand‑tracking accuracy, object motion fidelity, and contact‑maintenance terms, each weighted to encourage coordinated bimanual behavior.

Advantages are centered per reference clip to remove shared bias, and PPO updates are augmented with a behavioral‑cloning loss from verified demonstrations.

Verified rollouts are stored as demonstrations; during training they are sampled separately from PPO updates to provide supervised guidance.

Training runs in Isaac Gym with thousands of parallel environments, using a fixed MLP architecture and a non‑learned Gaussian policy scale.

**Table S2.** Reward and generalist-training settings. Distances in reward exponents use metres, rotations radians, and velocities metres per second.

Experimental Protocols

Describes evaluation splits, simulation settings, and bimanual tracking metrics.

The evaluation uses two held‑out splits: UNSEEN‑TRAJ with 53 new object trajectories and UNSEEN‑OBJ with 164 clips of unseen objects; neither split provides any training transitions or demonstrations.

**Table S3 | Optimization and simulation configuration.**

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers