Reflex: Visually Grounded Reactive Humanoid Control
Taoyang Jia, Weikai Huang, Linxin Song, Jared Darlington, Jieyu Zhao, Yue Wang, Jiafei Duan, Zhongzheng Ren, Ranjay Krishna
Reflex enables humanoids to catch thrown objects by decoupling whole-body control, object dynamics, and visual grounding.
How can a humanoid robot catch a thrown object using only onboard visual input without explicit object-pose estimation?
Humanoids struggle to catch fast-moving objects because they must simultaneously balance their entire body, predict trajectory under latency, and ground perception in egocentric vision. Reflex decomposes this into three stages: learning whole-body control in simulation, training a latent representation of object dynamics, and finally grounding that latent in visual history. This approach achieves 85.7% catch success in simulation and reaches 76% of human performance on a real Unitree G1 at 2.5 meters.
Paper Primer
The core challenge is that catching requires a tight, closed-loop coupling of perception and motor response: the robot must move into position and brace for impact before the object arrives, all while its own camera motion obscures the target. Reflex solves this by isolating the learning of motor skills from the learning of visual perception, using a compact latent representation as the bridge.
Reflex is a three-stage pipeline: first, a state-based policy learns whole-body catching using reinforcement learning; second, an encoder learns to compress delayed object observations into a latent dynamics representation; third, a visual encoder learns to predict this same latent from egocentric RGB-D history. The visual encoder is like a navigator: it reads the stream of visual landmarks and ego-motion to estimate the object's future path, feeding that estimate into a fixed controller that already knows how to move the body.
Visual grounding preserves nearly all the performance of privileged, state-based control.
In simulation, the visual controller achieves 85.7% catch success compared to 87.9% when provided with privileged, ground-truth box state.
Temporal visual history is the most critical component for successful interception.
Removing temporal history (using single-frame input) reduces catch success by 35.6 and 39.8 points on centered and laterally offset throws, respectively.
Why is this three-stage decomposition necessary instead of training end-to-end?
Jointly training visual perception and motor control from pixels is computationally expensive and difficult to optimize, as catch success provides only weak, delayed supervision. Decomposition allows each capability to receive a direct training signal and isolates expensive visual learning from the reinforcement learning process.
What is the scope of this system — does it work for any object?
The system is currently trained on boxes and is effectively size-blind; while it handles varying payloads well, performance drops significantly for boxes outside the reference training sizes, where the robot often intercepts but fails to retain the object.
Reflex demonstrates that complex, reactive whole-body tasks can be mastered by humanoids through modular learning, shifting the focus from monolithic end-to-end policies to structured, latent-based perception-control interfaces.
Reactive Humanoid Catching
We expose the gap in visual reactive whole‑body control and introduce Reflex, a framework that replaces explicit pose estimation with latent visual grounding.
Humans effortlessly react to fast‑moving objects using their whole body, such as catching a tossed cup or diving for a soccer ball. These actions require predicting the object's future trajectory, deciding a coordinated body response, and executing it within about a second. Replicating this visual reactive whole‑body control on humanoid robots remains difficult.
Current robotic approaches address only fragments of this problem. Manipulators on fixed bases can catch moving items but lack balance and impact absorption, while recent humanoid demos rely on motion‑capture markers or instrumented objects, avoiding the need for egocentric visual grounding. Consequently, the three core challenges—whole‑body coordination, fast dynamics inference, and egocentric perception—are rarely combined.
Reflex splits the catching problem into a perception function that turns raw RGB‑D input into a latent dynamics state, which a whole‑body controller then uses to react, eliminating the need for explicit pose estimation.
In simulation, Reflex attains 85.7 % catch success from RGB‑D alone, close to the 87.9 % achieved with privileged observations, and temporal visual history proves essential, as removing it drops performance by over 35 % points on centered and offset throws. On a real Unitree G1 robot, Reflex catches human‑thrown boxes with 65 % and 40 % success at 2.5 m and 5 m, reaching 76 % and 59 % of human catchers’ rates. Ablations further reveal that camera ego‑motion, regressing the learned latent rather than the full state, and action‑matching initialization each contribute measurable gains.
The shift from explicit pose estimation to latent visual grounding enables scalable visual learning for reactive whole‑body control.
Task Formulation
Defines the robot's visual and control interface and the criteria for successful catching.
The robot receives a continuous 192 × 144 Red Green Blue‑Depth (RGB‑D) image stream from a torso‑mounted camera, while the controller also gets the robot’s internal state needed for whole‑body stabilization, but no explicit box pose, velocity, or external‑camera measurement.
Because the camera moves with the robot, the box becomes increasingly occluded near contact, turning the problem into a closed‑loop coupling of visual estimation and whole‑body response; at each 20 ms control step the model emits a 29‑dimensional joint command (one per actuated DoF) that is executed as joint‑position targets.
Simulation runs in Isaac Lab with physics integrated at 200 Hz and the controller operating at 50 Hz, providing a high‑fidelity testbed for the catching task.
The catching task requires the 29‑DoF Unitree G1 to move into the incoming box’s path, catch it on its forearms against the torso, dissipate its momentum, and retain it without falling, all without choosing the contact moment.
We evaluate three outcomes: Interception (box slows below 1 m/s near the torso), Catch Success (the box is supported in the chest region with wrists on opposite sides and remains low‑speed for 0.6 s), and Post‑Catch Stability (torso tilt stays below 38° and no fall occurs).
For the box‑size sweep, where a fixed chest‑region test is not comparable, we use a geometry‑agnostic sustained‑hold criterion as described in Sec. V‑A.
The Reflex Learning Framework
We decompose catching into perception and control, then train each via a three‑stage privileged learning pipeline.
Jointly learning a visual encoder V$\psi$ and a whole‑body controller $\pi$$\theta$ from raw RGB‑D pixels is prohibitively costly: on‑policy rollouts must render high‑throughput depth images, while the policy must explore a 29‑DoF action space and a large visual manifold.
The policy is split into a vision module that predicts a compact object‑dynamics latent, and a controller that consumes that latent together with the robot’s own state to output joint commands.
Why not train a single end‑to‑end policy that maps RGB‑D directly to joint commands?
End‑to‑end learning would require the policy to simultaneously discover a useful visual representation and a stable whole‑body control law, which dramatically increases sample complexity and makes credit assignment across vision and motor modules ambiguous.
The controller receives rₜ = (joint angles, velocities) and the latent z.
$\pi$$\theta$ computes a normalized action aₜ = (0.05, ‑0.02, … , 0.03) for the 29 joints.
Scaling with s = (0.5, 0.5, …, 0.5) yields s ⊙ aₜ = (0.025, ‑0.01, …, 0.015).
Adding to the nominal pose q₀ = (0, 0, …, 0) gives joint targets qⁿᵒᵐₜ = (0.025, ‑0.01, …, 0.015).
The latent z captures object motion in a form the controller can use directly, avoiding the need for the controller to parse raw images.
During training we give the encoder access to “privileged” simulator data (full box state) so it can learn a compact latent that the controller later consumes, even though the encoder will only see noisy RGB‑D at test time.
How does “privileged” learning differ from standard supervised learning on RGB‑D?
Privileged learning uses perfect simulator state (full box pose, velocity, etc.) as input to the encoder during training, allowing it to learn an ideal latent; at test time the encoder only receives noisy visual cues, but the latent it predicts has been shaped to be useful for control.
The linear projection reduces the 26‑dim vector to 16 dim.
The residual MLP adds a non‑linear transformation, producing an intermediate 16‑dim vector.
Applying tanh squeezes each component into [‑1, 1], yielding the latent zₜ = (0.12, ‑0.08, 0.05, 0.10, ‑0.02, 0.00, 0.03, ‑0.01).
Even with a single delayed observation, the encoder can generate a latent that captures enough motion information for the controller to act.
After the privileged encoder is fixed, a visual network learns to predict the same latent from egocentric RGB‑D histories, so the controller can operate without any privileged state.
Why not train the visual encoder directly on the control objective instead of regressing the privileged latent?
Regressing the privileged latent provides a dense, well‑behaved supervision signal that is easier to learn from limited visual data; directly optimizing control would require sparse reward signals and would entangle perception errors with control performance.
Geometric layer converts the heatmap peak to a camera‑frame point (x=0.30 m, y=0.45 m, z=0.50 m).
The GRU receives the point, depth minima, confidence, and ego‑motion, updating its hidden state.
After processing both frames, the GRU outputs a latent \hat{z} = (0.11, ‑0.07, 0.04, 0.09, ‑0.03, 0.01, 0.02, ‑0.01).
Even with only two noisy frames, the GRU can infer a latent that closely matches the privileged encoder’s output, thanks to the temporal aggregation of geometric cues.
Stage 1: Train a state‑based whole‑body controller $\pi_{s}$tate with PPO on perfect box state (170‑dim observation) while using an asymmetric critic that sees uncorrupted body poses.
Stage 2: Freeze the teacher policy, train a privileged encoder E$\phi$ to map delayed, partially observed box data to an 8‑dim latent, and initialize the decoder via behavior‑cloning and DAgger‑style action matching.
Stage 3: Freeze the controller and privileged encoder, then train a visual encoder V$\psi$ (CNN + GRU) to regress the latent from RGB‑D histories, using auxiliary position/velocity supervision and a second DAgger‑style data aggregation.
**Fig. 2. Training pipeline and policy architecture.** Stage 1 learns whole-body control by training the state-policy teacher with PPO. Stage 2 learns the object dynamics: it initializes the privileged encoder and decoder latent-input weights by matching teacher actions, then jointly refines the encoder and decoder with PPO. Human motion examples supply reference tracking and an AMP style reward. Stage 3 learns visual grounding by predicting the latent from RGB-D histories and camera ego-motion, using box-state supervision and DAgger rollouts. The pretrained CNN and heatmap head stay fixed during GRU training; the privileged encoder and decoder remain fixed throughout Stage 3. The decoder also receives robot/reference observations. Because Stage 3 learns only the latent predictor, the visual controller reuses the PPO-trained decoder without modification.
After visual grounding, a downstream module handles carrying and handover: a supervisor monitors box stability and robot posture, then transfers authority to a walking controller (SONIC) for transport and handover.
Experimental Evaluation
Reflex achieves high catch rates in simulation and real‑world tests, approaching privileged input performance.
Reflex with visual latent input catches 85.7 % of throws in simulation, within 2.2 pp of the privileged latent input baseline.
Evaluated on 3840 throws across six controllers; privileged latent input achieves 87.9 % catch rate.
**Fig. 3. Throw and appearance variations.** Simulation examples show changes in release distance, box size, color, and landing location. The box-size panel illustrates measured real boxes; simulated weight and distance tests use the 34x26x22 cm training box. Color augmentation is applied when training the visual component. These variations define the operating conditions evaluated in Tab. I.
**Fig. 4. Catching across lateral landing offsets.** Catch rates for the ground-truth state-policy reference, the YOLO detector with the state policy, our CNN detector with the latent decoder, privileged latent input, and visual latent input. Shading indicates the training-offset region and its tail. Each point pools 640 throws for one policy checkpoint (320 throws for the YOLO curve). Visual latent input exceeds the ground-truth state policy by about 12 points at 0.6 and 0.9 m, whereas YOLO detection lowers the state policy's catch rate at every reported offset.
**Table III.** **Reward ablation.** Catch rates (%, mean ± s.d. over three training seeds) after PPO. Removing all three task groups drops success to 33.7% and 2.7%. *Foot-height termination; no episode reaches the throw. ‡Stage 2 recipe trained by PPO from random initialization, one seed, 1000 iterations.
**Table V.** **Visual-encoder ablation.** Catch rates (%, mean ± s.d. over five policies). Removing temporal history costs the most.
**Table IV.** Added perception delay: Tab. IV delays the visual encoder input in simulation to test the Stage 2 latency randomization. With visual input, catch success is 88.4%, 87.2%, and 83.0% with 0, 20, and 40 ms of added delay and 72.7% at 60 ms, versus 65.8% at 60 ms for the controller.
**Table VI.** **Color augmentation.** Catch rates (%), decoder fixed. Augmentation raises blue and black boxes to 88.0 and 88.8%.
Ablation Studies
We ablate key components to quantify their impact on catching performance.
**Table.** Ablation study results showing catch rates (%) for different training objectives and reward components at lateral offsets of 0.0m and 0.6m.
With the full training objective the controller catches 88.6 % of objects at zero lateral offset.
Table III, “Full objective” row, zero‑offset column.
Removing all three task‑group rewards reduces catch rate to 33.7 % at zero offset.
Table III, “w/o all three task groups” row, zero‑offset column.
Eliminating temporal history from the visual encoder cuts performance by 35.6 percentage points.
Table V, comparison between full visual encoder (87.9 %) and “w/o temporal history” (49.6 %) at zero offset.
A latency‑randomized controller achieves 91.9 % catch at 60 ms delay, outperforming the non‑randomized version (84.5 %).
Latency experiments described in the “Object Dynamics” paragraph; 60 ms column shows 91.9 % vs. 84.5 %.
Without latency randomization, performance drops to 84.5 % at zero delay and degrades to 27.0 % at 120 ms.
Same latency experiment; zero‑delay and 120 ms columns for the non‑randomized controller.
Real-Robot Performance and Limits
Reflex maps raw RGB‑D to a latent state, letting a whole‑body controller catch thrown objects without pose estimation.
We evaluate the learned controller on a real Unitree G1 robot, testing catches of human‑thrown boxes while varying distance, size, payload, and lateral offset.
Visual latent input exceeds the privileged‑state policy by roughly 12 percentage points at 0.6 m and 0.9 m lateral offsets.
Table II shows 88.3 % vs 73.8 % (0.6 m) and 40.9 % vs 28.9 % (0.9 m).
Using the YOLO detector reduces the state‑policy catch rate at every offset, dropping up to 20 percentage points.
Table II reports state‑policy with YOLO at 67.2 % (0 m), 33.8 % (0.6 m), and 16.9 % (0.9 m) versus the ground‑truth policy.
Adding visual input preserves higher success than a controller without visual input even with a 60 ms perception delay (+6.9 pp).
Perception‑delay experiment: visual input 72.7 % vs. no‑visual 65.8 % at 60 ms.
Color augmentation boosts success on blue and black boxes from ~60 % to ~88 % (+28 pp).
Color‑augmentation results: blue 60.2 % → 88.0 %; black 61.2 % → 88.8 %.
Action‑matching latent initialization outperforms weight‑mapping by 2.3 percentage points.
Table 6: Action‑matching 82.8 % ± 3.2 vs. weight‑mapping 80.5 % ± 2.9.
At 5 m throws, privileged input outperforms visual input by 36.7 percentage points (79.7 % vs. 43.0 %).
Section V‑F reports visual 43.0 % and privileged 79.7 % at 5 m.
On the real robot, the controller catches 76 % of human performance at 2.5 m.
Table II: robot catch rate 76 % of human benchmark.
At 5 m, the robot achieves 59 % of human success.
Table II: robot catch rate 59 % of human benchmark.
Related Work
We situate our approach among prior catching, whole‑body control, and privileged‑learning methods.
Early catching systems used fixed‑base arms and external vision to predict trajectories, while later work added learning of object dynamics and impact‑aware bimanual motions. Most of these approaches still rely on motion‑capture or fixed cameras; only a handful employ onboard egocentric RGB‑D sensing, which we adopt for catching a thrown box.
Whole‑body control and loco‑manipulation have leveraged human motion data, motion imitation, and adversarial motion priors to generate realistic humanoid behaviors, extending to object‑centric tasks such as lifting and carrying. However, these works assume the object is static at contact, whereas our contribution is a dedicated catching controller that reacts to a moving box.
Privileged learning trains a teacher with simulation‑only information to supervise a student that only has onboard sensors, often combined with asymmetric critics and domain randomization. Following the RMA paradigm, we keep the action decoder fixed and regress a visual latent from RGB‑D history and ego‑motion to emulate the privileged latent.