Geometric Action Model for Robot Policy Learning
Jisang Han, Seonghu Jeon, Jaewoo Jung, René Zurbrügg, Honggyu An, Tifanny Portela, Marco Hutter, Marc Pollefeys, Seungryong Kim, Sunghwan Hong
Geometric Action Model (GAM) repurposes geometric foundation models as a unified substrate for robot policy learning.
How can we integrate 3D geometric reasoning into vision-language-action models to improve robot policy robustness and generalization?
Robot policies often struggle with spatial reasoning because they rely on 2D image representations that obscure 3D depth, scale, and occlusion. The Geometric Action Model (GAM) splits a pretrained geometric foundation model into an encoder and decoder, inserting a causal transformer between them to jointly predict future latent geometry and robot actions. This architecture achieves state-of-the-art robustness to camera perturbations while running up to 55× faster than diffusion-based baselines.
Paper Primer
GAM treats the geometric foundation model as a shared medium for perception and control. By autoregressively predicting future latent tokens conditioned on language and proprioception, the model forces the policy to learn the physical dynamics of the scene alongside the action sequence.
GAM significantly improves robustness to out-of-distribution camera perturbations.
Evaluation on the LIBERO-Plus benchmark, which introduces controlled environmental shifts. ↑9.7 percentage points over the best-performing foundation-model-scale baseline.
GAM provides a substantial reduction in inference latency compared to existing world-action models.
Comparison of feed-forward pass times on a single GH200 GPU. 6.9 ms latency (≈145 Hz), representing a 55× speedup over diffusion-based Cosmos-Policy.
Why use a geometric foundation model as the policy backbone instead of a standard vision-language model?
Standard vision-language models operate on 2D image priors, leaving 3D cues like depth and occlusion implicit. GAM uses a geometric foundation model to explicitly resolve 3D structure, which is essential for contact-rich manipulation tasks.
What is the primary trade-off in the GAM architecture?
GAM relies on a frozen text encoder for language reasoning, meaning its commonsense capabilities are bounded by the pretrained text model. It is designed for geometric and temporal world modeling rather than open-ended linguistic reasoning.
Researchers can now achieve high-performance, real-time robot control by repurposing existing geometric foundation models rather than training massive, 2D-centric vision-language-action models from scratch.
Introduction
We expose the lack of 3D awareness in VLAs and present GAM to jointly predict geometry and actions.
Vision‑language‑action models (VLAs) and video world‑action models (WAMs) inherit strong semantic priors from large‑scale foundations, yet they remain fundamentally 2‑D: depth, scale, and occlusion are implicit, causing policies to break when camera viewpoints shift.
The Geometric Action Model (GAM) addresses this gap by repurposing a pretrained geometric foundation model as a shared substrate that simultaneously predicts future 3‑D geometry and robot actions, achieved by splitting the backbone at an intermediate layer and inserting a causal future predictor.
VLAs are robot policies that map visual and language tokens directly to action commands, treating perception and language as a joint 2‑D embedding space.
**Figure 1.** **GAM repurposes geometric foundation models into fast and robust robot policies.** (a) GAM jointly predicts future 3D geometry and action chunks within a shared geometric backbone. (b) By leveraging explicit 3D geometric priors, GAM improves robustness and real-world performance while reducing latency and model size compared to existing baselines.
The shift from 2‑D pixel‑based action prediction to 3‑D geometry‑aware prediction underpins GAM’s robustness and speed.
The Geometric Action Model
GAM predicts future geometry and actions jointly by inserting a causal predictor into a pretrained GFM.
Standard vision‑language‑action pipelines treat images as flat feature maps, so a change in camera viewpoint can break the policy. The core failure is the lack of an explicit 3D spatial representation that persists across time. GAM fixes this by weaving geometry prediction directly into the action stream.
GAM inserts a temporal predictor at a split layer of a pretrained geometric foundation model, letting the network forecast latent geometry and the next action token together.
How does GAM differ from a vanilla vision‑language‑action model?
A vanilla VLA maps raw images directly to actions, ignoring explicit 3‑D structure. GAM first extracts a geometric latent, predicts its future, and only then decodes both depth and action, guaranteeing spatial consistency across time.
Encode each multi‑view observation with the shallow GFM encoder $E_{\le L_s}$ to obtain latent geometry tokens $Z(L_s)_{t'}$.
Feed $Z(L_s)_{t'}$ together with proprioceptive tokens $p_{t'}$, action‑history tokens $q_{t'}$, and language tokens $L_\ell$ into the block‑causal predictor $g_\phi$.
Read out the predicted future geometry $\tilde Z(L_s)_{t'+1}$ and the next‑action token $\tilde a_{t'}$ from $g_\phi$’s final hidden states.
Propagate $\tilde Z(L_s)_{t'+1}$ and replicated action tokens $\tilde a_{v,t'}$ through the deep decoder $D_{>L_s}$.
Decode depth maps with the original GFM depth head and regress the executable action chunk with a lightweight action head.
The model learns to anticipate the next latent geometry token, effectively “seeing” the scene a few steps ahead.
Why not predict raw depth maps directly instead of latent tokens?
Raw depth maps are high‑dimensional and noisy; predicting them forces the model to learn a costly pixel‑wise regression. Latent tokens are compact, already encode 3‑D structure, and let the pretrained decoder handle the expensive rendering step.
The mask enforces a strict temporal ordering: tokens may attend only to themselves and earlier tokens, never to future ones.
Mask row for $g_1$ allows attention to $c_1$, $p_{1,1}$, $p_{1,2}$, $c_2$, $p_{2,1}$, $p_{2,2}$ (all past tokens) but blocks $a$ and $g_2$.
Mask row for $a$ allows attention to everything up to $g_1$ (including $g_1$) but blocks $g_2$.
Mask row for $g_2$ allows attention to all previous tokens including $a$, ensuring it can use the newly predicted action.
The block‑causal pattern guarantees that each future token is computed only from information that would be available at its time step, preserving a physically plausible prediction order.
**Figure 1.** (a) Geometric Action Model Pipeline (b) Block-causal Attention Mask
Training combines an $\ell_1$ action loss, a latent‑feature regression loss, and a depth supervision loss (eq 8). At inference the model caches key‑value pairs so each new step only processes the latest observation and the previous action.
Causal future predictor forward pass (simplified)
Related Work
We position our approach among prior vision‑language‑action and geometric foundation models.
Vision‑Language‑Action Models (VLAs) repurpose pretrained vision‑language models as robot policies by autoregressively decoding discrete action tokens. Recent variants add parallel decoding, diffusion‑based heads, or frequency‑space tokenizations, but they all rely on 2‑D image priors and therefore lack explicit 3‑D spatial awareness.
World Action Models for manipulation predict future states to learn a policy. One line fine‑tunes large video‑generation models to jointly predict future frames and actions, while another freezes the visual backbone and trains a separate temporal predictor for model‑predictive control.
Geometric Foundation Models (GFMs) infer dense 3‑D structure from multi‑view RGB inputs and have become perceptual substrates for robot policies. Early integrations treat GFMs as frozen feature extractors, whereas recent work repurposes them for predictive control, jointly predicting actions and 3‑D properties.
GAM departs from these paradigms in two ways: (1) it jointly models action and future‑scene predictions within a single autoregressive token sequence, and (2) it reuses the deep blocks of a GFM to decode predicted future tokens rather than only processing observed frames.
**Figure 2.** (a) Video WAMs [3, 8, 23] operate in 2D pixel space, predicting future latents and actions via video diffusion. (b) Geometry-aware VLAs [17, 18] predict actions using a VLA with passive feature distillation from an external GFM. (c) GAM (ours) unifies perception, geometry prediction, and action decoding by inserting a geometric world model inside a single GFM.
In the next section we formalize the tokenization and attention mechanisms of GFMs, establishing the foundation on which GAM builds its joint prediction head.
Experiments and Results
GAM delivers a 9.7 %p boost on camera‑perturbed LIBERO‑Plus and excels in real‑world tests.
Standard vision‑language‑action models lack explicit 3‑D spatial awareness, causing failures when camera viewpoints shift; the Geometric Action Model (GAM) restores robustness by jointly predicting future geometry and actions.
GAM outperforms all baselines on the camera‑perturbed LIBERO‑Plus benchmark, improving success rate by +9.7 %p.
Table 1 shows GAM achieving the highest success rate on LIBERO‑Plus, surpassing the strongest geometry‑aware VLA baseline by +9.7 percentage points.
**Figure 4.** Real-world robot tasks and results. Each task is evaluated under both in-domain (Light bar) and out-of-domain (Dark bar) settings. The illustration of each task is shown on the right.
**Figure 5.** Success rate vs. camera perturbation difficulty.
LIBERO is a lifelong single‑arm manipulation suite that systematically varies object identities, spatial layouts, and task goals to probe a policy’s ability to generalize across unseen configurations.
GAM consistently outperforms 2D‑pixel baselines across LIBERO tasks.
Ablation Studies
We quantify how split‑layer placement and model design affect performance and latency.
This section isolates the contribution of the split‑layer position and reports the resulting inference speed and model footprint.
**Table 3.** Layer ablation.
Shifting the split layer away from $L_s=12$ collapses performance.
At $L_s=0$ the success rate falls to ≈0 % versus 97 % at $L_s=12$; similar drops occur for $L_s=27,33,39$.
This table lists various hyperparameters and their corresponding values used in the model configuration.
RoboCasa Performance
GAM sets a new record on RoboCasa Kitchen, achieving the highest average success rate.
GAM achieves the highest average success rate on RoboCasa Kitchen, surpassing the previous best by 2.3 %.
Table 10 shows GAM at 69.4 % versus the next‑best Cosmos Policy at 67.1 %.
A simulated kitchen suite of 24 manipulation tasks, each scored as success or failure, used to evaluate robot policies.
**Table 10.** Average success rates on RoboCasa Kitchen.
**Table 11.** Task-wise success rates on RoboCasa-Kitchen. We report per-task success rates and the overall average.
Extended LIBERO Results
GAM sets new LIBERO‑Plus robustness records across all perturbations.
GAM outperforms all prior policies on every LIBERO‑Plus perturbation category.
Table 12 shows GAM (Ours) achieving the highest success rates across Spatial, Object, Goal, and Long suites; Figure 9 confirms the lead persists as difficulty rises.
**Figure 9.** Detailed zero-shot robustness on LIBERO-Plus. We report success rates across difficulty levels L1–L5 for each perturbation category in the LIBERO-PLUS benchmark. The Average panel summarizes performance across all perturbation categories.
Table 13 confirms that on the original LIBERO benchmark GAM reaches near‑perfect scores, with most tasks at 100 % and the hardest “Long” tasks still above 80 %.
Future Depth Map Analysis
GAM sets a new 99.6% success rate on future depth map prediction.
GAM attains the highest future depth‑map success rate, reaching 99.6%.
GAM (Ours) 99.6 % / 91.4 % versus the runner‑up RIPT‑VLA $\pi$0 at 98.8 % / 80.1 % and $\pi$0.5 at 98.2 % / 86.4 %.
Diagnostic Analyses
We probe how the model degrades under targeted visual and contextual perturbations.
We evaluate the Geometric Action Model (GAM) on a suite of controlled perturbations that isolate visual and contextual factors. Across camera, robot pose, language instruction, lighting, background noise, and layout changes, GAM remains above 90 % on all but layout, where performance dips noticeably. These diagnostics expose the primary failure mode that motivates the next design iteration.
**Figure 10.** Future depth visualizations predicted by our model on representative LIBERO tasks.
Implementation Details
Implementation details cover the backbone, causal predictor insertion, and evaluation table overview.
We adopt DA3‑Giant [refs] fine‑tuned on Track4World [refs] as the model backbone, providing a strong pretrained visual‑language foundation for downstream robotic tasks.
A 12‑layer causal predictor of width $d_g = 1024$ is inserted at layer $L_s = 12$, the point where alternating attention starts, enabling the model to forecast future states while preserving temporal causality.
Table 1 reports success rates (in %) for each method on LIBERO and LIBERO‑Plus, highlighting absolute performance drops in parentheses and color‑coding the top three performers per column.
Appendix: Experimental Details
Appendix A details the training data, implementation, and additional experimental analyses.
Section A enumerates the experimental settings, implementation details, simulation and real‑world evaluation protocols, baseline configurations, and inference benchmark methodology.
GAM is trained in two stages: the first stage jointly learns the predictor, action head, and the GFM backbone on a large single‑arm data mixture using 64 NVIDIA GH200 GPUs, batch size 1024, for roughly 96 hours; the second stage fine‑tunes the full model on each benchmark’s official training set with 16 GH200 GPUs, batch size 160, for about 48 hours.
Pre‑training mixes three sources—Open‑X Embodiment (72 %), MimicGen (18 %), and RoboCasa365 (10 %)—and supplies depth supervision via teacher pseudo‑depth for OXE and re‑rendered simulator depth for the other two.
Only OXE actions that map to the common control interface are kept, and RoboCasa365 is limited to its manipulation‑task subset; the language encoder remains frozen throughout pre‑training.
All observations are normalized to 224 × 224 pixels, and whenever possible two RGB views (an external camera and a wrist‑mounted camera) are provided; standard augmentations such as random cropping, rotation, and color jitter are applied during training but disabled at test time.
Simulation experiments adopt the LIBERO evaluation protocol across the four standard suites—Spatial, Object, Goal, and Long—each containing ten tasks; demonstrations are filtered to discard unsuccessful episodes and idle frames before fine‑tuning a separate policy per suite.
Success rate (SR, %) is the primary metric; the original LIBERO benchmark evaluates each task with 50 randomized trials (500 rollouts per suite), while LIBERO‑Plus uses a single rollout per perturbed task instance.
Training proceeds with a global batch size of 160 for up to 110 k steps until convergence; baseline policies ($\pi$0.5, Cosmos‑Policy, and their variants) are re‑evaluated using publicly released checkpoints under the same protocol.
Real‑world experiments employ a wrist‑mounted ZED camera and an external RealSense camera; four tasks—pick‑and‑place, stack milk & cube, place pot & pan, and insert cube into covered pot—are trained on 284, 202, 184, and 169 teleoperated demonstrations respectively.
Each task is tested over 20 trials, split evenly between in‑distribution (ID) and out‑of‑distribution (OOD) conditions; OOD trials apply an 85 cm translation and a 45° rotation to the external camera.
Baseline training follows the original recipes and hyperparameters, altering only the task data and camera streams to match the GAM setup; $\pi$0.5 uses flow‑matching action decoding with ten integration steps, while Spatial Forcing employs its feature‑alignment loss.
Inference latency (model‑only) measured on a single GH200 GPU shows GAM at 17.5 ms, $\pi$0.5 at 29.2 ms, OpenVLA‑OFT at 70.1 ms, and Cosmos‑Policy at 382.4 ms; enabling CUDA Graphs for GAM reduces latency to 6.9 ms, yielding ≈145 Hz control and a 55.4× speedup over Cosmos.
The DA3‑based GAM architecture contains 1.4 B parameters, of which 983.2 M are trainable; most trainable weights reside in the later ViT‑Giant blocks, while the early geometric layers and the DPT depth head stay frozen to preserve pretrained geometry.
**Figure 6.** Training Dataset Mixture. We illustrate the dataset mixture utilized during pretraining, detailing the relative proportions of each constituent dataset. The pie chart shows the high-level source mixture, and the bar chart shows the percentage of each constituent dataset relative to the entire training corpus.
**Figure 7.** Real-world experiments environment setup and ID vs. OOD Camera setup.
Table 5 summarizes model‑only latency across policies and optimization flags; with CUDA Graphs GAM achieves the lowest latency of 6.9 ms, while other configurations range from 29.2 ms ($\pi$0.5) to 382.4 ms (Cosmos‑Policy).
Table 7 reports success rates for 24 robotic tasks, with individual task SRs spanning 11.3 % to 100 % and an overall success rate of 69.4 % for the evaluated policies.
Additional Results and Data
Additional ablations and visual diagnostics clarify GAM’s robustness.
This section reports the ablation studies and qualitative diagnostics that support the robustness claims for the Geometric Action Model (GAM).
Section C.1 evaluates a direct‑action supervision variant that skips the deep geometric decoder; Table 14 shows that feeding the action token through the remaining GFM layers yields consistent gains, especially on the LIBERO‑Plus suite.
Section C.2 inspects attention maps of action tokens across GFM layers (Figure 11). Intermediate layers attend to task‑relevant objects and contact regions, while deeper layers refine the focus for precise action decoding.
**Figure 11.** Attention visualizations of action tokens.
Overall, GAM retains strong performance on the original LIBERO tasks while markedly improving robustness on LIBERO‑Plus, particularly under camera‑viewpoint perturbations that demand stable geometric reasoning.
Figure 10 (B.3) visualizes future depth predictions generated by GAM for each task suite, illustrating that the model can forecast accurate geometry from a single RGB observation.
Per‑task success rates on LIBERO Original are high across categories: Spatial 98.6 %, Object 99.0 %, Goal 97.4 %, Long 94.6 %.