Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots

Ling Xu, Chuyu Han, Borui Li, Hao Wu, Shiqi Jiang, Ting Cao, Chuanyou Li, Sheng Zhong, Shuai Wang

A portable C++ inference runtime for embodied AI that decouples model execution from robot-side control logic.

How can we unify the fragmented, Python-heavy deployment stacks for diverse embodied AI models (VLAs and WAMs) into a single, portable C++ runtime?

Embodied AI models are currently trapped in fragmented Python stacks, forcing developers to write custom glue code for every new robot, simulator, or edge device. Embodied.cpp provides a unified C++ runtime that organizes model execution into five modular layers, allowing developers to swap model-specific heads and adapters while keeping the core inference engine stable. This approach enables closed-loop control on heterogeneous hardware, achieving 100% task success on tested vision-language-action models while significantly reducing memory footprint for world-action model components.

Paper Primer

Embodied deployment requires a different runtime contract than standard LLM serving: it must handle multi-rate execution, batch-1 latency sensitivity, and diverse sensor inputs. Embodied.cpp addresses this by treating the model as a set of pluggable modules—input adapters, sequence builders, backbone execution, head plugins, and deployment adapters—that share a single, portable C++ backend.

Maintains high task success rates for VLA models in closed-loop deployment.

Evaluated HY-VLA and pi0.5 models on RoboTwin tasks.

Significantly reduces memory usage for WAM transformer blocks via quantization.

Compared PyTorch BF16 baseline against C++ `Q4_K` quantized implementation on LingBot-VA. 312.2 MiB reduced to 88.1 MiB (approx. 72% reduction).

Why can't existing LLM runtimes like vLLM or llama.cpp simply be used for embodied models?

Existing runtimes are optimized for request-response throughput and uniform token interfaces. Embodied models require closed-loop control, multi-rate execution of different modules (e.g., perception vs. action), and the ability to handle heterogeneous inputs like tactile or force signals, which standard serving runtimes do not support.

What is the primary architectural move that allows this runtime to support both VLA and WAM models?

The runtime separates the shared execution path—the backbone—from model-specific components like heads and predictive branches. By organizing these into a five-layer architecture, the system keeps the deployment boundary stable while allowing the specific embodied logic to be swapped as a plugin.

Embodied.cpp shifts the burden of embodied deployment from custom, model-specific glue code to a reusable, portable C++ infrastructure, enabling consistent performance across diverse edge hardware.

The Fragmentation of Embodied Deployment

We expose the fragmented deployment problem and introduce Embodied.cpp, a unified C++ runtime for embodied AI.

Embodied AI models now include vision‑language‑action (VLA) systems and world‑action (WAM) predictors, yet their deployment is scattered across model‑specific Python stacks, backend assumptions, and robot‑side glue code.

Deploying embodied models forces engineers to stitch together custom Python wrappers, hardware‑specific backends, and ad‑hoc sensor interfaces for each new model family.

Existing inference runtimes target request‑response serving and assume uniform token I/O, so they fail to meet the three core requirements of embodied deployment.

The first requirement is **multi‑rate execution**: perception encoders, transformer backbones, predictive branches, and action heads must run at different frequencies within the same control loop.

The second requirement is **latency‑first closed‑loop control**: low, predictable latency and jitter for batch‑1 inference on heterogeneous edge hardware are essential, outweighing raw throughput.

The third requirement is **extensible embodied interfaces**: models need to consume arbitrary sensor streams and emit diverse outputs such as action primitives, future state predictions, or multimodal embeddings.

Embodied.cpp addresses these needs with a portable C++ runtime that abstracts a shared execution path into five layers: input adapters, sequence builders, backbone execution, head plugins, and deployment adapters.

By decoupling components, the runtime supports modular multi‑rate execution, latency‑first fused inference, and plug‑in operators, enabling deployment on Jetson, RK boards, x86 edge boxes, and workstation‑class robots through a single backend abstraction.

We evaluate Embodied.cpp on two VLA models (HY‑VLA, pi0.5) achieving 100 % and 91 % task success, and on a WAM benchmark that reduces block memory from 312.2 MiB to 88.1 MiB.

The shift from model‑specific glue code to a unified runtime eliminates integration bottlenecks and makes embodied AI deployment portable and efficient.

Taxonomy of Embodied Models

Survey of embodied model families and existing inference runtimes.

This section maps the landscape of embodied models and the runtimes that serve them, highlighting where existing solutions fall short of the unified C++ target.

VLA models ingest raw visual frames and language commands, then directly emit robot actions.

WAM models first predict future world states, then derive actions from those predictions.

These are the de‑facto reference implementations that run embodied models inside a Python process, typically using PyTorch or TensorFlow.

**Figure 1.** Architectural taxonomy of embodied models. We first separate VLA models (a-d) from WAMs (e-h), and then organize each family by internal inference structure.

Autoregressive token generation where a single shared backbone produces a sequence of action tokens.

Uses a strong vision‑language model as backbone, adding dedicated continuous‑action heads.

Separates high‑level planning from low‑level control via a planner that emits sub‑goals.

Modules run at different rates and exchange buffered state, decoupling perception from actuation timing.

Explicit two‑stage pipeline: a world model predicts future states, then an action expert consumes them.

Collapses prediction and action into a single autoregressive token stream.

One backbone serves both world modeling and action generation, exposing separate downstream blocks.

Predicts a compact latent future or subgoal, which a downstream expert maps to actions.

The Embodied.cpp Runtime

Embodied.cpp unifies heterogeneous modules into a single, latency‑first C++ runtime.

The core difficulty is that embodied inference must juggle heterogeneous modules at real‑time rates, unlike the batch‑oriented pipelines of LLM services. This forces the runtime to decouple module schedules, prioritize latency, and stay extensible for new model families.

A single C++ runtime hosts any embodied model while exposing modular execution units, typed adapters, and a reusable kernel warehouse.

How does this differ from a typical LLM inference server?

LLM servers treat the whole model as a single graph executed on every request; Embodied.cpp splits the graph into independent modules that can be scheduled at different rates, avoiding unnecessary recomputation and guaranteeing deterministic control timing.

Step 1: Action runs (step 1), perception does not run, prediction does not run.

Step 2: Action runs (step 2), perception runs (its 2‑step period), prediction does not run.

Step 3: Action runs (step 3), prediction runs (its 3‑step period), perception does not run.

This schedule saves compute by skipping expensive perception updates while preserving tight control through the always‑on action module.

Input adapters convert raw sensor streams and dataset entries into a unified tensor representation.

The scheduler dispatches modules according to their configured refresh policies (e.g., perception every 2 steps, prediction every 3 steps).

The shared execution zone runs the selected modules, reusing intermediate feature pools when possible.

Operator fusion and the kernel warehouse provide hardware‑specific implementations for each module.

Output adapters translate the final tensors into robot actions or simulator commands.

Multi‑rate scheduling loop – each iteration corresponds to one control step.

**Figure 2.** Project overview of Embodied.cpp. Diverse sensors and datasets enter through input adapters, model execution is unified around one embodied-model runtime that covers VLA models, WAMs, and future variants, and outputs are bridged to simulators and real robots through deployment adapters. Beneath this shared path, three runtime capabilities directly address the core challenges: modular multi-rate execution, latency-first batch-1 execution on heterogeneous hardware, and an embodied AI kernel warehouse for reusable operators and model-specific kernels.

Performance Evaluation

Evaluation shows Embodied.cpp matches VLA behavior while cutting latency and memory.

Embodied.cpp reduces per‑block latency by 0.065 ms compared to the Python baseline.

Table 3 shows latency per block of 3.236 ms for the Python original and 3.171 ms for Embodied.cpp.

Both VLA models run correctly through the C++ runtime while preserving their respective success rates—HY‑VLA achieves 100 % and pi0.5 reaches 91 %—but latency and memory scale with backbone size, visual input complexity, and action‑chunk length.

**Table.** Comparison of inference runtime performance between Python original and Embodied.cpp.

**Table 4.** LingBot-VA single-block quantization microbenchmark.

Moving from Python to Embodied.cpp cuts per‑block latency and memory, enabling faster, lighter embodied inference.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers