Lame M-VLA: Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang, Xinlei Yu, Rui Yan, Wenguan Wang, Xiangbo Shu, Shuicheng Yan

LaMem-VLA integrates historical robotic experience as native latent tokens to resolve long-horizon manipulation tasks.

How can we integrate long-term historical context into Vision-Language-Action (VLA) models without relying on external, policy-side memory banks that struggle with temporal dependencies?

Vision-Language-Action (VLA) models typically predict actions based only on the current observation, causing them to fail in long-horizon tasks where understanding past state transitions and task progress is essential. LaMem-VLA treats historical experience as context-native latent memory: it curates past visual and semantic data into two vaults, retrieves relevant evidence, and weaves it into the model's internal reasoning sequence as compact latent tokens. This approach allows historical memory to participate directly in the model's multimodal reasoning, achieving a 73.9% success rate on the SimplerEnv-Bridge benchmark and outperforming existing memory-augmented baselines.

Paper Primer

Existing memory-augmented VLA models often treat history as an auxiliary scaffold, attaching it to the policy after the model has already finished its internal reasoning. This rigid separation prevents the model from fluidly interleaving past perceptual evidence with current scene understanding, limiting its ability to resolve temporally ambiguous states.

LaMem-VLA functions like a librarian managing two specialized archives: a short-term vault for visual perceptual details and a long-term vault for semantic task-progress cues. The system retrieves relevant evidence from these archives and compresses it into fixed-length latent tokens, which are then prepended to the current input sequence so the model can attend to past and present information simultaneously.

LaMem-VLA significantly improves success rates on long-horizon robotic manipulation tasks compared to memory-free and policy-side conditioning baselines.

Evaluations on the LIBERO benchmark across five suites, including long-horizon reasoning tasks. Achieved a 97.6% average success rate, outperforming the MemoryVLA baseline by 1.1 points and the CogACT baseline by 4.4 points.

Native latent integration of memory outperforms external policy-side conditioning.

Ablation studies comparing the full LaMem-VLA model against a version where memory is only used as an external condition for the action policy. The latent-native approach achieved 73.9% success on SimplerEnv-Bridge, compared to 71.9% for policy-side conditioning.

Why is it necessary to distinguish between short-term and long-term memory vaults?

The two vaults serve complementary roles: short-term memory preserves visually grounded evidence like object locations and transient state changes, while long-term memory tracks semantic task progress and action continuity across extended horizons.

Does this method increase the computational cost as the robot performs longer tasks?

No. The system uses a fixed-length latent interface and a memory curator that compresses redundant historical units when the vault capacity is exceeded, ensuring the reasoning context remains bounded regardless of the episode length.

Robotic policies can achieve better temporal awareness by treating historical experience as a native component of the model's latent reasoning space rather than an external auxiliary input.

The Markovian Bottleneck in VLA Models

We expose the Markovian limitation of current VLA models and motivate latent‑memory integration.

Current Vision‑Language‑Action (VLA) models operate under a Markovian assumption, treating each timestep as independent and basing action decisions solely on the immediate observation. This simplification yields a temporal short‑horizon bias: models can react to the present scene but cannot reason about earlier states, completed steps, or the phase of multi‑step tasks, leading to poor performance on long‑horizon manipulation.

The assumption that a VLA’s action at time t depends only on the observation at time t, ignoring any prior observations or actions.

**Figure 1.** Paradigm comparison of memory-augmented VLA Models. (a) Unlike previous VLA models that store historical experience in an auxiliary memory bank and consume retrieved memory as external policy-side context, (b) LaMem-VLA treats historical experience as context-native latent memory, which is stored, retrieved, and consumed in the model embedding space.

The core limitation of current VLA models is their reliance on a Markovian view, which hampers performance on long‑horizon robotic tasks.

Prior Approaches to VLA Memory

We situate LaMem-VLA among prior VLA approaches, highlighting their memory limitations and categorizing related work.

Directly discretize continuous actions within a unified vision‑language backbone and autoregressively predict action tokens.

Use pretrained vision‑language models for high‑level reasoning and a separate generative policy (diffusion or flow‑matching) to synthesize smooth action trajectories.

Interleaves historical frames with language tokens or aggregates multi‑frame features within a predefined context window.

Compresses past interactions into lightweight proxies such as recurrent latent states, motion‑centric cues, action summaries, or a sparse set of representative observations.

Stores historical observations and action tokens in an auxiliary memory bank and retrieves a compact set of task‑relevant evidence to condition the current policy.

LaMem-VLA Architecture

We embed dual‑scale latent memory into the VLA token stream to overcome short‑horizon bias.

Markovian VLA models predict actions from the current observation and instruction alone, which creates a temporal short‑horizon bias in long‑horizon manipulation tasks. Historical cues—completed subtasks, state transitions, and progress signals—are essential for reliable control, yet existing memory augmentations sit outside the model’s token space. LaMem‑VLA resolves this by weaving memory directly into the VLA reasoning sequence.

Think of the model as a two‑drawer filing cabinet: one drawer ($M_{\text{short}}$) holds recent visual snapshots, the other ($M_{\text{long}}$) holds longer‑term action summaries, and both are pulled into the token stream whenever the model needs context.

Curator updates $M_{\text{short}}$ by appending $v_3$ and discarding the oldest token $v_1$, so $M_{\text{short}} = \{v_2, v_3\}$.

Seeker forms query $Q_{\text{con}}$ from $X_t$ and $I$, matches it to $v_2$ and $a_2$, and retrieves those two tokens.

Condenser compresses $v_2$ and $a_2$ into latent tokens $m_{\text{short}}$ and $m_{\text{long}}$ (e.g., via a small linear projection).

Weaver builds $S_t = [m_{\text{short}};\;m_{\text{long}};\;v_3;\;i;\;Q_{\text{action}}]$, a sequence of five tokens ready for the VLA encoder.

This toy walk‑through shows how the short‑term vault slides over recent visual evidence while the long‑term vault preserves action semantics, enabling the model to reason with both fresh perception and enduring intent.

How does this dual‑scale latent memory differ from the external memory banks used in prior VLA extensions?

External banks store history outside the token stream and are accessed via a separate read/write interface, so the model cannot attend to memory entries while processing other tokens. Dual‑scale latent memory lives inside the same token sequence, allowing the attention mechanism to treat memory tokens exactly like any other token, which yields seamless interleaving of perception, memory, and action reasoning.

**Figure 2.** The Framework of LaMem-VLA. Given an instruction and the current observation, the vision-language encoder first encodes the inputs into a multimodal representation. The memory curator (§3.3) organizes historical experience into dual memory vaults, and the memory seeker (§3.4) then retrieves task-relevant evidence from dual memory vaults based on this multimodal representation. This retrieved evidence is compressed into fixed-length latent memory tokens by the memory condenser (§3.4). Finally, the memory weaver (§3.5) injects these latent memory tokens into the reasoning sequence, producing memory-grounded action tokens that guide the action expert to generate future action chunks.

Core loop of LaMem‑VLA at timestep $t$.

Curating and Retrieving Latent Memory

Memory modules store, compress, and inject historical evidence into VLA reasoning.

Standard VLA models treat each timestep in isolation, missing the continuity needed for multi‑step manipulation.

A 7 B Prismatic vision‑language model pretrained on the Open‑X Embodiment dataset provides rich visual‑linguistic features for downstream reasoning.

It splits historical evidence into a short‑term visual vault and a long‑term action vault, each updated and compressed to keep memory compact.

Insert $(k_1^{s},v_1^{s})$ into $M_{\text{short}}$ → $n_s=1$ (no compression needed).

At timestep 2 a new visual batch yields $v_2^{s}$ and $k_2^{s}=[0.14,0.16,0.34,0.36]$; insert → $n_s=2$ (still within capacity).

At timestep 3 a third batch produces $k_3^{s}=[0.80,0.80,0.20,0.20]$; insertion makes $n_s=3>L$.

Compute cosine similarities: $\cos(k_1^{s},k_2^{s})\approx0.999$, $\cos(k_2^{s},k_3^{s})\approx0.12$; $i^{*}=1$ (most redundant pair).

Merge pair 1: $\tilde{k}^{s}=[0.145,0.155,0.345,0.355]$, $\tilde{v}^{s}$ is the element‑wise average of $v_1^{s}$ and $v_2^{s}$.

Resulting short‑term vault contains $\tilde{m}^{s}$ and $m_3^{s}$, keeping the size at $L=2$.

By merging only the most redundant adjacent entries, the curator preserves diverse visual evidence while guaranteeing a fixed memory budget.

How does this curator differ from a conventional external memory bank?

Typical external banks simply append new entries, leading to unbounded growth. The curator actively compresses adjacent redundant keys and stores action hidden states without keys, ensuring a bounded, information‑rich vault.

A context‑aware query pulls the most relevant short‑term visual and long‑term action units from the vaults.

Retrieved evidence is reshaped into fixed‑size latent tokens using lightweight transformers, keeping the VLA input size constant.

The condensed memory tokens are concatenated with the current observation and fed back into the VLA, letting self‑attention blend memory with perception.

A diffusion model denoises an action chunk conditioned on the memory‑augmented tokens, yielding smooth, history‑aware robot commands.

**Table 1.** Quantitative comparison results on SimplerEnv-Bridge [36] (§4.2) with WidowX robot.

Performance on Robotic Benchmarks

Dual‑scale latent memory lifts VLA performance on both SimplerEnv and LIBERO benchmarks.

Standard VLA models ignore historical context; LaMem‑VLA injects dual‑scale latent memory to provide that context.

LaMem‑VLA attains the highest average success of 73.9 % on SimplerEnv‑Bridge.

Table 1 shows LaMem‑VLA achieving 73.9 % avg, a 16.6 % gain over the CogACT baseline.

On the LIBERO benchmark LaMem‑VLA reaches an overall average success of 97.6 %.

Table 2 reports 97.6 % avg across five suites, surpassing the previous best MemoryVLA by 1.1 points.

**Table 3.** Detailed analysis of dual-scale latent memory on SimplrEnv and LIBERO.

**Table 3.** Ablation study results for the dual-memory design on SimplerEnv and LIBERO-90 benchmarks.

Component Contributions

Ablation experiments quantify how each memory component contributes to performance.

We systematically remove each memory component from the LaMem‑VLA input sequence to measure its impact on success rates.

Short‑term memory supplies current‑episode visual cues, while long‑term memory preserves task progress; removing either degrades temporally grounded reasoning, and removing both collapses performance to a memory‑free VLA.

Adding policy‑side memory raises SimplerEnv success to 71.9%.

Table 4 reports 71.9% for the Policy‑side Memory variant.

Conditioning on raw retrieved evidence yields 69.8% success on SimplerEnv.

Table 4 shows 69.8% for the Raw Retrieval Conditioning variant.

**Figure 3.** Ablation study of the latent memory token number on SimplerEnv [36] and LIBERO-90 [35] (§4.4).

**Table 5.** Ablation study of the retrieved memory unit number $K$ on SimplerEnv and LIBERO-90.

Retrieval Budget and Token Analysis

We ablate memory budgets and token counts to see how they affect success rate.

We ablate the memory retrieval budget $K$ and the latent token counts $L_s$ and $L_l$ to quantify their impact on success rate ($SR$).

Increasing $K$ from 2 to 4 yields clear gains, but the peak performance appears at $K=8$; beyond that the extra retrieved units become redundant and burden the memory condenser, causing a slight drop.

Increasing short‑term latent memory tokens $L_s$ improves success rate.

$SR$ rises from 61.4% at $L_s=2$ to 65.6% at $L_s=16$ on SimplerEnv.

With $L_s$ fixed at 8, raising the long‑term token count $L_l$ further pushes $SR$ up to the same best values observed for $K=8$, but larger token budgets also expand the VLA self‑attention context and increase compute, so we adopt the balanced default $(L_s, L_l)=(8,4)$.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers