ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-Modal Memory

Jiayi Tian, Shiao Liu, Yuting Xu, Jia Lu, Zihao Guan, Honglin Han, Di Yang, Minqi Gu, Yifei Qian, Tianlin Zhang, Yanqing Zhu, Zeqian Ye, Menglin Yang, Fei Wang, Xu Hu, Xiuxian Li, Wei Zhang, Shihui Su, Yiyan Ji, Jingbo Wang, Ziteng Feng, Jiaheng Liu, Zhaoxiang Zhang, Xiaolong Wu, Mingyang Yin, Zedong Chu, Mu Xu

ABot-AgentOS provides a modular, memory-augmented operating system layer for long-horizon robotic reasoning and execution.

How can we build a general-purpose robotic operating system that bridges high-level reasoning with low-level robot control through a lifelong, multi-modal memory system?

Robotic agents often struggle to bridge the gap between high-level semantic reasoning and reliable physical execution, frequently failing due to procedural drift or an inability to maintain persistent, multi-modal experience. ABot-AgentOS introduces a deliberative agent layer that sits above low-level controllers, using a hierarchical framework of a main planner, a skill runner for isolated subtasks, and a multi-stage verifier to ensure actions are grounded in environment facts. On the EmbodiedWorldBench benchmark, this architecture improves task success and goal completion over single-controller baselines, while its self-evolving memory system achieves up to 89% accuracy on multi-modal memory tasks.

Paper Primer

The system addresses the "reasoning-execution gap" by decoupling high-level planning from procedural motor control. It uses a "Skill Runner" to handle subtasks in an isolated context, preventing the main reasoning thread from being overwhelmed by low-level sensor noise or repeated movement failures.

The core memory mechanism is a Universal Multi-modal Graph Memory: it converts raw observations, dialogue, and spatial relations into typed nodes and edges, allowing the agent to retrieve source-grounded evidence subgraphs rather than relying on unstructured text caches.

ABot-AgentOS significantly outperforms single-controller baselines in long-horizon embodied tasks.

Evaluation on the EmbodiedWorldBench suite, which tests navigation, object search, and NPC interaction across 16 indoor/outdoor scenes. Consistent improvements in both Task Success Rate (TSR) and Goal Completion Rate (GCR).

Lifelong self-evolution improves memory retrieval and reasoning accuracy over time.

Performance on the Mem-Gallery benchmark increased from 88.6% to 89.0% after applying failure-driven self-evolution assets. Incremental but consistent gains across LoCoMo, OpenEQA, and Mem-Gallery benchmarks.

Why does the system need a separate "Skill Runner" instead of just letting the main LLM issue all commands?

The Skill Runner provides context isolation; it absorbs the procedural complexity of repeated movements, collisions, and local recoveries, returning only a compressed summary to the main LLM to keep the global reasoning thread coherent.

How does the system ensure that its "self-evolution" doesn't just memorize the test set?

The system enforces a strict no-leakage constraint: failure-driven "evo-assets" generated from one evaluation split are gated and only promoted for use in later, unseen splits.

ABot-AgentOS demonstrates that decoupling cognition from actuation via a structured OS layer is a viable path toward scalable, persistent, and auditable robotic intelligence.

Abstract

We introduce ABot‑AgentOS and EmbodiedWorldBench, demonstrating memory‑driven gains on long‑horizon embodied tasks.

Recent VLM and VLA advances improve perception, but long‑horizon embodied agents still need a runtime layer for reasoning, memory, tool use, verification, and cross‑embodiment execution. We present ABot‑AgentOS, a deliberative Agent Operating System that sits above low‑level controllers and provides scene‑conditioned planning, context‑isolated skill execution, multi‑stage verification, multi‑modal memory, and edge‑cloud collaboration. In the new EmbodiedWorldBench (16 scenes, 200+ tasks), ABot‑AgentOS outperforms a single‑controller baseline and, with its universal multi‑modal graph memory and failure‑driven self‑evolution loop, achieves 87.5 % LoCoMo, 59.9 % OpenEQA EM‑EQA, 88.6 % Mem‑Gallery, 76.5 % Acc@All on NExT‑QA, with self‑evolution raising these scores to 88.7 %, 60.4 %, and 89.0 % respectively.

The Need for Agent Operating Systems

We expose the reasoning, memory, and embodiment gaps and introduce ABot-AgentOS to close them.

Embodied intelligence is moving AI from purely digital settings into the physical world. Recent Vision‑Language‑Action (VLA) models give robots natural‑language understanding, visual scene comprehension, navigation, and action prediction.

However, three fundamental challenges remain: translating high‑level semantic reasoning into reliable multi‑step execution, generalizing capabilities across diverse robot morphologies without extensive retraining, and building persistent, multi‑modal memory for continuous interaction.

Existing work addresses parts of these challenges but leaves gaps. Foundation‑model systems inspired by Dual‑System Theory (e.g., Galaxea G0, Hi Robot, RoboBrain) connect perception to motor commands but are tied to specific model stacks or hardware. General‑agent research (ReAct, Toolformer, OpenAI Operator, etc.) shows the value of explicit reasoning and tool use, yet physical execution adds partial observability, actuation uncertainty, and verification needs. Long‑term memory systems (MemoryBank, MemGPT, Mem0) store experience, but lack the multi‑modal, source‑grounded structure required for embodied agents.

ABot-AgentOS is a deliberative runtime layer that sits between high‑level reasoning and low‑level robot controllers, separating multi‑modal memory from hardware so agents can learn from failures and improve over time.

The ABot-AgentOS Architecture

How the Agent Harness orchestrates reasoning, execution, and memory for embodied robots.

The system stacks a deliberative Agent Harness atop existing robot controllers, linking microphones, cameras, and apps to a dual‑LLM loop that runs a Tiny LLM on‑device and escalates to a cloud Large LLM when needed.

The Harness closes the reasoning‑execution gap by inserting a verifier between a high‑level planner and low‑level skill actors, so the agent only sees a concise summary of what actually happened.

Step 1: Main LLM outputs plan = {pick‑up A, place B}.

Step 2: Verifier queries the camera, finds the cup at location C, returns “mismatch”.

Step 3: Harness rewrites the plan to “navigate to C; pick‑up C; place B”.

Step 4: Skill Runner executes the revised plan, reports success.

The verifier prevents the agent from blindly executing a stale plan, turning a potential failure into a corrective replanning step.

How does Verification‑aware ReAct differ from the original ReAct loop?

ReAct interleaves reasoning and tool calls but trusts the tool’s output. Verification‑aware ReAct inserts a separate verification submodule that checks the physical outcome before the next reasoning step, so the LLM can revise its plan based on reality rather than on assumed success.

The planner tailors its high‑level plan to the robot’s current visual and spatial context, because the same instruction can require different actions in different scenes.

Why not let the LLM plan without looking at the scene?

Without scene grounding the LLM would generate a plan that assumes the object is already reachable, leading to wasted motion or collisions. The scene check injects the missing reality check that aligns language intent with physical feasibility.

The Skill Runner isolates low‑level loops (e.g., repeated navigation steps) so the main LLM sees only a concise outcome.

How is the Skill Runner different from calling a tool directly?

A tool call is a single atomic operation with an immediate result. The Skill Runner is a persistent sub‑agent that can loop, observe, and adapt over many timesteps before summarizing its outcome, which is essential for tasks like navigation that require iterative refinement.

Verification happens before, during, and after a subtask, ensuring the agent’s belief never diverges from reality.

Why not verify only at the end of the task?

Late verification can only report failure after costly execution has already occurred. Early and intermediate checks catch drift early, saving time and preventing damage.

The routing layer learns when a cheap on‑device model suffices and when to summon the powerful cloud LLM, balancing latency and capability.

How does the router avoid “always‑escalate” behavior?

The policy incorporates a cost term for cloud usage and a success predictor; it only escalates when the predicted gain outweighs the latency penalty, as learned from past verification outcomes.

The Memory System sits beneath the runtime, providing a persistent, multi‑modal graph that stores compact, source‑grounded records instead of raw video or transcript dumps.

Private edge memory holds robot‑specific experience, while a shared cloud memory aggregates reusable knowledge, enabling both personalization and collective learning.

What prevents private sensory data from leaking into the common memory?

A gating module inspects each record for personally identifiable information (faces, names, personal objects). Only records passing a 99 %‑accurate classifier are uploaded; everything else stays in the edge‑side private graph.

The memory is a typed directed graph where nodes are entities or evidence units and edges encode temporal, spatial, and semantic relations.

Why not store memories as plain text chunks?

Plain chunks lose relational information; the graph preserves who, what, where, and when, which is essential for embodied queries that involve identity continuity or spatial reasoning.

Adapters convert raw sensor streams, dialogue turns, and tool outputs into typed graph nodes, ensuring all modalities speak the same schema.

How does the system avoid exploding graph size over months of operation?

It merges near‑duplicate entities, caps per‑entity evidence nodes, and prunes stale facts by linking newer observations with temporal edges rather than deleting them outright.

Compute sem$(q, v_1)=0.92$, lex$(q, v_1)=0.8$, meta$(q, v_1)=0.9$, type$(q, v_1)=1.0$ → $s(q, v_1)=0.93$.

Compute $s(q, v_2)=0.68$ (older timestamp lowers meta).

Compute $s(q, v_3)=0.15$ (type mismatch).

Select $v_1$ as seed, expand one hop to retrieve the “location A” node and its spatial edge.

The scorer quickly discards irrelevant nodes (different color or stale location) and surfaces the freshest, semantically matching evidence for grounding the answer.

The system treats each failed interaction as a diagnostic case, compiles a repair asset, and only deploys that asset in future splits, guaranteeing no leakage from the current episode.

Why can’t the system apply a newly compiled asset immediately to fix the current failure?

Applying it instantly would change the execution graph mid‑run, making the recorded failure trace inconsistent with the repaired behavior. The no‑leakage constraint ensures a clean separation between diagnosis and deployment.

Edge‑cloud collaborative memory management enforces a privacy‑by‑default policy: private edge memory never uploads unless a classifier deems the record shareable, while public observations are synchronized to the common cloud graph.

**Figure 1** System architecture of the proposed robot agent. Inputs from multiple sources (microphone, APP, camera) to a dual-LLM core, where a Tiny LLM on the edge handles every turn and escalates to a cloud-based Large LLM on demand. The Agent Harness manages verification-aware ReAct loop, context, and skill evolvement, enabling an extensible skill library (Manipulation, Navigation, Motion, Vision, etc.). A hierarchical Memory System synchronizes private edge memory with shared cloud memory to support cross-robot knowledge transfer. Outputs are dispatched to the robot hardware for execution.

**Figure 2** Overview of the Agent Harness. The main LLM performs scene-conditioned planning with memory and context, delegates procedural subtasks to the Skill Runner, and receives corrective feedback from the Verifier to form a reasoning-execution-verification loop.

**Figure 3.** Overview of the multi-modal memory architecture. During online execution, ABot-AgentOS writes observations and interactions into a source-grounded memory graph, retrieves task-relevant evidence, and records retrieval and answer traces. Offline, failure traces are diagnosed and converted into gated runtime evo-assets for later deployments.

Evaluating Embodied Agents

A unified benchmark that tests navigation, interaction, and reasoning across indoor and outdoor scenes.

Prior embodied AI benchmarks split indoor and outdoor evaluation, each covering only a single capability and using static scenes.

EmbodiedWorldBench is a collection of fully executable scenarios that jointly test navigation, object search, NPC dialogue, dynamic instruction response, and state tracking across indoor, outdoor, and hybrid environments.

The benchmark follows four design principles: (1) each case is an executable scenario, (2) a single case tests multiple capabilities, (3) outcomes are trace‑grounded, and (4) execution is reproducible.

UnrealZoo provides photo‑realistic UE5 environments with NavMesh navigation; annotators explore each scene, marking objects, points of interest, and interaction targets to build a structured semantic map.

Queries pair a natural‑language instruction (e.g., “find the elderly person in a red jacket”) with formal success criteria; a large language model expands scene‑specific task templates, and humans verify executability and information isolation.

Semantic maps are normalized into three layers—point, room, and polygon—so that tasks can be generated automatically while preserving spatial relationships.

Difficulty levels are calibrated by human‑in‑the‑loop evaluation of exploration scope, procedural length, interaction complexity, evidence requirements, and need for dynamic replanning.

During evaluation, the agent receives only the filtered map and instruction; the UE engine runs the episode, records the full trajectory, and a deterministic evaluator computes Task Success Rate (TSR) and Goal Completion Rate (GCR).

**Figure 5. Overview of EmbodiedWorldBench.** EmbodiedWorldBench evaluates embodied agents on compound tasks that span indoor and outdoor spaces and require tightly coupled navigation, NPC interaction, and environment perception across diverse scenes. The benchmark covers 16 scenes across four difficulty levels with over 200 tasks, revealing the challenges of achieving cross-scene generalization and adaptive replanning under dynamic events.

Training the Deployable Agent

We detail a closed‑loop training pipeline that turns teacher trajectories into a self‑evolving reward engine for a deployable student policy.

The biggest obstacle to deploying ABot‑AgentOS‑style planning is that large teacher models are too heavy for on‑robot inference, yet we still need their long‑horizon tool‑use expertise.

Construct a controllable text‑based sandbox for each task instruction.

Run a strong teacher agent in the sandbox to generate ReAct‑style interaction traces.

Filter the traces with an LLM‑as‑a‑Judge and convert them into supervised fine‑tuning (SFT) examples.

Initialize a smaller student policy with the SFT data.

Continue training the student policy online via reinforcement learning, using the same sandbox and the LLM‑as‑a‑Judge reward engine.

Close the loop: the reward engine self‑evolves by validating its own outputs and updating its prompt.

**Figure 6** Overview of the training pipeline for a deployable ABot-AgentOS student policy. The pipeline constructs controllable text-based environments, distills teacher trajectories for SFT initialization, and improves the policy through online RL with LLM-as-a-Judge rewards and GiGPO advantages.

The pipeline turns reward generation into a feedback‑driven loop: the LLM‑as‑a‑Judge first scores trajectories, the Meta‑Judge checks those scores for reliability, and a set of lightweight agents rewrites the reward prompt wherever the validation fails.

Compute $\text{repisode}(\tau)=\tfrac13(0.8+0.9+1.0)=0.9$.

Sum turn rewards: $\sum_{t=1}^{2}\hat{r}_t = 0.4+0.2 = 0.6$.

Total return $R(\tau)=0.6+0.9=1.5$.

Meta‑Judge checks the episode‑level score and flags a low consistency weight because the agent never verified the door was unlocked before “open”.

Cluster groups this case under the “manipulation” skill; Analyzer pinpoints the missing “precondition check” rubric entry; Refiner adds a short example of a failed unlock attempt to the prompt; Validator re‑runs the judge on a held‑out set and sees the consistency score rise from 0.9 to 0.96.

The example shows how a single low‑quality case triggers a localized prompt edit, which immediately improves the consistency rubric without touching other skills.

How does the Self‑Evolution Pipeline differ from simply retraining the reward model on more annotated trajectories?

Retraining expands the model’s parameters globally, which can unintentionally shift well‑behaved rubrics. The Self‑Evolution Pipeline keeps the reward model frozen and instead edits only the textual prompt components that caused the failure, guaranteeing that improvements are isolated to the problematic skill or rule.

By closing the loop between reward generation, validation, and prompt refinement, the training pipeline continuously upgrades its own supervision signal, enabling a compact student policy to acquire the same long‑horizon planning abilities that were originally only present in massive teacher models.

Agent Performance Results

Agent evaluation shows the hierarchical architecture outperforms single‑controller baselines.

The ABot‑AgentOS runtime decouples high‑level reasoning and multi‑modal memory from low‑level robot control, letting agents learn from failures and improve over time.

Using the hierarchical architecture with Qwen3.6‑Plus raises Task Success Rate by +11.99 % and Goal Completion Rate by +10.84 % compared to the single‑controller ReAct baseline.

ReAct achieves 49.97 % TSR and 57.95 % GCR; ABot‑AgentOS + Qwen3.6‑Plus reaches 68.18 % TSR and 74.62 % GCR.

**Table 1.** Agent evaluation results on the EmbodiedWorldBench subset.

The gains stem from maintaining structured task state while delegating sustained local execution to skill sub‑agents and verifying progress, which mitigates context drift, premature termination, and weak evidence‑gathering failures.

Memory Module Analysis

We isolate the memory module to see how each component impacts performance.

The ABot‑AgentOS runtime separates high‑level reasoning from low‑level hardware, letting the memory layer learn from past failures while the controller stays unchanged.

LoCoMo measures a robot’s ability to recall facts from many past conversational sessions spanning weeks.

How does LoCoMo differ from ordinary long‑term QA benchmarks?

LoCoMo forces the system to combine dozens of separate conversation logs, preserving temporal order and speaker identity, whereas typical long‑term QA only aggregates a single document collection.

EM‑EQA tests embodied agents that must answer open‑ended questions about a 3‑D scene using visual, textual, and proprioceptive cues.

Why is EM‑EQA considered “embodied” rather than a standard VQA task?

Because the agent must incorporate its own motion history and sensor timestamps, not just a single image, to resolve the query.

All experiments share the same hybrid graph retriever (semantic + lexical seed selection followed by typed edge expansion). The memory writer converts raw observations, dialogue turns, images, video frames, timestamps, and interaction traces into source‑grounded graph records; the answerer consumes the retrieved evidence context. Benchmark‑specific base models (Qwen3.6‑Plus, GPT‑5.4, Qwen3.5‑Flash) are used as writer/answerer.

ABot‑AgentOS Static scores 87.5 overall on LoCoMo, surpassing the strongest reproduced baseline by 1.9 points.

Table 2 shows 87.5 vs. Mem0’s 85.6.

On the EM‑EQA benchmark, ABot‑AgentOS Static attains 59.9 overall, the highest among listed memory‑based methods.

Table 3 reports 59.9 versus lower scores for scene‑graph and caption‑memory baselines.

ABot‑AgentOS Static reaches 88.6 overall on Mem‑Gallery, outperforming textual and multimodal memory baselines.

Table 4 lists 88.6 compared with the next best memory baseline below 86.

On NExT‑QA, the static system achieves 76.5 Acc@All, a 3.2‑point gain over the prior GraphVideoAgent baseline.

Table 5 shows 76.5 vs. GraphVideoAgent’s 73.3.

ABot‑AgentOS – Qwen3.5‑Flash obtains 65.4 % average accuracy on EgoLife, the best among listed agentic memory methods.

Table 6 reports 65.4 versus the next best 63.2.

Self‑evolution adds runtime policies (query‑rewriting, temporal‑normalization, evidence‑selection, answerer‑calibration, frame‑selection) after a split’s failures are diagnosed. Across all five benchmarks, this yields consistent gains: +1.2 on LoCoMo, +1.2 on OpenEQA, +4.1 on NExT‑QA, +0.4 on Mem‑Gallery, and +0.8 on EgoLife, demonstrating that the same memory graph can improve without altering the retriever.

**Figure 4.** Concrete memory failure-to-evolution examples. Left: visual memory QA retrieves image-grounded identity evidence but can expose missing breed-specific cues. Right: temporal text memory QA uses session metadata to resolve relative dates but can reveal temporal-normalization errors. In both cases, the failure trace is converted into targeted memory-writing, evidence-selection, frame-selection, or answering improvements.

**Figure 7** Lifelong memory self-evolution across sequential splits. Each split uses only evo-assets promoted from previous splits; failures from the current split are diagnosed and gated after evaluation, and accepted assets are used only by later splits.

**Figure 8.** Self-evolution gains by benchmark and category. Bars show absolute score changes from Static to + Self-evolution; hatched bars indicate the primary metric for each benchmark.

Conclusion

We wrap up the system’s impact, limits, and list contributors.

ABot-AgentOS unifies a deliberative agent layer with VLM/VLA models, offering modular reasoning, context management, skill execution, and verification, while its typed graph memory supports long‑term recall and self‑evolution improves later task splits.

Limitations include the need for large‑scale real‑world validation under noisy perception and heterogeneous robots, incomplete benchmark diversity, reliance on structured traces, and privacy‑aware edge‑cloud memory sharing; future work will expand scenario coverage, improve privacy controls, and integrate visual observations into the training pipeline.

The paper credits the research, engineering, and acknowledgment teams, lists benchmark construction contributors, and names the project leads and advisor.

Memory Benchmark Prompts

Judge prompts used for the three LLM‑graded memory benchmarks.

This appendix lists the exact prompts that drive the three LLM‑judged memory benchmarks. Each prompt encodes the answer format and scoring protocol required by the corresponding benchmark.

OpenEQA Judge Prompt

Mem‑Gallery Judge Prompt

Self-Evolution Trace Details

Details the self‑evolution loop, candidate generation, safety gating, and resulting runtime policy.

The LoCoMo Memory‑Recall Judge is a JSON‑based evaluator that labels generated answers as CORRECT or WRONG according to a set of semantic credit rules.

This appendix also records a full OpenEQA lifelong self‑evolution run, illustrating how the system iteratively improves its memory‑based retrieval policy without touching the dataset schema.

Incumbent evaluation – run the current memory system on the split.

Diagnosis – the Diagnoser clusters failures by memory root cause.

Hypothesis generation – the HypothesisGenerator proposes generic JSON DSL candidate assets for writer, retriever, answerer, or frame‑policy layers.

Compilation and safety review – the CompilerCritic discards executable code, schema migrations, direct answer resolvers, fixed answer tables, and qid/gold‑answer dependencies.

Gate analysis – the GateAnalyst adds protected‑category, activation‑guard, cost/lifecycle, and regression‑risk constraints.

Candidate evaluation – each materialized asset must improve target metrics, avoid global regression, keep low‑score count down, and respect protected categories.

Stack confirmation – only assets whose combined stack beats the incumbent are retained; otherwise they are deprecated.

In `split_00` the Diagnoser flagged a missing typed object node: an observed blanket on a bed was not represented in the graph, causing retrieval to return unrelated objects and the answerer to miss the question.

The resulting retriever policy ranks observation‑grounded object memories ahead of generic scene summaries, requiring matching room/place cues, explicit last‑seen timestamps, and directed spatial relations, while forbidding inferred room priors or reversed relations.

Key observations: protected object‑state recognition never regressed, and object‑localization improved by roughly $+0.073$, confirming that the self‑evolution loop refined the memory‑based retrieval without introducing answer‑specific shortcuts.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers