KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill

KnowAct-GUIClaw improves GUI agent performance by combining a host-orchestrator with a self-evolving skill library.

How can mobile agents automate complex, cross-platform GUI tasks by combining self-evolving memory with a structured host-executor architecture?

Autonomous agents struggle to interact with graphical user interfaces (GUIs) because they lack persistent memory of past workflows and often fail to distinguish between high-level planning and low-level screen manipulation. KnowAct-GUIClaw addresses this by splitting the agent into a host-orchestrator that manages long-horizon tasks and a lightweight GUI sub-agent that executes screen-based actions using a self-evolving library of validated skills. On the MobileWorld benchmark, this framework achieves a 64.1% success rate, outperforming both general-purpose models and specialized GUI agent frameworks.

Paper Primer

The framework organizes execution into a four-stage loop: Know, Route, Act, and Reflect. The host agent delegates tasks to the GUI executor only when visual interaction is required, while the executor distills successful trajectories into reusable, state-validated skills that bypass redundant navigation.

KnowAct-GUIClaw achieves state-of-the-art performance on long-horizon mobile tasks.

MobileWorld benchmark (GUI-only subset) success rate. 64.1% pass@1, surpassing GPT-5.5 and specialized GUI models.

Experience and skills are transferable across different base models.

A Qwen3.5-35B executor using skills distilled from Kimi-K2.6 trajectories. 16.2% improvement over the base executor's performance.

Why separate the host from the GUI executor instead of using a single monolithic agent?

A monolithic agent often suffers from inefficiency and fragility when handling long-horizon tasks that span multiple apps. Separating roles allows the host to manage high-level context and non-GUI tools, while the executor focuses on visual grounding and device-specific state management.

What happens when a previously learned skill fails during execution?

The reflection stage identifies the failure, records the screen evidence and error, and triggers an evolution prompt to repair the skill in place—such as by adding guarded obstacle handling—rather than discarding it or replacing it with an unrelated workflow.

Introduction and Motivation

We expose why existing GUI agents falter and introduce a host‑executor framework to overcome them.

Current GUI‑centric agents such as OpenClaw struggle with complex, multi‑step tasks because they cannot retain interaction history nor coordinate work across heterogeneous applications, leading to brittle and inefficient behavior.

Existing agents miss two essential capabilities: (1) a cross‑platform memory that remembers what was done on one app for use on another, and (2) a mechanism to evolve their own skills from past successes and failures.

The shift from monolithic agents to a host‑executor framework is the key to scalable, efficient GUI automation.

Related Work

We situate KnowAct‑GUIClaw among prior personal assistants, GUI agents, and memory‑reuse research.

We introduce a two‑tier host–executor collaboration where the host delegates bounded GUI subtasks to an executor and receives standardized outcomes (completed, partial, or blocked) together with progress cues. This design lets the host resume unfinished work or reorder workflows, improving success rates while reducing compute.

Our routing layer classifies requests as single‑app or cross‑app flows and routes them accordingly; a temporary shared board carries explicit input‑output tuples between subtasks, while a persistent routing memory caches candidate applications and auxiliary context. The explicit subtask interface enables reliable data hand‑off across applications.

We distill execution traces into parameterized, state‑validated skills that the executor can commit to as a single decision, covering common click‑then‑type patterns and Android deeplink shortcuts when their launch contracts are satisfied. This abstraction unifies frequent UI primitives under a reusable skill contract.

After each run, a reflection step summarizes successes and failures, updates an experience memory, and refines the skill library; the refreshed knowledge feeds back into the router before decomposition and into the executor during execution. The loop closes the gap between past runs and future decisions.

Personal‑assistant agents pair large language models with tool interfaces, memory, and control loops; early systems such as ReAct, the Agents framework, and OS‑Copilot pioneered this pattern. More recent configurations like OpenClaw and Nanobot expose declarative channels, tools, and long‑lived workspace memory, while Hermes emphasizes continual adaptation via reusable routines.

GUI‑focused agents extend this paradigm to visual interfaces. Web‑centric works (Mind2Web, WebArena, VisualWebArena, SeeAct, WebVoyager) explore long‑horizon navigation via HTML or rendered screenshots. Mobile‑centric suites (Android in the Wild, AndroidWorld, OSWorld, OmniACT, AppAgent, Mobile‑Agent) and model‑centric systems (CogAgent, OS‑Atlas, ShowUI, UI‑TARS, Aguvis) improve perception and action generation on phones and desktops.

Memory and skill reuse enable agents to improve without weight updates. Approaches such as Reflexion, Generative Agents, Voyager, and KnowAgent store feedback or executable primitives for later retrieval. ReasoningBank, LearnAct, and CUA‑Skill further distill success/failure rationales into searchable memory, while recent surveys (Du) formalize the write‑manage‑read loop for agent memory.

The KnowAct-GUIClaw Framework

KnowAct‑GUIClaw adds a host‑executor stack with memory and a blackboard to enable reliable multi‑app GUI tasks.

Existing GUI agents stumble on long‑horizon tasks because they cannot remember past interactions nor route subtasks across apps, leading to dead‑ends and wasted tokens.

The host holds high‑level user intent and long‑term context, while the executor (GUI subagent) runs low‑level actions on a device, feeding results back to the host.

The framework cycles through four stages—gathering context, planning a workflow, executing actions, and learning from the outcome—so each long‑horizon task is treated as a reusable loop.

**Figure 2.** Overview of the KNOWACT-GUICLAW execution loop. Two persistent stores—a memory and history store and a skill and shortcut store—supply advisory context to every stage. Know gathers evidence and assembles a reasoning context; Route ranks app candidates and turns the request into either a single GUI task or an ordered multi-app workflow whose subtasks exchange typed values through a blackboard; Act runs GUIClaw’s observe-reason-act loop over the hybrid action space of GUI primitives, skills, deeplink/intent shortcuts, and intervention actions; and Reflect distills each trajectory into updated skills and experience memory that feed back into the stores.

Before any GUI action, the host actively retrieves relevant memories and candidate skills, presenting them as advisory hints that never override the current instruction.

**Figure 3.** Experience memory improves a GUI task by changing the task context before low-level control begins. Without the retrieved memory (Top), GUIClawinvites continues through Mastodon’s mobile settings and reaches a nonproductive path for invite-link creation. With the retrieved memory (bottom), the Know stage supplies an advisory lesson that invite links with advanced settings that require the web administration panel; GUIClaw then opens the web interface, navigates to account settings, and reaches the invite-people page. The example shows that experience memory guides app choice, decomposition, and recovery while live screen observations still ground each action.

The router emits either a single GUI task or an ordered multi‑app workflow, each subtask expressed as a typed tuple that declares exactly what it needs and what it returns.

Blackboard B₀ is empty.

Subtask 1 reads no inputs, launches Twitter, extracts the article URL, and writes $B₁\{url\}="https://…"$.

Subtask 2 reads $url$ from $B₁$, composes a tweet, posts it, and writes no outputs.

The workflow finishes with $B₂$ containing the same $url$ entry, confirming successful data transfer.

The blackboard enforces a strict contract: only declared keys flow between subtasks, preventing accidental leakage of unrelated screen state.

**Figure 4.** Blackboard-mediated execution in the Route stage. The short-lived blackboard $B$ stores typed inputs and outputs known so far. Each subtask $(g_i, h_i, I_i, O_i)$ checks its declared inputs $I_i$, reads their values from $B$, runs GUIClaw's observe-reason-act loop to produce a trajectory $\tau_i$, and writes only its declared outputs $O_i$ back to $B$ (2). A missing required input or output makes the workflow fail closed, so later subtasks consume observed typed values rather than free-form summaries.

GUIClaw executes each subtask using a unified action space that blends raw GUI primitives, learned skills, validated shortcuts, and user‑prompted interventions.

Observe–Reason–Act loop for a single subtask.

After a GUI task finishes, Reflect compresses the raw action trace into a concise note and, when appropriate, extracts a reusable skill that enriches experience memory.

```python @skill(name='jd_splash', app='com.jingdong.app.mall', platform='android', tags=['shortcut', 'deeplink', 'validated'], success_count=1) async def jd_splash(device): await action('open_deeplink', target='JDAnalytics:', package='com.jingdong.app.mall', valid_state='No need to verify') @skill(name='jd_search', app='com.jingdong.app.mall', platform='android', tags=['shortcut', 'intent', 'validated'], success_count=1) async def jd_search(device, query): await action('open_intent', intent_action='android.intent.action.SEND', package='com.jingdong.app.mall', component='com.jingdong.app.mall/...SearchBridgeActivity', mime_type='text/plain', extras=[['android.intent.extra.TEXT', '{{query}}']], valid_state='No need to verify') ```

Experimental Results and Analysis

Key performance numbers and efficiency analysis for KNOWACT‑GUICLAW.

KNOWACT‑GUICLAW achieves a 64.1 % success rate (SR) on the MobileWorld GUI‑Only benchmark, the highest reported among all evaluated models.

Table 1 shows the +host & memory & skills configuration of the open‑source Kimi‑K2.6 model reaching 64.1 % SR, surpassing the next best 61.5 % and all specialized GUI models.

**Figure 1:** The success rate (SR) comparison on MobileWorld GUI-Only tasks. The bars summarize Table 1 together with the additional Kimi-based KNOWACT-GUICLAW runs; gray bars denote specialized GUI models, colored external bars denote general model families, and highlighted bars denote KNOWACT-GUICLAW variants with memory and skills. The experimental results show that KNOWACT-GUICLAW achieves SOTA performance and that the memory and skill are effective for different base models.

**Figure 8.** Cart-to-SMS cross-app execution. The TaoDian GUI task extracts the product names, order number, and recipient phone number and transfers them to the downstream messaging task. A validated messaging shortcut opens the SMS compose view with the recipient and message body already populated, illustrating both blackboard information transfer and navigation compression.

On the AndroidDaily benchmark, KNOWACT‑GUICLAW’s “Resolved” setting attains a total score of 78.61, outpacing all baselines across task types, complexities, and ambiguity levels.

Supplementary Case Studies and Details

Appendix details action definitions, prompt templates, and additional case‑study references.

This appendix aggregates external papers that contextualize the GUI‑agent literature, providing a quick reference for readers interested in related benchmarks and frameworks.

The action space exposed by GUIClaw unifies desktop and mobile interactions under a common core, while adding platform‑specific extensions for key‑based shortcuts and an explicit enter key on mobile.

The table lists available actions categorized by environment: Shared, Desktop, and Mobile. Each row provides the action name (with parameters) and a brief definition of its function.

| Action | Description | JSON Format Example | | :--- | :--- | :--- | | 'click' | Tap visible element (describe clearly) | '{"`action_type`": "click", "coordinate": [x, y]}' | | '`double_tap`' | Double-tap visible element (describe clearly) | '{"`action_type`": "`double_tap`", "coordinate": [x, y]}' | | '`long_press`' | Long-press visible element (describe clearly) | '{"`action_type`": "`long_press`", "coordinate": [x, y]}' | | 'drag' | Drag from visible element to another visible element (describe both clearly) | '{"`action_type`": "drag", "`start_coordinate`": [x1, y1], "`end_coordinate`": [x2, y2]}' | | '`input_text`' | Type into field | '{"`action_type`": "`input_text`", "text": "Hello"}' | | 'answer' | Respond to user | '{"`action_type`": "answer", "text": "It's 25 degrees today."}' |

The Execution Prompt defines the runtime contract for Android agents: the model must output exactly one JSON action from the table, obeying ordering rules such as using answer to reply to the user and wait to signal idle periods.

The Skill Extraction Prompt instructs a secondary agent to compress a full trajectory into a reusable, parameterized skill, discarding non‑executable traces and preserving state contracts for robustness.

The Skill Evolution Prompt takes a failing reuse instance, the original skill JSON, and a failure trace, then returns an updated skill that guards against the identified error while keeping the original intent.

The Shortcut Validation Prompt verifies whether a deeplink or intent successfully opened the intended page and whether the payload survived, emitting a concise capability descriptor when usable.

The Experience‑Memory Induction Prompts extract up to three actionable lessons from successful or failed trajectories, focusing on navigation patterns, form‑filling tricks, and UI pitfalls without leaking concrete query strings.

Finally, concrete skill examples illustrate compact actions (click‑then‑type) and multi‑step skills (`search_product`) together with validated shortcuts (`jd_splash`, `jd_search`), showing the full syntax expected by the GUIClaw runtime.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers