MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks

Yi Zhu, Xiongwei Wu, Qiyi Wang, Tingyu Qu, Jiajun Liu, Sihan Cao, Long Chen, Weigao Sun, Feida Zhu, Yiran Zhong, Steven Hoi

MobilePA-Bench evaluates mobile AI agents on stateful, multi-step tasks requiring tool use, memory, and sub-agent coordination.

How do current LLM-based mobile agents perform when tasked with complex, stateful, and memory-dependent mobile operations?

Current mobile benchmarks are either limited to surface-level GUI clicking or rely on static, offline function-calling tests that ignore real-world runtime constraints. MobilePA-Bench provides an interactive, stateful sandbox where agents must orchestrate APIs, retrieve user memory, load composite skills, and delegate tasks to specialized sub-agents. Even the strongest frontier models achieve an overall weighted score of only 75.52%, revealing significant reliability gaps in complex, multi-stage mobile workflows.

Paper Primer

The benchmark treats the mobile agent as a central planner that manages a unified action space. It uses a stateful simulation sandbox: the agent issues function calls, and the environment returns dynamic feedback—such as system errors or database updates—forcing the agent to repair plans in real-time.

The core mechanism is a four-dimensional capability evaluation: Basic Tool Use (API execution), Sub-agent Collaboration (delegating visual/GUI tasks), Memory Usage (retrieving personal context), and Skill Usage (invoking multi-step procedures). The sandbox acts like a flight simulator: it tracks the "state" of the phone's databases and logs, ensuring that an agent's actions have persistent, verifiable consequences rather than just matching static text strings.

Frontier LLMs struggle with compound reliability in mobile environments.

The top-performing model (Claude-Opus-5) achieves an overall score of 75.52%, with performance dropping significantly in memory-bound and collaborative tasks.

Memory retrieval and sub-agent delegation are the primary system bottlenecks.

While models reached up to 83.85% in Basic Tool Use, performance in Memory Usage peaked at only 64.63% and Sub-agent Collaboration at 77.53%. Models fail more than one-third of memory-bound tasks on average.

Why is a stateful sandbox necessary for this benchmark?

Static benchmarks only check if an agent outputs the correct string, but mobile agents must handle runtime friction like permission denials or database conflicts. The sandbox provides real-time feedback, allowing the benchmark to test if an agent can actually recover from errors during execution.

How does this benchmark handle tasks that aren't just simple API calls?

It uses three evidence-aligned "Query Buckets": Tool Call (exact sequence), State Change (final database result), and Agent Behavior (evaluating if the agent correctly delegated to a sub-agent or asked for user clarification).

Mobile agent development must shift from isolated capability testing to joint reinforcement learning that prioritizes error recovery and the integration of memory with tool execution.

Introduction: The Need for Mobile Benchmarking

We expose the missing tool‑centric, stateful evaluation gap for mobile planner agents.

Large Language Models combined with autonomous agents are turning smartphones into on‑device copilots that must act, not just converse.

The field needs a benchmark that forces agents to plan across heterogeneous tools, retain personal context, and coordinate sub‑agents inside a live mobile OS.

Consequently, the experimental gap is clear: prior benchmarks either stop at surface‑level GUI clicks or evaluate isolated function calls, leaving tool‑centric, stateful planning unmeasured.

**Figure 1.** A representative end-to-end task in MobilePA-Bench, demonstrating four core planner capabilities: (1) Memory Retrieval for accessing local preferences and personal details; (2) Skills Execution for multi-step flight and hotel bookings; (3) Basic Tool Use for scanning multimodal QR codes; and (4) Sub-Agent Collaboration for delegating form filling to a visual GUI agent when structured APIs are unavailable.

The key shift is from evaluating isolated GUI actions to assessing unified, tool‑centric mobile agents that plan across APIs, memory, and sub‑agents.

Related Benchmarks and Paradigms

Survey of mobile agent benchmarks and how MobilePA‑Bench unifies their gaps.

Related work falls into three families: GUI‑centric benchmarks, static function‑calling suites, and advanced planning platforms.

AndroidWorld provides a simulated Android UI where agents receive screenshots and output pixel coordinates to interact with apps.

OSWorld models a full mobile operating system stack, exposing system calls and app lifecycles to agents.

**Table 1.** Holistic comparison of stateful, interactive, and tool-centric agent benchmarks. We cross-examine frameworks across distinct paradigms (GUI-centric, static function matching, sandboxed environments, and standalone algorithmic frameworks). MobilePA-Bench uniquely provides a lightweight, high-throughput mobile OS sandbox that maintains live application databases, unifying four core capability dimensions (Basic Tool Use, Sub-agent Collaboration, Memory Usage, and Skill Usage) while remaining optimized for agentic reinforcement learning rollouts. ($\checkmark$ = Supported, $\times$ = Not Supported).

MobilePA-Bench Design and Scope

Defines the experimental setup, variables, and measurement axes for MobilePA‑Bench.

The benchmark evaluates a central mobile planner’s ability to turn a natural‑language request into a verified multi‑step action sequence inside a stateful phone simulation.

The planner is the decision‑making core that observes the user request, the dialogue history, and the currently available actions, then selects the next action to execute.

Step 1: Planner calls `search_user_memory` with query “Alice contact”. Feedback $f_1$ returns Alice’s phone number.

Step 2: Planner invokes skill loader `load_meeting_assistant`, which expands $A_1$ with `create_event` and `add_attendee` tools.

Step 3: Planner calls `create_event` with date “tomorrow” and time “10 am”. Feedback $f_3$ creates a new calendar entry.

Step 4: Planner calls `add_attendee` with the event ID from $f_3$ and Alice’s phone number from $f_1$. Feedback $f_4$ confirms the attendee was added.

Step 5: Planner emits `Finish`. The final state $S_5$ contains a fully populated meeting event.

This trial shows how the planner dynamically expands its action space via a skill loader, then stitches together memory retrieval and tool calls to achieve the end‑to‑end goal.

The benchmark isolates four orthogonal abilities a mobile planner must demonstrate, each exercised by a dedicated set of tools and evaluation buckets.

Define a user intent $q$ and optionally provide a dialogue history $H_0$.

Initialize the active action set $A_0 = R_N(q, H_0; G) \cup L_q$ (top‑$N$ recalled tools plus any skill loaders).

Iteratively:

**Figure 2.** Overview of the MobilePA-Bench evaluation paradigm. The framework formalizes mobile intelligence as a tool-centric orchestration loop. At its core, the central planning agent performs high-level decision making, directly invokes structured business tools, and routes specialized steps to modular Sub-agents, including GUI and visual-processing Sub-agents. It may also retrieve user Memory and load reusable Skills. A stateful Mobile Env, backed by domain databases, executes each action and returns observable Feedback about state changes or system errors, enabling the planner to update its next decision.

Table 2 lists the 13 tool domains (e.g., Audio & Entertainment, Security & Privacy) that populate the unified catalog $G$, providing the concrete API surface the planner can invoke.

Sandbox Architecture and Execution

Describes how benchmark tasks are built, verified, and scored.

The benchmark is assembled from realistic mobile scenarios, turned into executable tasks, and then evaluated with a shared verification engine.

Collect real‑world mobile usage scenarios.

Generate natural‑language queries for each capability dimension.

Annotate each query with a fixed verification bucket (tool, state, or behavior).

Instantiate a Stateful Sandbox instance that mirrors the scenario state.

Execute the planner agent inside the sandbox, recording the interaction trajectory.

Apply the bucket‑specific primary checker and any capability‑specific gates.

Aggregate per‑task outcomes into the overall benchmark score.

A sandbox that persists mutable state across tool calls, so each action can read or write a shared backend just like a real mobile device.

How does a Stateful Sandbox differ from a generic simulator?

Unlike a generic simulator that may ignore persistent side‑effects, the Stateful Sandbox records every tool‑induced mutation in a durable database, so later steps see the exact state resulting from earlier actions.

Three evidence‑aligned query buckets capture the diversity of mobile tasks: deterministic tool calls, state‑change outcomes, or open‑ended agent behavior.

**Figure 3.** Closed-loop execution and verification protocol in MobilePA-Bench. At each step, the Mobile Planner Agent selects an action $a_t$ from the candidate tools $A_t$ using the query and interaction history $H_t$. The stateful mobile sandbox executes the action, updates the environment state, and returns feedback $f_t$ for the next step. After the loop terminates, the resulting tool calls, final state, or interaction trajectory are evaluated by the corresponding verifier.

**Figure 4.** An overview of the stateful simulation sandbox architecture in MobilePA-Bench, illustrated via an `add_contact` tool invocation. The sandbox tightly integrates three layers: (1) a structured Tool Schema defining parameters and domain categories; (2) executable Tool Code handling validation and execution logic; and (3) a persistent Tool Database tracking live state mutations (Contacts) and execution logs (`Operation_logs`). The Tool Executor processes calls, updates the backend state, and returns dynamic, structured execution feedback to the central planner.

**Figure 5.** MobilePA-Bench evaluation framework across three evidence-aligned query buckets. Tasks are executed against a *User Memory* profile and a stateful *Mobile Environment*. Depending on task completion semantics, the fixed primary checker evaluates: (1) **Bucket 1: Tool Call**, using exact tools and arguments; (2) **Bucket 2: State Change**, using the terminal database delta; or (3) **Bucket 3: Agent Behavior**, using reasonable observable behavior. Memory retrieval and gold-skill loading are applied as additional capability-specific gates when required.

**Figure 6.** Hierarchical scenario distribution over a 1,530-query analysis snapshot. The inner circle partitions the scenario-labeled queries into 13 high-level mobile scenarios, while the outer ring details 89 level-2 functional subcategories. This descriptive taxonomy snapshot is separate from the 1,705-task evaluation denominator.

**Figure 7.** Coverage of the 376 Memory Usage tasks along three diagnostic axes: (a) memory reasoning type (single-record grounding, conflicting-record resolution, and multi-record composition); (b) target operation (personalized phone actions and memory addition, replacement, or removal); and (c) application domain.

Verification uses a primary checker $C_{b(q)}$ specific to the bucket, then optionally applies memory‑retrieval and skill‑loading gates.

Step 1: Agent issues

Step 2: Agent issues

Step 3: No further actions; the loop terminates.

Verification: primary checker confirms the exact tool sequence and arguments; memory‑retrieval gate passes because ID 42 matches the gold memory set.

This concrete run shows how a deterministic tool‑call bucket can be fully validated by matching both the ordered API calls and the retrieved memory identifiers.

Capability dimensions guide task synthesis: Basic Tool Use defines deterministic APIs, Sub‑agent Collaboration routes to external agents, Memory Usage forces persistent‑state queries, and Skill Usage requires dynamic skill loading.

Scoring aggregates the four dimensions with fixed weights (0.50, 0.10, 0.20, 0.20) to produce a single overall benchmark score.

Experimental Evaluation

We detail the benchmark configuration and report the main performance results.

Current mobile planners still fail on at least a quarter of tasks, indicating unreliable deployment.

Overall weighted scores show a failure rate of 24.48 % even for the best model.

All evaluations use a fixed candidate recall of N = 15, a maximum of 15 interaction steps per task, and the benchmark’s immutable weighting scheme (50 % Overall, 10 % Basic Tool Use, 20 % Sub‑agent, 20 % Memory).

**Table 3.** Main evaluation results across four MobilePA-Bench capability dimensions. Basic Tool Use reports aggregate accuracy; Sub-agent Collaboration reports routing-and-handoff Joint success; Memory Usage reports end-to-end (E2E) success; and Skill Usage pools Joint success over SOR and MTSR settings. Overall uses the fixed 50/10/20/20 weighting (Section 3.5). Capability and Overall scores are percentages (%); the best capability and Overall scores per column are bolded. Avg. Output Tokens reports the mean visible model output per task, including assistant text and structured tool calls while excluding input context, tool responses, judge outputs, and hidden reasoning.

Performance and Stability Analysis

Benchmark scores are stable and reveal key capability gaps across models.

MobilePA‑Bench measures how well Mobile Planner Agents navigate stateful environments and manage personal memory, moving beyond simple GUI tasks to true multi‑step planning.

Overall benchmark score varies by less than 0.5 percentage points across three independent runs.

Table 4 shows the overall score range 57.22 %–57.63 % (0.41 pp) for Qwen3.6‑27B.

All runs used the same Qwen3.6‑27B model, identical prompts, and the same Stateful Sandbox configuration, ensuring that observed differences stem from stochastic generation rather than experimental drift.

The table presents performance metrics across three experimental runs, categorized by "Basic Tool Use," "Sub-agent," "Memory," "Skills," and an "Overall" score. It includes summary statistics for Mean, Standard Deviation (Std.), and Range for each category.

Basic Tool Use scores range from 68.94 % to 83.85 %, with Claude‑Opus‑5 achieving the highest success (872/1,040). The 13‑model mean is 76.58 %.

Skill Usage peaks at 78.00 % (Claude‑Opus‑5) and averages 66.77 % across models, highlighting that even top planners struggle with consistent skill selection.

GPT‑5.6‑Sol attains 62.68 % overall, with a strong 70.00 % Skill Usage but weak Sub‑agent Collaboration (49.44 %) and Memory (44.15 %), exposing gaps in delegation and personalized retrieval.

Even the strongest planner achieves only 75.52 % overall success, indicating a substantial gap to fully autonomous mobile agents.

Aggregated weighted scores across Basic Tool Use, Sub‑agent Collaboration, Memory, and Skills yield a maximum of 75.52 %.

Three system‑level insights emerge: errors cascade across capability boundaries, strengths are fragmented across models, and planners lack adaptive error recovery, all driving end‑to‑end success below 80 %.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers