UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks

Zhekai Chen, Chengqi Duan, Kaiyue Sun, Bohao Li, Yuqing Wang, Manyuan Zhang, Xihui Liu

UniClawBench evaluates proactive agents in live, multi-turn environments using a capability-driven taxonomy.

How can we reliably evaluate proactive agents on real-world tasks that require multi-turn interaction and tool use?

Existing agent benchmarks rely on static sandboxes and single-turn evaluations, failing to capture the iterative, unpredictable nature of real-world personal assistance. UniClawBench uses a three-role closed-loop system—executor, hidden supervisor, and user simulator—to evaluate agents in live Docker containers without leaking grading criteria. Even top-tier models struggle, with overall pass rates remaining below 50% and a persistent "halfway failure" gap between partial progress and task completion.

Paper Primer

The benchmark decomposes agent performance into five foundational capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. By isolating these dimensions, the system identifies specific bottlenecks—such as memory loss or poor visual grounding—rather than just reporting a generic failure.

The evaluation architecture acts as an information firewall: the hidden supervisor holds the ground truth and rubrics, while the user simulator generates natural feedback based only on the agent's visible trajectory. This ensures the agent receives realistic, multi-turn guidance without the user simulator inadvertently revealing the evaluation criteria.

Current proactive agents exhibit a severe "halfway failure" phenomenon.

Models consistently achieve high checkpoint-based scores for intermediate steps but fail to complete the entire task, indicating a lack of reliability in long-horizon execution. Pass rates for state-of-the-art models remain strictly below 50% across the benchmark.

Framework architecture significantly impacts agent performance.

Comparing OpenClaw, EDICT, and Nanobot shows that centralized frameworks like OpenClaw outperform others by minimizing information loss during multi-turn trajectories. The performance gap between frameworks widens as base-model capabilities increase.

Why is a capability-driven taxonomy better than the standard scenario-based approach?

Scenario-based benchmarks (e.g., "office" or "travel") conflate multiple abilities, making it impossible to tell if a failure stems from poor visual perception, tool usage, or reasoning. UniClawBench isolates these bottlenecks to pinpoint the specific cause of failure.

How does the benchmark handle the instability of live, real-world environments?

Instead of relying on fixed, pre-recorded answers that quickly become outdated, the benchmark uses fine-grained, step-by-step completion checkpoints that allow the supervisor to evaluate reasoning and execution evidence dynamically.

Researchers should shift focus from generic end-to-end success rates to diagnosing specific capability bottlenecks, as current agents are reasonably capable of local tool use but fundamentally struggle with long-horizon memory and cross-application coordination.

The Need for Proactive Evaluation

We expose why static benchmarks miss proactive agents and propose a dynamic evaluation framework.

Large language models have evolved into proactive agents that can control everyday tools and assist users across modalities. Existing benchmarks, however, are built on sandboxed environments and single‑turn evaluations, which cannot capture the continuous, interactive nature of real‑world assistance.

These benchmarks also organize tasks by scenario (e.g., “office”, “research”), mixing distinct capabilities and obscuring the true source of failures. Consequently, when a model fails an “office” task, it is unclear whether the bottleneck lies in visual perception, long‑context reasoning, or tool usage.

A proactive agent continuously monitors its environment and initiates actions toward user goals without waiting for explicit commands.

Two fundamental challenges arise when building a benchmark for such agents. First, live environments lack a stable ground truth: answers that are correct today may become outdated tomorrow. Second, simulating realistic user feedback without exposing grading criteria is non‑trivial, as the simulator must react to the agent’s behavior while remaining information‑isolated.

The benchmark expects the answer “\$199”.

The page now shows “\$179”, causing a mismatch.

The agent receives a penalty despite having followed the correct procedure.

Static, pre‑recorded answers cannot accommodate dynamic price fluctuations, exposing a core limitation of existing benchmarks.

The shift from static benchmarks to dynamic, tool‑using evaluation is essential for measuring true proactive‑agent capability.

The UniClawBench Framework

Overview of UniClawBench’s taxonomy, benchmark composition, and three‑role closed‑loop evaluation framework.

UniClawBench is introduced as a concrete experimental platform that moves beyond static benchmarks to evaluate proactive agents in realistic, multi‑turn settings.

UniClawBench is a curated suite of 400 bilingual real‑world tasks, each built around a single dominant capability so that the bottleneck for an agent is clear.

The taxonomy assigns each task to the single capability whose absence would prevent successful completion, providing a diagnostic lens for failures.

**Figure 1.** **Overview of UniClawBench.** UniClawBench consists of 400 bilingual real-world tasks spanning 5 core capabilities: multimodal, long-context, skill usage, exploration, and cross-platform. We propose a three-role closed-loop evaluation framework, where an executor agent performs tasks in real environments, a supervisor evaluates trajectories and artifacts using hidden rubrics, and a user simulator provides natural feedback based on executor's performance and supervisor's signal, enabling multi-turn interaction. We conduct two sets of experiments to evaluate cross-model and cross-framework performance. The capability-level results reveal that framework choice exerts a stronger influence than model choice.

**Figure 2.** Three-role Closed-loop Evaluation Strategy. A task is executed by an agent (Executor) within a real-world environment using only public inputs (e.g., tools, files, and web access). The resulting observable trajectory and produced artifacts are assessed by a hidden Supervisor, which applies fine-grained checkpoint-based rubrics and private references to compute a structured evaluation state (e.g., pass / fail / continue) and score. To prevent leakage of evaluation criteria, only a coarse progress signal is exposed through an information firewall to a User Simulator, which generates natural-language feedback based solely on visible evidence. The sanitized feedback is returned to the executor for iterative refinement, forming a multi-turn closed loop. The example illustrates a multimodal task (Beef Wellington cooking with fridge grounding), where the agent must produce screenshots and grounded outputs, emphasizing strict evidence-based evaluation.

The evaluation separates three agents—Executor, hidden Supervisor, and public User Simulator—so the Executor can improve over multiple turns without ever seeing the grading rubric.

Turn 1: Executor reads the instruction, opens the fridge image, and saves a screenshot of the raw ingredients.

Supervisor evaluates the trajectory, sees the screenshot, and returns status = continue with a hidden rubric indicating missing pastry steps.

User Simulator receives the “continue” signal and the visible trajectory, then replies “The dish looks good but you’re missing the pastry layer—please add it.”

Turn 2: Executor adds the pastry layer, saves a new screenshot, and submits the final artifact.

Supervisor now finds the pastry correctly applied, matches the hidden reference, and returns status = pass.

This single trial shows how the Executor can iteratively refine its output using only sanitized feedback, while the hidden grading criteria remain inaccessible.

All experiments run inside fresh Docker containers with 2 GB memory each; the host machines are Intel Core i7‑13700 with 16 GB RAM. Global timeouts are 30 min (45 min for long‑context tasks) and per‑turn limits are 20 min (30 min for long‑context).

Evaluating Benchmark Reliability

We assess how well automated scores match human expert judgments.

Automated pass/fail decisions align closely with human majority judgments.

Aggregated judgments from 50 randomly sampled trajectories evaluated independently by three experts.

High correlation shows the automatic score reliably reflects nuanced human assessments, enabling large‑scale evaluation without costly human labeling.

We test whether the automatic evaluation pipeline produces scores that match human expert judgments on the same tasks.

**Table 1.** Average quantitative evaluation across English and Chinese subsets of UniClawBench. To isolate inherent model capabilities from framework-level design choices, all 10 models are evaluated as the executor under the same OpenClaw framework. Performance is measured by Pass Rate (PR) and checkpoint-based Average Score (AS) across the five foundational capability categories.

Framework Performance and Token Usage

Cross‑framework results reveal how architecture drives success and token efficiency.

OpenClaw outperforms the other frameworks in pass rate, and the advantage grows with stronger base models.

Table 2 shows OpenClaw’s pass rate exceeds EDICT by up to 15 % and Nanobot by up to 12 % for the largest model.

Framework Architecture specifies how the Executor, Supervisor, and User Simulator are wired together to produce a coherent execution trajectory.

How does this framework differ from a simple linear pipeline where the Executor runs independently?

In a linear pipeline the Executor proceeds without any supervisory feedback, so intermediate results are never re‑evaluated. Here the Supervisor closes the loop: it observes each tool outcome, queries the User Simulator, and can redirect the Executor, preserving the full context for the next step.

**Figure 4.** Token Usage and Performance Progression by Cycle. (a) and (b) display the token usage on the OpenClaw system, categorised by model and task dimension, (c) demonstrates the performance progression by cycle motivated by the user follow-ups.

End-to-End Execution Case Study

A full end‑to‑end run that recovers from an initial shortfall to achieve a passing score.

The central premise is that proactive agents need dynamic, multi‑turn evaluation; this case study shows a full end‑to‑end run that recovers from an initial shortfall to achieve a passing score.

The task asks the Executor to retrieve five Library of Congress records with clear item‑level rights and accessible images, explicitly forbidding reliance on the first search results or thumbnails.

The hidden evaluation rule defines a weighted rubric (completion, breadth, evidence, trap handling, reproducibility) and applies several score caps for common failure modes.

Read the agent‑browser skill description.

Research the LOC API to locate items with clear rights and images.

Open a filtered LOC search page.

Attempt a DOM snapshot and encounter a Cloudflare challenge.

Wait and retry, then switch to direct LOC JSON API via curl.

Perform HEAD requests to the LOC IIIF service to verify full‑image availability.

Read item‑level JSON for each candidate and normalize metadata.

Deduplicate by image stem and write the required output files.

The Supervisor reviewed the artifacts and, noting missing internal consistency, returned a continue verdict, indicating the attempt was recoverable.

Based on the continue verdict, the User Simulator sent a targeted follow‑up asking the Executor to re‑validate items, fix exclusion inconsistencies, and address the identified rights and duplicate issues.

Load the user‑simulator message and read the current exclusions file.

Launch a re‑validation script that fetches official item JSON for every candidate.

Detect a missing rights advisory for the Koussevitzky record.

Inspect the pipeline and note the Feraille‑o item is a digitized card catalog.

Update the fetch script to incorporate the new checks.

Run the corrected pipeline to generate a fresh selection of five records.

Produce an updated exclusions file listing seven items with concrete rejection reasons.

Write the corrected `loc_rights_images.json` and accompanying notes.

Verify the new outputs by displaying their contents.

Confirm that all artifacts now satisfy the rubric.

The Supervisor gave a pass verdict with a high score, confirming that the final artifacts fully meet the rubric and are auditable.

**Table 3.** Cycle 1 first eight executor turns. The model recovers from the Cloudflare block by switching probe strategy without giving up.

**Table 4.** Cycle 2 executor turns. The model goes beyond the user's stated complaints, finding two additional grounded issues (Koussevitzky rights field, Feraille-o item type) and rebuilding the pipeline rather than patching the saved files in place.

Three key properties of the closed‑loop design are highlighted: (1) the Supervisor flags recoverable failures instead of outright rejection, (2) the information firewall limits hidden evaluation data to the Supervisor, and (3) multi‑turn recovery enables the Executor to rebuild its pipeline and achieve a passing score.

The framework can improve agent reliability and safety by exposing real‑world failure modes, but it also risks over‑optimizing for benchmark performance and accelerating powerful autonomous systems without sufficient oversight.

Task Configuration and Format

Defines the YAML contract and hidden eval rule that structure UniClawBench tasks.

The three‑role closed‑loop framework consists of an Executor agent that carries out the task, a hidden Supervisor that judges outcomes against a private rubric, and a User Simulator that mimics interactive user behavior.

```yaml task_id: task_NNN_short_slug category: <suite_name> agent_sys: openclaw agent_id: main model: claude-opus-4.6 timeout_seconds: 1200 max_total_seconds: 1800 success_threshold: 0.90 task: | <natural-language request shown to the executor> task_snapshot: | <parallel-safe variant> references: - references/eval_rule.md - references/ground_truth.json - ... sources: - <relative_path> skills: - <skill_id> services: - name: <service> path: <subdir> start: bash install.sh oneshot: true pre_exec: - ops/populate.py pre_exec_parallel_safe: true codex: max_user_followups: 2 user_simulator: policy: | <override of the default user-simulator policy> supervisor: instructions: | <task-specific addendum to the default supervisor instructions> ```

Key fields include `task_id` (unique identifier), `timeout_seconds` (per‑turn limit), and `success_threshold` (minimum supervisor score to pass).

The codex block configures the closed‑loop interaction: `max_user_followups` caps user turns, `user_simulator`.policy can override default behavior, and supervisor.instructions adds task‑specific grading hints.

The hidden `eval_rule`.md follows a nine‑section schema: sections 1–4 define the contract, 5–6 list weighted checkpoints, 7–9 govern lifecycle, caps, and dynamic content handling.

Representative Task Examples

Representative task prompts illustrate the executor’s required actions across skill, exploration, reasoning, multimodal, and cross‑platform suites.

This appendix enumerates concrete task prompts that the executor will encounter, grouped by functional suite.

Read the SKILL.md entry for the Mermaid diagramming skill.

Locate the source excerpt at /`tmp_workspace`/clawbench/sources/`alphabet_def14a_excerpt`.md.

Parse the hierarchy and committee relationships from the excerpt.

Generate a Mermaid script describing nodes, solid arrows for board links, dotted arrows for advisory links, and subgraphs for committees.

Write the script to /`tmp_workspace`/results/org.mmd and render an SVG to /`tmp_workspace`/results/org.svg.

Export a CSV `governance_matrix` with columns person, `source_role`, `governance_group_or_board`, committee, `committee_role`, `edge_type`, `source_quote`.

This task showcases multi‑step tool orchestration: skill discovery, data extraction, format conversion, and multi‑output generation.

Read the GitHub skill description to determine required API endpoints.

Use the $GITHUB_TOKEN environment variable to call the GitHub REST API for the repository clawbench‑activity‑fixture.

Filter events between 2026‑04‑10 and 2026‑04‑25, distinguishing automation‑generated entries.

Aggregate counts per contributor: merged PRs, closed‑unmerged PRs, closed issues, commit count.

Create a markdown digest grouped by contributor and add a Follow‑up Flags section for risky items.

Write a highlights list of the five most recent merged PRs or closed issues, respecting tie‑breaking rules.

Save the digest to /`tmp_workspace`/results/digest.md and the full index to `activity_index`.csv.

This example illustrates dynamic data retrieval, conditional attribution handling, and structured reporting.

Load the legacy .zshrc from /`tmp_workspace`/sources/`existing_zshrc`.

Audit for duplicate PATH entries, PROMPT definitions, precmd overrides, and plugin load order issues.

Construct a cleaned .zshrc preserving aliases, plugins, PATH prepends, conda hook, and nvm sourcing.

Design a concise prompt showing path and git status while omitting username, hostname, and heavy glyphs.

Run the new .zshrc in eight scenarios (plain, git clean, git dirty, detached HEAD, rebase/merge, deep path, Python venv, non‑interactive) and log verification results.

Ensure startup time does not noticeably increase.

Save the final .zshrc and all audit, patch notes, verification logs to the results directory.

This task demonstrates careful stateful refactoring with exhaustive functional validation.

Enumerate each package’s metadata files (package.json, LICENSE, README, source headers) under /`tmp_workspace`/sources/`license_audit`/.

Apply policy rules: allow MIT/Apache‑2.0/BSD/ISC, allow MPL‑2.0 with notice, flag GPL/AGPL/unknown or mismatches.

Detect traps such as registry‑license mismatch, dual licenses, generated headers, vendored third‑party code, and COPYING.LESSER vs GPL confusion.

Record for each package the `final_license`, `policy_decision`, `primary_evidence`, and any conflicts.

Generate a file‑level matrix summarizing `source_path`, `claimed_license`, weight, `kept_or_ignored`, and reason.

Write method metadata describing `precedence_rules`, `package_scope_rules`, and `policy_mapping`.

Save all JSON artifacts and the audit script to the results directory.

This example highlights systematic license compliance checking against corporate policy.

Watch the referenced YouTube video and draft a markdown tutorial with screenshots.

Launch Slack, log in with the provided $EMAIL_ADDRESS, and resolve any verification codes from Outlook.

If a CAPTCHA appears, complete it manually before proceeding.

Configure or reset the OpenClaw bot entry and connect the local OpenClaw installation.

Send a test message to the bot and capture screenshots of the successful reply.

Save the tutorial, configuration notes, and any blocking screenshots to /`tmp_workspace`/results/`slack_openclaw_setup`.md.

This task combines multimedia consumption, interactive authentication, and cross‑application configuration.

Search United nonstop flights on Google Flights, Kayak, and Momondo for the specified dates and time windows.

Record the most suitable result from each platform with screenshots.

Search the three listed hotels on Booking.com and official sites, capture room type, price, taxes, cancellation policy, rating, and address.

Search Enterprise and Turo for rental cars, record the cheapest qualifying options.

Check restaurant availability for the specified venues and times.

Assemble a markdown itinerary and budget tables for a budget‑saving and a comfortable option, respecting the \$3,200 cap.

Save all screenshots and the final plan to /`tmp_workspace`/results/.

This example illustrates coordinated multi‑source data gathering and cost‑constrained planning.

Download the arXiv paper and locate the middle “Aspect Ratio” line chart in Figure 5.

Extract the original data points (or approximate them) for the three model‑size curves.

Write a Python script that plots the data with a logarithmic x‑axis, appropriate legends, and styling matching the original.

Iterate: generate the chart, compare visually to the source, and adjust scaling or colors.

Save the final script and PNG image, and write brief notes documenting the figure source and revisions.

This task demonstrates faithful visual reproduction of published research figures.

Open each artwork’s webpage, take a screenshot of the image and metadata, and save them.

Parse the artwork metadata (title, artist, date, medium, dimensions) into a CSV file.

Read `my_room_style`.json to obtain room color palette and lighting conditions.

Analyze each painting’s color palette, composition, brushwork, and atmosphere against the room style.

Select the painting that best matches the room, and write a justification while noting why the other two were rejected.

Save the CSV, visual analysis markdown, and best‑choice markdown to the results directory.

This example combines visual perception, metadata handling, and aesthetic decision‑making.

Open the PDF at /`tmp_workspace`/clawbench/sources/`rag_survey_pack`/`rag_survey`.pdf in a PDF viewer.

Launch Zotero, create a new item with full bibliographic fields, and attach the PDF.

Export the item to a BibTeX file with citation key gao2024retrieval.

Open Obsidian, create a vault at /`tmp_workspace`/results/`rag_survey_obsidian_vault`/.

Create a literature note linking to the Zotero citation and summarizing the paper’s objective, method, dataset, metrics, findings, limitations, and RAG evaluation relevance.

Save the note, the exported BibTeX, and screenshots of the Zotero record and Obsidian note.

This task showcases cross‑application bibliographic management and knowledge‑base integration.

Visit the official Rust Foundation event pages to confirm title, dates, start/end times, venue, and registration link.

Open GNOME Calendar (or equivalent) and create a new event titled “Rust Global: Tokyo”.

Set the start time to 30 minutes before the official start and the end time to the official end.

Enter the venue, registration link, and cost requirements in the description.

Save the event and capture a screenshot showing it in the calendar view.

If the calendar supports export, also save an .ics file.

Write an audit JSON recording the application used, event details, and evidence file paths.

Summarize the event in a markdown file using a graphical text editor.

This example emphasizes reliable cross‑platform state synchronization and evidence preservation.

System and Role Prompts

Defines the three prompts that drive each evaluation cycle.

Each evaluation cycle is orchestrated by three role prompts: a session wrapper, a supervisor, and a user simulator. The prompts are reproduced verbatim below and contain placeholders that are filled in for every attempt.

Session Wrapper Prompt – common header for supervisor and user simulator.

Supervisor Prompt – judges the executor’s attempt and enforces anti‑drift rules.

User Simulator Prompt – generates the next user turn when the verdict is continue.

```json user_handoff = { "verdict": "continue", "attempt_state": "complete_but_failed", "recoverable": true, "score": 0.65 } ```

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers