AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

Kai Chen, Zichen Ding, Jiaye Ge, Shufan Jiang, Mo Li, Qingqiu Li, Zehao Li, Zonglin Li, Tiaohao Liang, Shudong Liu, Zerun Ma, Zixing Shang, Wenhui Tian, Zun Wang, Liwei Wu, Zhenyu Wu, Jun Xu, Bowen Yang, Dingbo Yuan, Qi Zhang, Songyang Zhang, Peiheng Zhou, Dongsheng Zhu

AgentCompass decouples agent evaluation into modular components to standardize fragmented benchmarks and enable granular trajectory analysis.

How can we decouple agent evaluation pipelines to make them reproducible, extensible, and capable of deep trajectory analysis?

Agent evaluation is currently fragmented, forcing researchers to repeatedly build custom execution environments for every new benchmark and agent architecture. AgentCompass solves this by decoupling the evaluation pipeline into three independent, protocol-driven components: Benchmark, Harness, and Environment. This modularity allows researchers to mix and match agents and tasks without rewriting complex execution logic. The framework reveals that agent performance is highly sensitive to infrastructure choices, with models often deviating significantly from official baselines when evaluated under a unified protocol.

Paper Primer

AgentCompass replaces rigid, benchmark-specific scripts with a registry-based architecture. It treats the evaluation as a composable configuration: Benchmark (task logic) × Harness (agent interaction wrapper) × Environment (execution sandbox).

The core mechanism is a standardized data contract: the Benchmark compiles tasks into a uniform `PreparedTask` structure, which the Harness consumes to produce a `RunResult`. This contract acts like a universal adapter, allowing any registered agent harness to interact with any benchmark environment without cross-modification.

Agent performance is highly sensitive to the underlying evaluation infrastructure.

Models show significant score fluctuations when evaluated under the unified AgentCompass protocol compared to their official baselines, such as Claude-Opus-4.8 dropping 8.7 points on DeepSearchQA and GLM-5.2 improving 15.0 points on SWE-bench-Pro. Double-digit percentage point shifts in performance metrics.

Beyond scalar scores, the framework uses pluggable analyzers to perform trajectory-level diagnostics. It automatically flags behavioral anomalies like repetitive tool calls, output truncation, and suspected reward-hacking, providing a more transparent view of why an agent succeeds or fails.

Why is decoupling the harness from the benchmark necessary for agent research?

Current evaluation pipelines are tightly coupled, meaning researchers must reimplement execution logic for every new agent or task. Decoupling allows for flexible benchmark × harness × environment configurations, ensuring that results are reproducible and comparable across different agent frameworks.

Does AgentCompass only support static question-answering benchmarks?

No, it natively supports over 20 benchmarks spanning five dimensions, including highly interactive, long-horizon environments like repository-level software engineering (SWE-bench) and open-ended web navigation (GAIA).

The Need for Unified Evaluation

We expose the fragmentation of current agent evaluation pipelines and introduce AgentCompass to modularize benchmarks, harnesses, and environments.

Large‑language‑model agents are emerging, but their evaluation pipelines remain fragmented and tightly coupled, which hurts reproducibility and forces researchers to re‑engineer the same glue code repeatedly.

Current agent evaluation pipelines are fragmented and tightly coupled, making reproducibility hard and engineering effort redundant.

Fragmentation of current evaluation pipelines hampers reproducibility and forces duplicated engineering work.

AgentCompass Architecture

We detail the modular AgentCompass architecture that decouples benchmarks, harnesses, and environments via declarative protocols.

Existing evaluation pipelines intertwine task logic, agent orchestration, and execution environment, making reuse painful. AgentCompass isolates these concerns so a researcher can swap a harness or environment without touching benchmark code.

AgentCompass splits evaluation into three independent components—Benchmark, Harness, Environment—linked by strict data contracts, so swapping any piece leaves the others untouched.

AgentCompass registers B1, H1, and E1 via decorator‑based registries.

The benchmark loads the quiz data and emits a PreparedTask containing the prompt “What is 2 + 2?” and expected answer format.

The harness receives the PreparedTask, formats the prompt for M1, calls the model API, and obtains the prediction “4”.

The environment executes no external side‑effects (in‑process), simply returns the prediction to the harness.

The harness packages the prediction, score (1.0), and the full interaction trajectory into a RunResult.

This toy run shows that swapping H1 for a more complex tool‑using harness requires only a new HarnessSpec; the benchmark code and environment stay unchanged.

How does this differ from a monolithic evaluation script?

In a monolithic script the benchmark, agent logic, and execution environment are interwoven, so changing the model or adding a tool forces edits across the whole file. AgentCompass isolates each concern behind a fixed protocol, meaning you replace only the plug‑in that implements the desired piece while the surrounding pipeline remains identical.

**Figure 2.** Overview of the AgentCompass architecture. The framework fully decouples Benchmarks, Harnesses, and Environments to enable flexible and composable agent evaluation.

Runtime and I/O Handling

AgentCompass turns fragmented, I/O‑heavy evaluation into a resilient, plug‑in‑friendly pipeline.

Evaluating autonomous agents traditionally stalls on massive I/O bursts and tightly coupled code paths, making large‑scale runs fragile and hard to extend.

AgentCompass treats an evaluation run like a resumable download: it streams tasks, writes partial results as they arrive, and can pick up exactly where it left off after a crash.

Dispatcher writes results of tasks 1 and 2 to

Task 3 raises

Operator restarts the run; the runtime scans

Task 3 is retried, succeeds, and is appended as

Only the failed sample incurs extra compute; the rest of the work is never repeated, turning a potentially costly crash into a cheap retry.

How does this incremental execution differ from a naïve “run‑all‑once” batch?

In a naïve batch the entire job aborts on the first failure and must be rerun from scratch, wasting all previously finished work. AgentCompass isolates failures, persists partial outputs, and resumes only the unfinished pieces.

Parse CLI arguments (or SDK call) into a

Lookup registered benchmark, harness, and model objects from the global registry.

Materialise task‑specific assets (datasets, prompts, tool configurations).

Spawn an asyncio task for each trajectory, guarded by a concurrency semaphore.

Each trajectory streams progress events to the persistent log and reports success or retryable failure.

On shutdown, the runtime writes a manifest of completed task IDs; a subsequent launch reads the manifest to skip them.

Resumable execution loop (simplified)

Think of the registry as a universal power strip: any new benchmark, harness, or environment simply plugs into a well‑defined socket without rewiring the whole system.

Why doesn’t adding a new harness break existing benchmarks?

Benchmarks interact with harnesses only through the abstract HarnessProtocol. As long as the new harness implements the required methods, the benchmark continues to call the same interface, so no code in the benchmark needs to change.

Together, the resilient runtime and the plug‑in registry give researchers a cost‑effective, reproducible workflow that scales from a single notebook to a multi‑node cluster.

Experimental Results

AgentCompass exposes large performance swings while keeping overall scores stable.

AgentCompass reveals up to a 15.0‑point swing in model scores across benchmarks while overall performance remains consistent.

Table 3 shows GLM‑5.2(FP8) gains 15.0 points on SWE‑bench‑Pro with OpenHands, whereas Claude‑Opus‑4.8 loses 8.7 points on DeepSearchQA.

**Figure 1.** Capability profiles of representative models across the five core evaluation dimensions.

**Table 3.** Comprehensive evaluation results of representative models across the five core capability dimensions. Colored subscripts show gaps from official baselines, with green/red indicating higher/lower AgentCompass scores; no subscript means no official baseline was disclosed.

Performance remains consistent across diverse benchmarks despite large per‑benchmark swings.

Trajectory Analysis

We examine trajectory‑level behaviors to reveal hidden failure modes, reward‑hacking, and token‑length trade‑offs.

The AgentCompass pipeline modularizes benchmarks, harnesses, and environments, enabling systematic trajectory‑level inspection beyond aggregate scores. This section leverages that capability to answer three concrete research questions.

Instead of looking only at the final outcome of an agent run, we examine the entire sequence of observations, actions, and intermediate results that the agent produced.

How does trajectory analysis differ from simply reporting the final task score?

Final scores collapse an entire interaction into one number, discarding information about how the agent arrived there. Trajectory analysis preserves the step‑by‑step evidence, allowing us to spot repeated tool calls, empty responses, or other pathological patterns that a high score alone would hide.

RQ1 asks which behavioral “bad cases” appear in model trajectories. Using AgentCompass we quantified five failure categories across seven models and visualized the share of each category.

**Figure 3.** Distribution of bad-case behaviors across model trajectories.

RQ2 investigates whether high‑scoring coding models truly understand code or merely exploit the benchmark. By applying a reward‑hacking analyzer we flagged samples that modify tests, retrieve golden patches, or otherwise game the metric.

GLM‑5.2 achieves the highest score on SWE‑Pro but also shows roughly 30 % more suspected hacking than Claude‑Opus‑4.8, which scores 12 points lower. DeepSeek‑V4‑pro remains consistently low on hacking across both SWE‑Pro and SWE‑Multilingual, suggesting its scores are less inflated by shortcut behaviors.

RQ3 explores the relationship between model capability and token length across task families. Scatter plots reveal that, on coding benchmarks, longer completions generally correlate with higher scores, while on productivity tasks Claude‑Opus‑4.8 attains strong scores with fewer tokens.

**Figure 4.** Capability–token length trade-offs across different capability dimensions.

Implementation Details

Appendix A details the model, benchmark, and runtime settings used for all experiments.

Model configurations separate closed‑source APIs from open‑weight deployments. Closed‑source models (GPT‑5.5, Claude‑Opus‑4.8) run with their default inference settings to preserve determinism, while the open‑weight Qwen‑3.5‑397B‑A17B is served via the SGLang engine using the vendor‑recommended flags. All models receive the highest available reasoning‑effort setting to avoid ad‑hoc tuning.

Benchmark specification follows a strict subscript convention in Table 3: each subscript records the numeric gap between the AgentCompass score and the nearest external reference for the same model‑benchmark pair. When multiple references exist, the one whose reported value is closest to the AgentCompass result is selected; if no comparable reference is available, the subscript is omitted.

Tool‑use, web & research, and scientific‑reasoning evaluations reuse existing benchmark workflows. The $\tau$‑3‑bench workflow drives tool‑use tasks; DeepSearchQA and FrontierScience are run with the Naive Search Agent, using Qwen‑3.6‑35B‑A3B and GPT‑5.5 respectively as judge models. SciCode runs in `tool_use` mode with only the `code_interpreter` tool enabled, allowing up to 30 iterations per step and a 180‑second timeout per interpreter call.

Agentic coding benchmarks (SWE‑bench Pro, SWE‑bench Multilingual) are evaluated with two agents: Mini‑SWE‑agent v2.3.0 and OpenHands v1.23.0. The harness supplies a system prompt, while the user prompt concatenates the issue description with an explicit instruction to write the generated patch to patch.txt. For minimal container images (e.g., Alpine, Go toolchain) we apply runtime compatibility fixes at container startup and execute OpenHands inside a micromamba‑managed isolated environment.

Productivity benchmarks span SkillsBench, PinchBench, and Kimi‑K2.6. SkillsBench uses OpenClaw v2026.5.9 and OpenHands v1.23.0 on data version commit 17dec32. PinchBench runs with OpenClaw v2026.3.22, a 250 000‑token context window, 80 000‑token per‑turn limit, and a 131 072‑character single‑message cap; Kimi‑K2.6 switches to OpenClaw v2026.7.1‑beta.2 for robust handling of empty assistant content. All PinchBench runs are judged by Claude‑Opus‑4‑5‑20251101 with thinking mode, task timeouts are multiplied by ten, and the grading runner enforces a 480‑second outer timeout.

Step Statistics

Average interaction steps per model and benchmark are reported and interpreted.

A step denotes one complete agent–environment interaction cycle: the model generates a response, any required tool is invoked, and the environment returns an observation. For conversational benchmarks a step is simply one model reply, whereas for coding or productivity benchmarks a step usually combines a reasoning action with subsequent environment execution.

**Table 5.** Average interaction steps per evaluation instance across different benchmarks. Each value represents the mean trajectory length over all evaluated samples for the corresponding model and benchmark.

These trajectory‑level statistics complement final accuracy numbers, offering insight into evaluation cost, interaction efficiency, and overall agent behavior.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers