AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents

Kunlun Zhu, Xuyan Ye, Zhiguang Han, Yuchen Zhao, Bingxuan Li, Weijia Zhang, Muxin Tian, Xiangru Tang, Pan Lu, James Zou, Jiaxuan You, Heng Ji

AgentDebugX provides a closed-loop framework for localizing root causes in failed LLM agent executions and automating their repair.

How can we automate the localization and recovery of LLM agent failures when the error surface is disconnected from the root cause?

LLM agents often fail due to errors that occur many steps before the final symptom appears, making it difficult for developers to identify which specific decision caused the collapse. AgentDebugX addresses this by organizing debugging into a closed loop: Detect, Attribute, Recover, and Rerun. Its core mechanism, DeepDebug, uses multi-turn diagnostic passes—combining global context with structure-guided investigation—to pinpoint the responsible agent and step. On the Who&When benchmark, this approach achieves 28.8% strict attribution accuracy compared to 21.7% for the strongest single-pass baseline. On GAIA, it repairs 13 of 73 failed tasks in a single rerun, significantly outperforming decoupled self-correction methods.

Paper Primer

The framework treats debugging as an iterative pipeline rather than a static trace replay. It converts framework-specific events into a portable trajectory representation, allowing the same diagnostic logic to apply regardless of whether the agent was built with LangGraph, CrewAI, or raw ReAct.

DeepDebug functions as a multi-turn diagnostic agent: it performs a global read to establish context, uses structure-guided probes to isolate the failure, and cross-examines conflicting candidates to adjudicate the root cause. This process is like a detective who first surveys the entire crime scene, then re-examines specific rooms to resolve inconsistencies between witness accounts.

DeepDebug significantly improves root-cause localization accuracy.

Strict agent-and-step attribution on the Who&When benchmark. 28.8% accuracy vs. 21.7% for the best single-pass baseline.

Localized diagnosis leads to more effective automated repairs.

Repairs on failed GAIA validation tasks after a single rerun. 13/73 tasks repaired, compared to 4–6 for decoupled self-correction baselines.

Why is a multi-turn diagnostic agent necessary instead of just asking the model to find the error in one pass?

Single-pass attribution often suffers from "loudest symptom" bias, where the model anchors on the final visible failure rather than the earlier, decisive mistake. DeepDebug’s multi-turn structure allows it to separate the task objective from locally unusual but valid actions.

Does this framework require modifying the agent's original code to enable debugging?

No. AgentDebugX uses runtime adapters or offline log importers to capture execution traces, keeping the debugging logic entirely separate from the agent's implementation.

By connecting failure attribution to a policy-gated rerun loop, AgentDebugX transforms agent debugging from a manual, trace-reading task into a structured engineering workflow that accumulates reusable failure knowledge.

The Challenge of Agent Debugging

LLM agents fail silently; we need a workflow that links symptoms to their true causes.

LLM agents are increasingly deployed in complex, long‑horizon tasks that involve external tools, persistent memory, and coordination across multiple components. When these agents err, the visible symptom—such as a failed tool call or an implausible final answer—typically appears many steps after the actual mistake, making manual debugging a costly, trial‑and‑error process.

Existing observability platforms give developers a detailed execution trace, yet they stop short of answering “which earlier decision caused the failure?” Likewise, failure taxonomies and attribution benchmarks enumerate error modes but remain separate from any actionable debugging workflow, and self‑correction techniques assume the error location is already known.

The core obstacle is the disconnect between the step that triggers a visible failure and the earlier step that actually introduced the error.

The disconnect between a failure’s observable symptom and its true root cause is the primary barrier to efficient LLM‑agent debugging.

AgentDebugX System Architecture

AgentDebugX unifies detection, attribution, recovery, and rerun into a closed‑loop debugging workflow.

Existing observability stacks (LangSmith, Langfuse, Phoenix) record raw traces, while prior debugging suites (AgentDebug, MAST, Who&When, TRAIL, AgenTracer, AgentDiagnose) isolate failure symptoms but stop short of automated repair.

AgentDebugX stitches together four deterministic stages—Detect, Attribute, Recover, and Rerun—into a single loop that turns a raw execution trace into a repaired, comparable rerun.

Detect applies rule packs, flags E₂ as a malformed tool call.

Attribute runs a heuristic that checks the immediate predecessor of E₂; it ranks E₁ as the most likely cause with confidence 0.78.

Recover proposes a retry directive: “re‑invoke tool call with corrected arguments derived from E₁’s inputs.”

Rerun executes the directive, producing a new event E₂′ that succeeds, followed by the original E₃–E₅.

Comparison shows the new trajectory’s final output matches the goal, so the branch is saved as a resolved case.

The example shows how a single rule‑based detection can cascade through attribution and recovery to automatically fix a failure without human intervention.

**Figure 1.** Overview of AgentDebugX. AgentDebugX forms a closed debugging loop: Detect identifies step-level errors, Attribute locates their root causes, Recover proposes ranked fixes, and Rerun regenerates the trajectory around the failure point. Hard cases are escalated to DeepDebug, a multi-round diagnosis agent that produces an auditable root-cause report with evidence and a recommended fix. AgentDebugX supports local and web UIs, reusable error sharing, baseline evaluation, and broad agent-framework compatibility.

**Table 1.** Capability coverage versus prior work. • = first-class, ◦ = partial, – = absent. **Port. schema:** a portable, framework-agnostic trace format enabling re-analysis, comparison, and sharing (not a vendor span format). **Taxonomy:** a labelled failure-mode vocabulary. **Attribution:** localizing the responsible step/agent, not just the crash. **Recovery:** turning a diagnosis into a rerun-able fix. **Error hub:** a shareable cross-team corpus of diagnosed failures.

**Figure 2.** The AgentDebugX console on a failed run, as a four-step workflow: (1) select a stored trace in the run navigator; (2) jump from the visible failure to the attributed responsible event; (3) read the failure mode, evidence, and proposed fix in the diagnosis panel; (4) create a policy-gated rerun branch and compare it with the original timeline. All artifacts stay in the local store unless a bundle is explicitly exported.

How does AgentDebugX differ from earlier toolkits like AgentDiagnose?

AgentDiagnose scores trajectories but never ties a symptom to a concrete retry. AgentDebugX closes the loop: after attribution it automatically generates a repair directive and executes a new trajectory, turning diagnosis into verified recovery.

The closed‑loop nature ensures that every detected failure can be immediately acted upon, turning a static post‑mortem into an iterative, self‑healing process.

By exposing a portable trace format and a reusable loop, AgentDebugX lets developers plug in any LLM or tool stack while retaining a uniform debugging experience.

Root-Cause Attribution Mechanism

DeepDebug attributes failures via a multi‑turn read‑only trace analysis and extends the failure taxonomy.

Single‑pass attribution either focuses on the loudest downstream symptom or loses sight of the overall objective, leaving many failures unexplained.

DeepDebug runs a read‑only, four‑stage pass over the captured AgentTrajectory, narrowing the search from the whole trace to a single decisive step.

Stage 1 scans steps 1‑8, notes the objective “deliver package”, and flags step 8 as the loudest symptom.

Stage 2 bisects the range: first examines steps 1‑4, finds step 3 unusually high latency, proposes step 3 as a second candidate.

Stage 3 sees the candidates differ (8 vs 3) and compares their contexts: step 3’s input mismatch explains the downstream latency spike at step 8.

Stage 4 reports: “Agent A, step 3 caused the deadlock; fix by adding a timeout.”

The four‑stage flow turns an O(N) search into a constant‑time adjudication between two concrete hypotheses.

How does DeepDebug differ from a naïve single‑pass traceback that simply points to the last error?

Single‑pass traceback stops at the first abnormal output, ignoring whether that output was caused by an earlier hidden mis‑decision. DeepDebug adds a second, structure‑aware pass and a cross‑examination step, guaranteeing that the selected step is the earliest causal ancestor rather than the most visible symptom.

The system maintains a curated set of failure modes but can grow it automatically when novel patterns appear.

Run A logs “Agent X waiting on Agent Y”.

Run B logs “Agent Y waiting on Agent Z”.

Run C logs “Agent Z waiting on Agent X”.

The inducer clusters these three logs (identical lexical pattern) and proposes a new mode “multi‑agent deadlock”.

A maintainer reviews, sees it differs from “lost‑handoff” (which involves a single missed hand‑off), and adds the new mode to the taxonomy.

The clustering‑plus‑human‑approval pipeline lets the taxonomy evolve without manual enumeration of every long‑tail failure.

Why isn’t the existing “lost‑handoff” category sufficient for the deadlock examples?

“Lost‑handoff” describes a single agent failing to receive a hand‑off, leading to a pause. The deadlock pattern involves a circular wait among three agents, a distinct causal structure that requires a separate category to trigger the correct remediation logic.

Localization Performance

DeepDebug’s multi‑turn diagnosis outperforms single‑strategy baselines on attribution and repair.

We evaluate the two core duties of a deployed debugger in order: first localize the root‑cause step, then turn that diagnosis into a repair.

The benchmark measures two things for each failed trace: which agent caused the failure (“who”) and the exact step at which the mistake occurred (“when”).

DeepDebug attains 28.8 % strict “who + when” accuracy, a 7.1‑point gain over the best single‑strategy baseline (21.7 %).

Table 2 shows DeepDebug’s 28.8 % versus 21.7 % for the strongest single‑strategy (All‑at‑Once) on the full Who&When benchmark with $qwen3.5\!-\!9b$.

**Figure 3.** Who&When accuracy by trace length (n=184; qwen3.5-9b). Left: responsible-agent accuracy (who). Right: strict joint accuracy requiring both the responsible agent and the exact mistake step (who+when) to be correct.

**Table 2.** Failure attribution on the full Who&When benchmark (n=184; qwen3.5-9b). “Agent” measures responsible-agent accuracy (who); “Step” measures mistake-step localization (when) under exact and ±1 criteria; “A+S” requires both agent and step to be correct.

Localization accuracy improves markedly on longer traces, confirming that DeepDebug’s multi‑turn interrogation is most valuable when the failure is buried deep in the execution.

End-to-End Recovery Results

We showcase real‑world uses and quantify DeepDebug’s recovery gains on the GAIA benchmark.

AgentDebugX closes the loop on failure localization: once a step is pinpointed, the system can automatically retry with a targeted fix. This mirrors the paper’s core premise that attributing the root cause enables concrete recovery.

GAIA evaluates agents on multi‑step tasks that require planning, tool use, and reasoning, reporting success rates per difficulty tier.

Beyond numbers, AgentDebugX supports a full debugging workflow: capture a failed run, localize the responsible step, present a root‑cause explanation, propose a verified repair, and optionally rerun under policy control. The repaired trajectory is stored in the Error Hub for future regression testing or incident response.

System and Prompt Details

Provides the exact step and surrounding context needed for root‑cause attribution.

The diagnostic routine expects a concrete execution step from the AgentTrajectory log together with the preceding and following entries that establish its causal neighborhood.

Extracting the target step and its context from an AgentTrajectory log.

Evaluation Protocol

Evaluation protocol, trace schema, and attribution metrics for AgentDebugX.

The evaluation follows the Who&When benchmark in its full‑trace setting, providing a gold‑standard annotation for each of the 184 runs.

An AgentTrajectory is the complete execution log of an autonomous agent, ordered by time.

The diagnostic layers operate on compact, JSON‑constrained prompts; the most load‑bearing prompts are reproduced below.

The LLM Judge receives a single run’s events and must label each failed step with a permitted failure‑mode code.

This prompt asks the model to pinpoint the single event that caused a failure, or return null if the trajectory succeeded.

When two candidate error steps are presented, the model must choose the decisive one whose correction would most likely avert the failure.

GAIA validation (165 tasks across three difficulty levels) is used for end‑to‑end recovery scoring with the official question scorer.

Attribution follows the Who&When “ground‑truth” protocol: each method receives the task’s reference answer but never sees the gold agent or step; decoding runs at temperature 0 with model‑side thinking disabled, token limits of 4,096 (512 for arbitration), and agent names are normalized before comparison.

Metrics reported include responsible‑agent accuracy, exact and ±1 step‑localization accuracy, and the joint agent‑and‑step (A+S) score.

GAIA recovery runs a vanilla Open‑Deep‑Research agent (qwen3.5‑9b policy, Gemini‑2.5‑flash search) on all 165 tasks, isolates the 73 failures, diagnoses them with the LLM judge (or DeepDebug), and then reruns the failures under one of four strategies: AgentDebugX’s native DeepDebug directive, Reflexion, CRITIC, or AutoManual.

During recovery the diagnostic memory and Error Hub are cleared to keep the comparison clean; reruns use temperature 0, a fixed step budget, and the same search and tool stack as the baseline.

Localizer design ablation measured on a stratified 42‑trace Who&When sample shows that replacing the structure‑guided turn with a second global search drops strict accuracy from 0.310 to 0.262 on gpt‑5.4‑mini, while the shipped pairing lifts agent accuracy to 0.524 vs 0.429 for a single reading.

Implementation and API

Implementation details and known limitations of the AgentDebugX package.

AgentDebugX is distributed as a lightweight MIT‑licensed Python package installable via pip install agentdebugx and imported as agentdebug.

Typical usage of the AgentDebug context manager.

Recorded traces are written to an append‑only .jsonl file or an SQLite database, enabling offline replay and incremental analysis.

AgentDebugX can ingest events from any source—live agents, log files, or external tools—by normalizing them to a single schema.

The bundled CLI mirrors the library workflow: agentdebug ingest to load traces, agentdebug diagnose to run attribution, agentdebug inspect to view reports, agentdebug act to trigger verified recoveries, and agentdebug serve for an interactive console.

Our evaluation focuses on the correctness of automatic attribution and the effectiveness of suggested recoveries; it does not measure developer debugging time or the ergonomics of the console UI.

Who&When’s reference‑answer protocol and DeepDebug’s model‑dependent gains mean that extra attribution calls are not uniformly beneficial across all LLMs.

The GAIA experiment runs a single policy model on a pre‑selected failed subset and compares full retry recipes, so it cannot isolate the pure impact of attribution.

Error Hub retrieval and taxonomy induction are implemented but remain unevaluated; taxonomy induction also requires human acceptance before it can be applied.

The scrubber removes known credential and PII patterns from event inputs but does not redact arbitrary user‑generated content, leaving a residual privacy risk.

Finally, any recovery action produced by AgentDebugX is gated behind explicit human approval, preventing autonomous execution of potentially unsafe fixes.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers