BizAgentBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business

Enhan Zhao, Yongqi Tong, Xuejun Yu, Hang Wang, Pan Wang, Yichao Wang, Di He, Jianshe Li, Wei Wu

BizAgentBench evaluates long-horizon business agents on intermediate process compliance, not just final task success.

How do current long-horizon agents perform when evaluated not just on final task success, but on their adherence to complex, multi-step business process requirements?

Existing benchmarks for long-horizon agents prioritize final task completion, often ignoring whether the agent violated critical business constraints during execution. In high-stakes domains like payments, an unauthorized action remains a failure even if the final outcome appears correct. BizAgentBench introduces 50 multi-turn tasks grounded in real-world payment workflows, requiring agents to navigate outdated policies, conflicting rules, and shared resource constraints. The environment uses deterministic checks to validate both the final goal and the entire execution history against business requirements. Frontier models struggle with this rigor: even the top-performing model achieves a strict success rate of only 64.0%, with many models masking intermediate compliance failures behind successful final outcomes.

Paper Primer

The benchmark forces agents to reconcile fragmented, distributed knowledge—such as current versus historical policy documents—while managing stateful dependencies like shared cash reserves and timing deadlines. The core move is a deterministic evaluator that treats execution as a sequence of states, rejecting any path that breaches a constraint at any point, regardless of whether the final goal is reached.

Outcome-based evaluation significantly overstates agent reliability in complex business environments.

Models frequently achieve the requested goal while violating intermediate process or protection requirements, such as crossing cash-reserve floors or executing actions before obtaining necessary approvals. For DeepSeek-V4.1-Flash, 5 out of 150 runs reached the goal but failed the full strict success criteria due to intermediate violations.

Frontier models show substantial performance gaps when held to strict compliance standards.

Claude Opus 5 and GPT-6 Sol achieve strict success rates of 64.0% and 61.3%, respectively, leaving significant room for improvement in reliable, constraint-compliant execution. Lower-performing models like Qwen3.5-9B fail to achieve any strict successes across 150 runs, with 83.3% of runs incurring intermediate failures.

Why is final task success insufficient for evaluating business agents?

In business operations, intermediate steps—such as obtaining specific approvals or maintaining minimum reserve balances—are often legally or operationally mandatory. Violating these constraints constitutes a failure even if the final objective is eventually met, as the process itself is non-compliant.

How does this benchmark handle the ambiguity of real-world business rules?

The benchmark includes both current and historical guidance with explicit effective periods and scopes. Agents must actively identify which rules are authoritative for the specific business event, rather than simply retrieving information, making rule selection a core part of the task.

Introduction: The Gap in Agent Evaluation

BizAgentBench exposes the gap between final goal success and intermediate process compliance.

Long‑horizon agents often reach the desired end result while slipping on intermediate business rules; BizAgentBench measures this gap by checking both final goal achievement and process validity.

Prior long‑horizon agent benchmarks primarily report final task success, providing little insight into whether agents obey intermediate business constraints. In high‑stakes domains such as payments, violating a policy mid‑execution can incur losses or regulatory penalties even if the final outcome is correct. This mismatch motivates a more comprehensive evaluation that captures both ends of the execution spectrum.

BizAgentBench is a multi‑turn benchmark built on real‑world procedures from a leading payments company. Its executable environment integrates 789 knowledge documents and 601 discoverable business tools, all governed by consistent policies, states, and tool behavior. The benchmark comprises 50 tasks that feature resource constraints, temporal dependencies, prior commitments, and conflicting or outdated guidance.

Experimental results reveal substantial gaps: Claude Opus 5 and GPT‑6 Sol achieve strict success rates of only 64.0 % and 61.3 %, respectively. Qwen 3.8‑27B reaches 42.0 % goal attainment but drops to 32.7 % strict success, illustrating how outcome‑only metrics can hide invalid executions. DeepSeek‑V4.1‑Flash averages 191.3 tool calls per run, roughly 70 % more than the reference executions, exposing significant execution overhead.

Early FAQ‑based systems matched queries to curated Q&A pairs, limiting multi‑turn contextual support. Subsequent deep neural dialogue systems extended capabilities to multi‑turn task resolution but remained constrained by predefined dialogue acts and short‑horizon state tracking. Large language models now enable autonomous agents that interpret policies, plan multi‑step actions, and orchestrate external tools, raising new reliability challenges.

Agents must maintain execution validity at critical intermediate steps and correctly apply the authoritative business rules that are scattered across heterogeneous, sometimes outdated sources. Business policies may conflict across jurisdictions, products, or effective periods, requiring agents to discern which guidance is current and applicable. Existing benchmarks that focus on final success miss these essential compliance and reasoning aspects.

BizAgentBench tasks combine tool use with multi‑turn customer dialogue; a user simulator supplies customer‑specific information and responds to approval requests throughout execution. Current and historical guidance with explicit scopes and effective periods are provided, forcing agents to select the correct instruction for each request. The environment retains action consequences, so later decisions must respect prior commitments.

Even advanced models struggle on BizAgentBench: GPT‑6 Sol succeeds on only 61.3 % of tasks, and only 40 % of tasks are solved consistently across three attempts. DeepSeek‑V4.1‑Flash runs meet goal and cash‑reserve thresholds yet breach the reserve at intermediate steps, showing how outcome‑only evaluation can mask violations. Smaller models such as Qwen 3.5‑9B generate many tool calls, most of which error, and none achieve full task objectives.

The key insight is that achieving the final goal does not guarantee a valid execution; agents must also satisfy intermediate process and protection constraints.

Related Work

Survey of prior agent benchmarks and business‑oriented evaluation frameworks.

Long‑horizon agent benchmarks assess reasoning and tool use in interactive settings. Examples include web and desktop suites for shopping and browsing, SWE‑bench for repository‑level issue resolution, and Terminal‑Bench for technical tasks evaluated via final‑state checks. Variants such as SWE‑Milestone and Long‑Horizon‑Terminal‑Bench add intermediate progress metrics.

Business‑oriented benchmarks introduce operational rules and organizational context. The 𝜏 family studies policy‑governed interaction, while enterprise suites span browser‑based knowledge work, compositional workflows, and CRM operations. AutomationBench evaluates final application states without prescribing tool‑call sequences, mirroring BizAgentBench’s flexible exploration with enforced authorization checks.

Recent work on long‑horizon planning and adaptation provides the foundations for BizAgentBench. Tool‑use learning, reflection, skill reuse, and search support multistep reasoning, while clarification research identifies when extra information is beneficial. BizAgentBench extends this line by requiring replanning when intermediate actions affect later obligations.

Baseline agents that attempt to solve extended tasks by chaining tool calls, maintaining state, and optionally revisiting earlier decisions when new information arrives.

The BizAgentBench Framework

Describes the benchmark’s experimental setup: tasks, environment, and evaluation criteria.

BizAgentBench evaluates agents on real‑world business workflows by combining a customer request, an initial business state, and explicit acceptance criteria.

Goal validity checks that the final outcome satisfies the request, while process validity ensures every intermediate step obeys the business rules and protection constraints.

Sample a task i from the 50 benchmark tasks.

Load an independent copy of the reconstructed environment containing the 789 knowledge documents, 601 business tools, and the user‑simulator’s 10 tools.

Provide the agent with the Standard Operating Procedure (SOP) that encodes the workflow’s business rules.

Run the agent; it may query documents, invoke tools, and interact with the user simulator to obtain private customer information.

Record the execution history $\tau = (s_0, e_1, s_1, \dots, e_T, s_T)$ as the sequence of states $s_t$ and events $e_t$.

Evaluate the run using the three checks: $G_i(\tau)$ (goal), $P_i(\tau)$ (process), and $F_i(\tau)$ (protection). The composite condition Strict Success = $G_i \land P_i \land F_i$ determines whether the task is counted as a success.

Aggregate metrics across runs: overall success rate, process‑validity rate, and protection‑compliance rate.

Initial balance $B_0 = 60$ meets the reserve requirement.

Agent executes the payment, reducing the balance to $B_1 = 0$.

Process validity check $P$ fails because the reserve drops below $20$ after the payment, even though the goal check $G$ succeeds (the payment is completed).

A later refund restores the balance to $B_2 = 20$, satisfying the reserve again, but the earlier breach remains recorded as a process‑validity violation.

Process validity captures intermediate safety constraints that the goal check alone would miss; an agent can appear successful while violating critical business rules.

**Figure 1.** Overview of BizAgentBench construction. Business workflows are normalized into an executable environment; tasks combine consequential dependencies with separate goal, process, and protection criteria, and are validated using reference executions and targeted counterexamples.

Task Construction and SOPs

We detail how multi‑turn business tasks are built, from SOP extraction to validation.

Data construction proceeds through five stages, each adding concrete business constraints and interaction structure.

Task construction turns abstract business policies into concrete, multi‑turn interactions that an agent can execute and be evaluated on.

Step 1: Agent queries the reserve balance → receives $120$.

Step 2: Agent requests approval for the $60$ fee → SOP checks that $120 \geq 100$, approves.

Step 3: Agent charges the $60$ fee → reserve balance updates to $60$.

Step 4: Final check verifies that the reserve never fell below the $100$ threshold during execution → passes.

This example shows how the process‑validity check enforces a minimum reserve throughout the task, not just at the end.

How does Task Construction differ from simply writing a goal description?

Task construction embeds the goal inside a full business workflow, adding intermediate checks (process validity) and protected resources (protection criteria) that an agent must satisfy before reaching the goal.

Organize source business documents into structured SOPs (Section 4.1).

Build executable tasks by aligning SOP steps with synthetic records, tools, and temporal constraints (Section 4.2).

Configure a LLM‑based user simulator with a customer scenario and interaction protocol (Section 4.3).

Construct Golden traces that demonstrate at least one valid execution satisfying goal, process, and protection checks (Section 4.4).

Validate tasks by testing alternative executions, intentional violations, and expert review (Section 4.5).

**Figure 4.** A complete task example in BizAgentBench. In Q03, the customer asks to reconcile Acacia’s September receipts and complete Durian Trading’s account application. (a) The SOP connects two rule-applicability decisions—resolving superseded buyer instructions and selecting evidence valid at review time—to reconciliation, funding and institutional approval. Closing the file unlocks a \$60 reserve but freezes its allocations. (b) The agent must discover relevant documents and tools, obtain customer facts and request-specific approvals from the user simulator, and execute the linked workflows. Critical intermediate steps require correct reconciliation before closure and explicit consent to the revised \$60 total fee cap and the actual request terms before execution; final application approval alone cannot establish these requirements. Document labels report body word counts; amounts are USD and $t$ denotes logical minutes.

**Input:** Business documents $\mathcal{D}$ **Output:** SOPs $\widetilde{\mathcal{S}}$ 1: $C \leftarrow$ extract workflow candidates from $\mathcal{D}$, retaining source evidence links 2: $\mathcal{P} \leftarrow$ group related workflow candidates into hierarchical workflow plans with evidence links 3: $\mathcal{S} \leftarrow$ generate SOPs from $\mathcal{P}$ and their source evidence 4: $\widetilde{\mathcal{S}} \leftarrow \emptyset$ 5: **for** each SOP $s \in \mathcal{S}$ **do** 6: $\quad$ Specify steps, decisions, next-step links, and return links in $s$ 7: $\quad$ Review $s$ against its source evidence via the workflow plan and candidates 8: $\quad$ Exclude content with unresolved conflicts 9: $\quad$ Attach reviewed knowledge to $s$ and its steps, retaining applicability conditions and source evidence 10: $\quad \tilde{s} \leftarrow$ rewrite sensitive business content in $s$ for simulation 11: $\quad \widetilde{\mathcal{S}} \leftarrow \widetilde{\mathcal{S}} \cup \{\tilde{s}\}$ 12: **end for** 13: **return** $\widetilde{\mathcal{S}}$

Evaluation Metrics and Setup

We detail the task suite, model runs, and metrics used to evaluate BizAgentBench.

The benchmark scores agents on three orthogonal dimensions—goal achievement, process validity, and protection compliance—and combines them into a single success flag.

Select a model and a task from the 50‑task suite.

Initialize an independent copy of the benchmark database and load the public documents and tools.

Run the agent with the DeepSeek‑V4.1‑Flash user simulator under the default condition (no golden trace, no unrevealed events).

Enforce no cumulative limits on generated tokens, tool calls, or scheduler steps; set per‑response token limits high enough to avoid truncation.

Apply a wall‑clock timeout and a tool‑error threshold to each run.

If the model reports a context‑window overflow, replace the oldest quarter of eligible knowledge‑search and document‑read tool responses in the model‑facing context.

For models evaluated three times per task, repeat steps 3‑6 three times and average the single‑run success values.

Perform information ablations by disabling selected sources (e.g., documents or tools) while keeping the same tasks, database, and acceptance criteria.

Record infrastructure failures (e.g., timeouts) and simulator protocol failures separately from metric outcomes.

Step 1: Model receives the initial dialogue (5 turns) and the full set of 12 tool responses stored so far.

Step 2: After generating 1800 tokens, the context window (2048 tokens) is full; the overflow policy removes the oldest 3 tool responses (25 % of 12).

Step 3: The remaining 9 tool responses stay in the model‑facing context; the full execution record (including the discarded responses) is preserved for post‑hoc analysis.

Step 4: The run completes within the 300‑second timeout, producing a Goal Success = true, Process Validity = true, Protection Compliance = true, yielding Strict Success = true for this trial.

This concrete trace shows how the overflow policy preserves the execution record while keeping the model’s context size manageable, and how Strict Success is computed from the three binary outcomes.

Overall Performance and Process Validity

Performance results reveal wide gaps in strict success and highlight reliability challenges across models.

Recall: Long‑horizon agents often succeed by violating intermediate process and protection constraints; BizAgentBench measures both goal attainment and execution validity.

Strict success rates vary widely, with Claude Opus 5 achieving 64.0 % and Qwen3.5‑9B achieving 0 %.

Table 1 reports Claude Opus 5 at 64.0 % strict success and Qwen3.5‑9B at 0 % across 150 runs.

**Table 1.** Goal (G), process (P), protection (F), intermediate failure, and strict success (%). Intermediate failure means that a process or protection requirement failed. Claude Opus 5 and Muse Spark 1.3 have one run per task; the other models have three.

**Figure 2.** Reliability and resource use for models with three runs per task. Left: pass^k averages, over all k-run subsets, the fraction of tasks that succeed in every selected run. Right: mean assistant turns, distinct documents read, and assistant tool calls per run.

Behavioral Analysis and Ablations

Ablation analysis reveals how information, tool‑call limits, and simulator choice affect strict success.

Long‑horizon agents often reach their final goal by violating intermediate business‑process and protection constraints; BizAgentBench measures this gap via strict‑success, which requires goal achievement, process validity, and protection compliance.

Providing full customer disclosure raises strict success to 70.0%, a 26.7% absolute gain over the default condition.

Table 2 shows strict success of 43.3% (default), 50.0% (+Gold‑doc), and 70.0% (+Full disclosure).

**Table 2.** DeepSeek-V4.1-Flash strict success under three information conditions on 50 tasks.

Restricting tool calls to 1.2× the Golden‑trace count collapses strict success to 0.7%.

Table 3 reports strict success of 0.7% at the 1.2× cap, far below the uncapped 43.3%.

**Table 3.** DeepSeek-V4.1-Flash strict success with tool-call caps (three-run mean); $R_i$ is the Golden call count.

Changing the user‑simulator model shifts strict success by at most 3.3 percentage points.

Figure 3 shows DeepSeek‑V4.1‑Flash at 32.7% versus Qwen3.8‑27B at 36.0%.

**Figure 3.** User-simulator model comparison using Qwen3.8-27B as the agent. Bars show the mean strict success rate over three runs; DeepSeek-V4.1-Flash is the default simulator.

**Table 4.** Execution-trace statistics across stronger and weaker models. Counts are means per run; repeated database reads have identical tool names, arguments, and responses. The last column counts runs reaching the goal but failing process or protection requirements.

Supplementary Figures

Appendix details release routes and a full task execution trace.

This appendix expands the “A Complete Task Example” introduced earlier. It describes the three release routes shown in Table 5, the full Execution Trace 01 that the benchmark uses as a Golden trace, and the user request that drives the agent’s actions. The narrative also lists the grouped steps and concrete actions the agent must follow.

**Table 5.** Three release routes for the same $60 reserve. The \$50 review fee is additional.

Execution Trace 01 walks through the agent’s timeline from $t=0$ to $t=18$, showing how it clarifies scope, reads documents, proposes remittance advice, reconciles invoices, selects evidence, and finally authorizes the release. The trace highlights that the agent must respect the customer’s explicit cap increase from $50$ to $60$ and obtain approval before $t=19$.

Technical Appendices

Key statistics and execution analyses that support the benchmark evaluation.

This appendix compiles the quantitative tables and detailed execution‑history analyses that underpin the benchmark results presented earlier.

The table presents statistics for "Knowledge base: per document" and "Golden traces: per task" metrics, including Mean, Median, and Range values for document length, tool calls, documents read, business records read, and executed business requests.

The table presents statistics on various metrics related to agent and user-simulator interactions, including tool calls, database queries, and document reads. It lists the Mean, Median, and 90th percentile values for each metric.

**Table 8.** DeepSeek-V4.1-Flash tool use per run in the default condition.

Section D.1 analyses execution length and tool usage, noting that identical repeated database reads occur less frequently in successful runs (13.7 vs 18.5), suggesting that unnecessary re‑queries are a failure signal.

Section D.2 evaluates user‑simulator turn‑taking reliability; protocol violations drop from 8 % with the original prompt to 0 % with the revised prompt, supporting the adoption of the revised prompt for benchmark evaluation.

Section D.3 presents a concrete Q34 scenario where the agent achieves the goal while respecting process constraints, demonstrating that $G\!=\!1$, $P\!=\!0$ can still yield a valid execution if the final state satisfies the business objective.

Section D.4 discusses the scope of business‑document retrieval, emphasizing that agents must correctly filter documents by relevance to avoid spurious tool calls that inflate the execution trace.

Section C.1 illustrates a limited net‑delivery guarantee (Q24), where fee budgeting forces the agent to allocate service resources across multiple obligations, highlighting the interplay between cost constraints and process validity.

Section C.2 details payment reconciliation and delivery windows, showing that timely coordination of information gathering and service rights is essential for achieving the commercial objective despite correct final balances.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers