FDE-Gym: The Last Mile of AI Delivery
Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen
FDE-Gym evaluates AI agents on their ability to deliver measurable business improvements in realistic, ambiguous customer environments.
How do current AI agents perform on the "last mile" of real-world customer delivery, where success requires navigating stakeholder interactions and business constraints rather than just writing code?
Existing benchmarks evaluate agents on well-defined tasks, but real-world delivery requires engineers to clarify ambiguous requests, discover unstated constraints, and operate within live systems. Agents often build solutions that pass functional tests but fail to improve the customer's actual business metrics. FDE-Gym introduces 95 end-to-end delivery tasks across 18 sectors, where agents must interact with scripted stakeholders and modify production-like systems. The core move is a normalized scoring metric that measures business uplift relative to an expert-authored reference solution, penalized by the cost of stakeholder interactions. The best model configuration achieves a mean score of only 0.56, resolving just 14 of 95 tasks. Even when models successfully retrieve all necessary facts, they struggle to translate that information into effective system improvements.
Paper Primer
The benchmark forces agents to navigate the "last mile" of delivery: identifying the underlying problem when the sponsor's initial request is misdirected or incomplete. Agents operate in a sandbox with standard tools, and their success is determined by the net improvement to a customer's key performance indicator (KPI) after accounting for the "cost" of excessive or careless stakeholder communication.
The method hinges on a four-level task instruction hierarchy (L0–L3) that isolates the difficulty of constraint discovery. By keeping the environment and grader constant while stripping away explicit rules, the benchmark reveals that the primary bottleneck is not just information retrieval, but the engineering judgment required to build the right solution.
Leading models fail to bridge the gap between information discovery and effective delivery.
Even when models obtain every key fact from stakeholders, they achieve a normalized business uplift of 0.8 or above on only 37% of those tasks. The best-performing model (Claude Opus 5.5) resolves only 14 of 95 tasks (15%) at the L3 level.
Weak models frequently produce negative business outcomes.
Models like Claude Haiku 4.5 score below the "do-nothing" baseline on 79 of 95 tasks, meaning the customer would have been better off with no intervention. On 23 tasks, the system delivered by Mistral Small 4 caused more harm than the benefit produced by the reference solution.
Why does this benchmark use a business metric instead of standard functional tests?
Functional tests only measure if code runs, not if it helps. FDE-Gym uses a normalized business uplift metric to ensure that agents are penalized for "literal compliance"—building exactly what was asked for even when it fails to solve the underlying business problem.
What is the "stakeholder contact cost" and why is it included?
It is a penalty applied to the final score for every message, trial run, or day of waiting incurred during the task. It forces agents to balance the need for information against the reality that excessive or careless engagement is a drain on customer resources.
Performance on component benchmarks (like coding or terminal tasks) is a poor predictor of delivery success; closing the "last mile" gap requires agents that can prioritize business outcomes over literal instruction following.
The Last Mile of AI Delivery
Defines the delivery gap and introduces the FDE‑Gym benchmark to measure it.
Existing AI benchmarks test agents on isolated, well‑specified tasks, but real‑world delivery demands that agents clarify ambiguous requirements, interact with live systems, and uncover hidden constraints. To expose this mismatch we present FDE‑Gym, a suite of 95 end‑to‑end delivery tasks drawn from 20 customer scenarios across 18 sectors, scoring agents on normalized business uplift while penalizing stakeholder contact. Evaluations of 13 model configurations reveal a mean best score of 0.56 and only 14 tasks solved, highlighting a substantial gap between coding capability and measurable business impact.
It is the final stage where an AI agent must turn a technically correct solution into a concrete improvement for the customer’s own business metrics.
The agent builds the requested email and chat interface, passes all functional checks, and deploys the system.
Business‑metric evaluation shows zero improvement because the real bottleneck—incorrect downtime credit attribution—remains unaddressed.
Despite meeting the sponsor’s literal request, the normalized uplift score is 0.0, far below the reference solution’s 1.0.
This illustrates the gap between merely coding a requested feature and delivering measurable business uplift; the missing constraint (downtime‑credit rules) is the critical failure point.
The gap between coding capability and business delivery remains the central challenge for forward‑deployed agents.
Defining the Delivery Task
Defines the task environment, instruction levels, and scoring metrics.
The agent operates in a sandbox workspace that contains the task instruction, a copy of the customer’s production system, relevant documents, historical data, and a staff directory. The runtime provides standard Python and SQLite; installing extra packages or accessing the public internet is prohibited, and network calls are limited to the company’s LLM gateway, which incurs usage charges.
Four instruction tiers (L0–L3) vary how much problem context the sponsor reveals, forcing the agent to discover constraints and clarify the goal as the level rises.
U measures how much the delivered system improves the customer’s business metric relative to a reference solution, normalizing raw KPI gains to a 0‑1 scale.
How does Business Uplift differ from simply reporting the raw KPI gain?
Raw KPI gain depends on the scale of the metric and cannot be compared across tasks. Business Uplift normalizes the gain against a reference solution, producing a dimension‑less score where 1 means “as good as the expert” and 0 means “no improvement”.
C quantifies the total charges incurred when the agent contacts stakeholders, expressed in the same normalized units as Business Uplift so the two can be directly subtracted.
Why is Contact Cost expressed in the same normalized units as Business Uplift?
Because the final FDE‑Gym Score is defined as uplift minus contact cost; using a common scale lets the two quantities be combined arithmetically without additional weighting or conversion.
The FDE‑Gym Score $S$ combines the incident‑adjusted uplift $U_{\text{adj}}$ and the contact cost $C$: $S = U_{\text{adj}} - C$. If an incident occurs, $U_{\text{adj}}$ is clipped to non‑positive values; otherwise $S = U - C$. The average score aggregates $S$ over $N$ tasks and $R$ runs per task, flooring $U_{\text{adj}}$ at $-1$ before subtracting $C$ to limit the impact of extreme losses.
The image illustrates a four-stage workflow for an AI agent system: (01) Task Inputs, (02) The Agent, (03) Production Evaluation, and (04) The Score.
Building FDE-Gym
Describes how FDE‑Gym benchmarks are built from real‑world delivery scenarios.
The benchmark suite is assembled by adapting real‑world delivery projects into a set of tasks and variants, then simulating stakeholders and defining scoring anchors.
FDE‑Gym is a collection of end‑to‑end delivery tasks that evaluate an agent’s ability to increase business uplift while respecting realistic stakeholder constraints.
How does FDE‑Gym differ from typical AI benchmarks?
Typical benchmarks isolate a single capability (e.g., language modeling) and use a static test set; FDE‑Gym requires agents to navigate stakeholder dialogues, adapt to variant facts, and optimize business uplift, so success depends on reasoning about constraints rather than memorizing patterns.
Collect 20 real‑world delivery projects from China, Singapore, and the United States (spanning 18 sectors).
Extract the core customer environment and define the original task for each project.
Generate 2–7 variants per scenario by altering key facts (outdated rules, information holders, sponsor intent).
Script over 130 stakeholder responses, fixing knowledge, misconceptions, patience, and contact restrictions for reproducibility.
Define four scoring anchors (reference ≥0.95, do‑nothing = 0, literal ≤0.2, shortcut ≤0.8) and validate against 270+ anti‑gaming strategies.
Iteratively refine tasks through expert review (≈10 000 hours) and publish them on the online platform fde‑gym.com.
Grader evaluates the reference implementation → score 0.96 (passes the reference anchor).
Grader evaluates a literal‑only implementation (follows the instruction verbatim) → score 0.18 (within the literal limit).
Grader evaluates a shortcut that overfits historical data → score 0.73 (below the shortcut ceiling).
These anchors force agents to go beyond surface instructions, preventing easy gaming by overfitting or ignoring hidden constraints.
**Figure 1.** Construction of FDE-Gym. Experts adapt 20 real-world delivery projects from 18 sectors into operational customer environments, alter key facts to create variant tasks, and retain the 95 tasks whose graders have been validated against the predefined scoring anchors.
**Figure 4.** A worked example: the drilling scenario (`oil_npt`), original task.
Experimental Setup
We detail the models, protocol, metrics, and resources used to evaluate FDE‑Gym.
The experimental setup evaluates a diverse set of language models on the FDE‑Gym benchmark using a uniform agent framework. All models share the same five‑tool interface and interact with stakeholders via workspace commands.
The framework equips the agent with five basic tools—execute, read, write, edit, and list—so it can manipulate files and run commands, analogous to a Swiss‑army knife where each blade performs a distinct operation.
How does this five‑tool setup differ from richer agent toolkits that include search or web‑access capabilities?
Those toolkits add extra primitives that let the agent retrieve external information, whereas our minimal set forces the model to solve tasks using only the provided files and commands, isolating its reasoning ability.
Select a task instruction level (L0–L3) and the corresponding task variant.
Run the model on the selected task using the minimal agent framework.
Repeat the run three times and average the scores to reduce stochastic variance.
Terminate the run when the agent submits a delivery or hits any of the limits: 200 turns, 150 minutes, or USD 8 model usage cost.
Record the FDE‑Gym score, normalized business uplift, stakeholder contact cost, and average model usage cost per run.
The run proceeds for 180 turns, reaching 140 minutes and incurring \$7.5 in model usage cost before the agent submits its delivery.
This run yields an FDE‑Gym score of 0.54, a normalized business uplift of 0.62, and a stakeholder contact cost of 0.18.
A second repetition stops earlier at 150 minutes and \$6.9 cost, producing a score of 0.51.
The turn and time limits often bind before the cost ceiling, showing that computational budget is the primary constraint in practice.
We report the mean FDE‑Gym score per instruction level with 95 % confidence intervals obtained by bootstrapping over tasks, as well as normalized business uplift before stakeholder contact cost and the average model usage cost per run.
For context we compare FDE‑Gym scores to task success rates on Terminal‑Bench 2.1, Vibe Code Bench, and Finance Agent, using a single independent source that runs each model on the same external benchmarks. We multiply FDE‑Gym scores by 100 to place them alongside percentage success rates, noting that the metrics are not directly equivalent.
The harness that creates task workspaces, simulates stakeholders, and grades deliveries, along with the experiment runner, are released as open source (github.com/deeplethe/fde-gym, version 1.1.0). The 20 original tasks and their graders are available on request, with three tasks publicly released; the 75 variants and reference solutions remain held out.
Main Results and Performance
Detailed quantitative comparison of model performance and business uplift.
The benchmark measures end‑to‑end business uplift for last‑mile delivery, evaluating how agents translate task instructions into real‑world value.
Claude Opus 5.5 leads the benchmark, beating the runner‑up Claude Sonnet 5.5 by $+0.07$ in L3 score.
Opus 5.5 achieves an L3 score of $0.56$, while Sonnet 5.5 scores $0.49$ on the full 95‑task set.
All models were evaluated on the same 95 tasks under instruction level L3, with stakeholder contact cost subtracted to obtain the normalized business uplift.
**Figure 2.** Model performance across task instruction levels. (a) Mean L3 score for each model on all 95 tasks, on a normalized scale where 0 represents the do-nothing baseline and 1 represents the reference solution. Solid bars show scores after deducting stakeholder contact cost; lighter bars show normalized business uplift before this deduction. (b) Scores as progressively less information is provided in the task instruction. L0-L2 are evaluated on the 20 original tasks and L3 on all 95 tasks.
**Figure 7.** Mean score at L3 on each scenario, for each model; scenarios are ordered by the best model’s score, and the highest-scoring cell in each row is outlined. Color runs from red (below the do-nothing baseline) to blue (at the reference solution).
Models struggle to achieve high business uplift even when instructions are clear.
Fact Retrieval vs. Effective Delivery
Key ablations reveal fact retrieval, cost, and variability limits model delivery.
Leading models retrieve 72–85% of key stakeholder facts yet deliver only 0.60–0.70 of the reference business uplift.
At L3 the top three models ask 10.0–12.3 questions per task, obtain 72–85% of facts, and their normalized uplift stays in the 0.60–0.70 range; even when all facts are obtained, average uplift is 0.66 and reaches ≥0.8 in only 37% of cases.
Stakeholder contact cost erodes a large fraction of the uplift delivered by top models.
For Claude Opus 5.5, contact cost consumes 0.14 of the 0.70 uplift; for GPT‑6 Astra it consumes 0.22 of the 0.60 uplift, leaving substantially less net benefit.
Weak models fall below the do‑nothing baseline on the majority of tasks (60–83%).
Claude Opus 5.5 scores below zero on 5 tasks, Claude Sonnet 5.5 on 7, Claude Haiku 4.5 on 79, and GPT‑6 Luna on 63; before deducting contact cost, Mistral Small 4 is below baseline on 51 tasks.
Model performance varies dramatically across scenarios; five scenarios have no model reaching a 0.5 average score.
Across 20 scenarios, the best scores for the hardest scenarios (`news_incident`, `revenue_dispute`) are only 0.18 and 0.27; the median range of leading models within a scenario spans 0.34–0.44, exceeding 0.5 in 6–8 scenarios.
Higher spending and effort do not guarantee higher scores.
GPT‑6.1 Sol spends \$0.24 and 25 turns per run, scoring 0.37, whereas Gemini 3.8 Flash spends \$1.54 and 121 turns, scoring only 0.19; Qwen 3.8 Flash runs 169 turns and 110 min, scoring 0.07.
**Figure 5.** (a) Share of key stakeholder facts obtained against normalized business uplift, at L3. Leading models learn most of the facts and deliver 0.60–0.70 of the reference uplift. (b) Mean model usage cost per run against score at L3.
**Figure 6.** Distribution of each model's 95 tasks at L3 by score band. Scores of 0.2 or below correspond to the cap on literal compliance with the sponsor's task instruction.
Comparison with Prior Benchmarks
Benchmark comparison reveals how FDE‑Gym scores differ from prior task benchmarks.
Figure 8 shows how five AI models perform on three established benchmarks and on the end‑to‑end FDE‑Gym score, highlighting the gap between task success rates and business uplift.
Measures whether an agent can finish a self‑contained task without external assistance, reporting a simple success‑rate percentage.
Evaluates an agent’s ability to build a complete web application from a specification, testing end‑to‑end coding, integration, and deployment.
Simulates a financial analyst’s research workflow, requiring the agent to retrieve market data, perform calculations, and generate a concise report.
**Figure 8.** Model performance on Terminal-Bench 2.1, Vibe Code Bench, Finance Agent, and FDE-Gym.
Table 4 enumerates prior benchmarks by the delivery capability they assess—ranging from framing to outcome—and clarifies what FDE‑Gym contributes beyond each. For example, existing “Implementation” benchmarks measure code correctness, whereas FDE‑Gym adds the downstream business impact and stakeholder‑contact cost.
Related Work
We position our benchmark against prior work, highlighting gaps and design inspirations.
Table 4 groups prior benchmarks by the delivery stage they evaluate. Existing suites each isolate a single capability, and none combine the stages into a causal chain where a misread problem renders good code useless and the loss appears on a business metric.
The closest prior work, $\tau$‑bench [9], frames agent construction as a task and achieves 23.9 % pass rate versus 82.2 % for an expert reference; our best model falls similarly short of expert performance. Other benchmarks such as “Trains but Doesn’t Learn” [10], XYEval [11], $\tau$‑2‑bench [34], UnderSpecBench [35], and Implicit Intelligence [23] explore instruction misdirection, underspecification, and failure modes that align with our task‑instruction levels. Methodological warnings from works on LLM‑played users [14], single‑run reliability [36, 37], admission anchors [38], long‑horizon evaluations [39], and Collaborative Gym [40] informed our design choices, leading to scripted stakeholders, multiple runs per task, and do‑nothing baselines.
Limitations and Conclusion
Summarizes benchmark findings and stresses the need for full‑lifecycle evaluation.
The paper argues that existing AI benchmarks focus on isolated tasks, while real‑world delivery requires end‑to‑end business uplift; FDE‑Gym evaluates that full delivery lifecycle.
The benchmark draws scenarios from many sectors, emphasizing regulated agent workflows. Although the companies are fictional, the scenarios are adapted from real deliveries, so some training‑data overlap cannot be ruled out.
FDE‑Gym scores agents by the customer’s business improvement rather than mere system delivery, revealing that the last‑mile gap remains substantial. The best configuration still leaves considerable room for improvement.
Leading models suffer little loss when sponsor requests are misdirected and capture most stakeholder facts, but they lose most of their performance when converting those facts into an effective system and when incurring stakeholder contact costs.
Weaker models perform worse; on most tasks the customer would have been better served by no delivery at all, highlighting the value of expert judgment in delivery.
The need for benchmarks that capture the full delivery lifecycle.