OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners
Peisong Wang, Feixue Liu, Yize Meng, Bowen Liu, Ruoyu Chen, Gaoyang Dai, Xing He, Aiqian Li, Fengchi Yuan, Aochuan Chen, Chenyi Zi, Yuhan Li, Hongchuan Zeng, Yimo Yan, Nuo Chen, Jie Fu, Zuo-Jun Max Shen, Yong-Hong KUO, Jia Li
OR-License evaluates LLM agents on real-world operations research tasks using client-simulated information acquisition.
Can LLM agents solve real-world operations research problems directly from raw client materials, or do they require expert-preprocessed inputs?
Operations research (OR) requires experts to transform incomplete business materials into executable decision processes, yet existing benchmarks rely on self-contained, synthetic problems that bypass the critical work of requirement elicitation and problem formulation. The authors introduce OR-License, an end-to-end benchmark of 31 industrial OR instances derived from \$3.68 million in paid engagements. It uses ORSim, a controlled client simulator that forces agents to acquire business facts through targeted queries throughout the workflow, and evaluates them against original expert solutions accepted by real clients. Across eleven frontier LLMs, no agent matches the expert reference in Deployment-Adjusted Quality (DAQ), with the strongest model achieving a DAQ of 0.87 compared to the expert's 1.0.
Paper Primer
OR-License treats OR as an end-to-end professional practice: agents receive de-identified client briefs and must navigate ambiguity, missing information, and constraint validation. The core move is the integration of ORSim, which acts as a gatekeeper for business facts, ensuring agents cannot solve the problem without actively eliciting information from the client simulator during formulation and refinement.
Current LLM agents fail to match expert-level deployment reliability.
The strongest model (Grok 4.6) achieves a Deployment-Adjusted Quality (DAQ) of 0.8655, leaving a 13.45-point gap behind the expert reference. Approximately 86% of the performance deficit is attributed to infeasible deliveries (zero credit) rather than objective quality shortfalls on successful cases.
Trajectory analysis reveals that agents treat information acquisition as a front-loaded requirement-gathering task rather than a continuous process. While later-stage queries (during solver development or self-checking) yield higher confirmed issue-resolution proportions, they remain rare, indicating a failure to revisit business assumptions as solution complexity evolves.
Why is DAQ a more meaningful metric than standard objective quality?
DAQ gates objective quality by feasibility, assigning zero credit to infeasible deliveries. This reflects the professional reality that an optimized solution is worthless if it violates operational constraints.
What is the primary failure mode for agents in this benchmark?
The dominant gap is incomplete feasible delivery. Agents struggle to translate business requirements into executable code that satisfies all hidden constraints, rather than simply failing to find a high-quality numerical optimum.
Reliable OR agents require more than just coding proficiency; they must master evidence-linked self-improvement and sustained information-gap resolution to bridge the gap between initial business briefs and validated, executable solutions.
The Client Acquisition Gap in OR
We define the Client Acquisition Gap and expose why prior OR benchmarks miss real‑world challenges.
Existing OR benchmarks start from self‑contained problem statements, hiding the difficulty of turning raw client briefs into actionable solutions. This obscures three concrete gaps—task complexity, information acquisition, and professional grounding—that our OR‑License benchmark explicitly surfaces.
The gap is the mismatch between what benchmarks provide (pre‑formulated tasks) and what real OR work demands (full‑workflow handling of ambiguous client materials).
**Figure 1.** Three gaps and corresponding improvements in OR-LICENSE. Each column links an evaluation gap (top) to its improvement (bottom). The client dialogue is illustrative.
The shift from pre‑formulated problems to raw client briefs is the core challenge our benchmark addresses.
The OR-LICENSE Benchmark Framework
We build the OR‑LICENSE benchmark by curating real OR instances, adding a client‑interaction simulator, and defining evaluation metrics.
We construct the OR‑LICENSE benchmark by first curating realistic OR instances, then embedding them in a client‑interaction simulator, and finally specifying hierarchical evaluation metrics.
Curate a diverse set of industry‑level OR instances and de‑identify all proprietary information.
Define each instance’s decision space, objective function, and hidden verifier that enforces hard constraints.
Wrap the instances in ORSIM, a simulator that permits up to three client‑fact queries per episode.
Specify feasibility, expert‑relative quality, and the deployment‑adjusted quality (DAQ) metric hierarchy.
Annotate agent trajectories along three dimensions: client‑information acquisition, scientific‑inquiry chains, and error‑category profiles.
The benchmark presents a full OR workflow: an agent receives a brief, may query hidden client facts via ORSIM, and must output a feasible decision that is also high‑quality.
Agent receives the brief and requests the “budget” fact; ORSIM returns “budget = 4”.
Agent requests the “minimum” fact; ORSIM returns “minimum = 1”.
Using the disclosed facts, the agent proposes $y=(2,2,1)$, which satisfies $y_1+y_2=4$ and $y_3=1$.
Verifier evaluates the decision: both constraints pass, so $F_i=1$.
Objective value $f(y)=2+2+1=5$ is compared to the expert solution $f_{\text{expert}}=5$, yielding $Q_i=1$.
This walk‑through shows how a single query round can provide exactly the information needed to construct a feasible, optimal decision, illustrating the tight coupling between fact disclosure and solution quality.
The metrics hierarchy—Feasibility, expert‑relative quality, and DAQ—ensures that only feasible solutions contribute to the final score, while the DAQ$_\alpha$ family lets researchers probe robustness to low‑scoring instances.
Performance of Frontier Models
We benchmark eleven frontier models on feasibility, quality, and DAQ, revealing key performance gaps.
Grok 4.6 attains the highest DAQ of 86.55 % but remains 13.45 points below the expert benchmark.
Table 1 shows Grok 4.6 DAQ = 86.55 % versus expert DAQ = 100 %.
Trajectory analysis shows complete‑chain proportions from 24 % (Muse Spark 1.3) to 45 % (Grok 4.6), with all models below the 50 % threshold.
Client questioning is heavily front‑loaded: stage S1 accounts for 86.7 %–98.3 % of all questions, leaving only ~13 % for later stages.
Failure‑pattern analysis reveals formulation errors dominate for most models; static‑constraint errors reach 42.7 % for Claude Opus 5, while implementation errors dominate GPT‑5.6 Sol (33.3 %).
**Table 1.** Claude Code performance metrics across various models.
The figure displays the $DAQ_{\alpha}$ metric plotted against the penalty exponent $\alpha$ for various models, including Grok 4.6, Hy3, Hy4, Muse Spark, Qwen3.8, and an Expert baseline.
Frontier models struggle significantly with the initial acquisition phase compared to expert baselines.
Defining Success: The DAQ Metric
Defines the DAQ metric and its components for evaluating agent deployments.
The paper’s premise is that OR‑LICENSE forces agents to execute the full client‑engagement workflow, exposing gaps that benchmark‑only problems hide.
DAQ (Delivery‑Adjusted Quality) scores a system by rewarding only those instances that are fully feasible and then averaging the expert‑relative quality of the feasible ones.
Count the total number of benchmark instances $N$.
For each instance $i$, evaluate the submission on all official test cases; set $F_i=1$ only if every case passes.
Compute the feasibility rate Feas = $K/N$ where $K$ is the sum of $F_i$.
For every feasible instance, obtain the raw objective $J_{\text{agent},i}$ and the expert objective $J_{\text{expert},i}$ (apply the positive transform $T_i$ if needed) and form the ratio $Q_i$.
Average the ratios over the feasible set: $Q_{\text{feas}} = (1/K)\sum_{i\in F} Q_i$.
Multiply feasibility and average quality: DAQ = Feas × $Q_{\text{feas}}$.
Optionally, choose a sensitivity $\alpha\ge1$ and compute $DAQ_{\alpha}$ using the power‑mean formula.
How does DAQ differ from a plain feasibility rate?
Feasibility alone only counts whether an instance is passed (a binary 0/1). DAQ multiplies that pass‑rate by the average expert‑relative quality of the passed instances, so it rewards both coverage and how well the system performs on the instances it does cover.
Analyzing Agent Workflows
We detail the annotation protocol for HCAEU chains and error categories used in the trajectory analysis.
The client‑acquisition process is divided into six event‑based stages: $S_1$ (before solver code), $S_2$ (before solver feedback), $S_3$ (before the first candidate), $S_4$ (before a complete candidate), $S_5$ (before passing self‑checks), and $S_6$ (before final submission). Self‑checks are the agent’s own verification steps performed after a complete candidate but before the final hand‑off.
The HCAEU chain records a full scientific‑inquiry loop inside a trajectory: a hypothesis (H), a mechanistic explanation (C), a targeted diagnostic experiment (A), an evidence‑based interpretation (E), and an actual update (U) that changes the model or plan.
How does the HCAEU chain differ from a generic hypothesis‑testing loop?
A generic loop often stops after the experiment (A) and interpretation (E). The HCAEU chain explicitly requires a mechanistic explanation (C) before the experiment and an actual update (U) after interpreting evidence, ensuring that the insight is turned into a concrete change.
Assign every agent question to one of the six stages $S_1$–$S_6$ based on when it is asked.
Classify each observed error event into one of the eight categories (UND, DATA, DEC, CON, DYN, UNC, OBJ, IMP).
Identify research opportunities by locating consequential uncertainties that are reducible through investigation.
For each opportunity, label the presence of H, C, A, E, and U stages and assign a quality score (1–3) per stage.
Compute the task‑level quality score $R_{mt}$ using the aggregation formula (see MathBlock).
Record expert annotations, linking each label to concrete evidence from client exchanges, code revisions, and solver feedback.
**Figure 4.** Research-chain analysis for RQ1. (a) Task-equalized stopping locations along the HCAEU chain. (b) Mean stage quality among opportunities with a complete HCAEU chain versus model-average DAQ.
Simulating Client Interaction
This section details the ORSIM simulator design, selector variants, and their reliability evaluation.
ORSIM simulates a business client during an OR engagement, exposing the full client‑side workflow to the evaluated agent. The simulator isolates evidence selection from response generation to reduce hallucinations.
The simulator presents a controlled client brief, then iteratively selects disclosed business rules and feeds only the authorized evidence to a response model.
We explore three selector designs: a flat selector that sees the full rule texts, a summary‑plus‑detail selector that first scans concise summaries, and a summary‑plus‑KG selector that leverages a knowledge graph of business entities.
The selector receives the user question and a representation of the rule catalog, returns authorized identifiers, and the responder generates a reply using only the selected rule texts.
How does the flat selector differ from a conventional “search‑by‑keyword” approach that also scans full texts?
The flat selector is deterministic and returns a set of rule identifiers that satisfy the disclosure policy, whereas a keyword search would return arbitrary text spans without guaranteeing that the returned pieces correspond to complete, authorized business rules.
Evaluation uses an expert‑annotated set of 204 queries (119 standard, 85 adversarial) across five benchmark instances. Each query is run five times per selector variant, yielding 3 060 replies per variant.
Evaluation loop for each selector variant.
Metrics include selector‑level precision/recall, reply‑level faithfulness, disclosure violation rate, and end‑to‑end pass₁/pass₅. The flat selector attains 95.64 % precision and 98.01 % recall on standard queries, outperforming the other variants.
**Figure 5.** Left: question timing by workflow stage. Segment width shows the task-equalized question share; height shows the pooled proportion with confirmed issue resolution. Right: bar length shows the infeasible rate from Table 1; colored segments show the fixed annotated error-event distribution, scaled to that length. A trajectory may contain multiple events.
**Table 2.** Pre-adjudication inter-annotator agreement. Raw agreement is the percentage of units receiving identical labels from their two assigned reviewers, including both positive and negative agreement, rounded to two decimal places. Evidence support is evaluated over 5,293 factual claims; the other dimensions are each evaluated over 3,060 replies. These units cover only the three selector variants; the full-statement baseline is excluded. Krippendorff's $\alpha$ is computed using Equation 17 and rounded to three decimal places.
**Table 3.** Simulator reliability for the three selector variants and the full-statement direct-answer baseline. Each variant is evaluated on 119 standard and 85 adversarial queries, with five runs per query. Selector P/R are macro-averaged over queries with nonempty reference rule sets: 119 standard and 54 adversarial queries, corresponding to 595 and 270 runs per selector variant. The remaining 31 adversarial queries are assessed using reply-level criteria. “—” indicates that selector P/R are inapplicable to the full-statement baseline; its output-level rule coverage is reported in Table 4. Faith. is the percentage of supported factual claims. Viol. and pass$^1$ are the percentages of violating and successful replies, respectively; pass$^5$ is the percentage of queries with five successful replies. Annotation agreement in Table 2 covers the three selector variants. Percentages and percentage-point differences are calculated before rounding and reported to two decimal places.
**Table 4.** Full-statement direct-answer diagnostic. $P^{out}$ and $R^{out}$ are reply-level macro-averages over 119 standard and 54 adversarial queries with nonempty reference rule sets, with five runs per query. They must not be interpreted as the selector-level P/R in Table 3. The remaining four metrics repeat the full-statement rows of that table, using all 119 standard and 85 adversarial queries.
Characterizing Optimization Strategies
We annotate LLM solution strategies across technique, objective, and structure dimensions and compare them to expert baselines.
We annotate each final solution by three orthogonal dimensions. The technique dimension (T) records which algorithmic mechanism generated the decisions; the objective dimension (O) records whether the agent optimized the original business objective (O1) or a self‑designed surrogate (O2); and the structure dimension (S) records any explicit exploitation of problem structure such as compression (S1) or valid tightening (S4).
The technique taxonomy comprises seven mutually exclusive categories: T1 = general‑purpose solver call, T2 = specialized structural algorithm, T3 = greedy or rule‑based construction, T4 = local/neighbourhood search, T5 = metaheuristic or randomized optimization, T6 = explicit enumeration, and T7 = branch‑and‑bound style tree search. A hybrid workflow receives multiple labels when distinct modules appear in the final pipeline.
Objective handling is binary. O1 denotes direct optimization of the prescribed business goal (or an equivalent formulation that preserves its priority ordering). O2 denotes the use of an independently crafted surrogate score that guides construction or staged optimization, even when the original business objective is still present in the problem statement.
Problem‑structure exploitation distinguishes four patterns: S1 = equivalent compression or reformulation, S2 = autonomous decomposition into sub‑problems, S3 = application of a specialized structural algorithm (e.g., network‑flow), and S4 = valid tightening such as symmetry reduction or bound propagation. Mere repetition of required periods or decision rounds does not count as additional structure.
Across the E.2 analysis we see agents invoke general‑purpose solvers (T1) far more often than experts, yet they rely less on autonomous surrogate objectives (O2) and on valid tightening (S4). Enumeration (T6) appears in a notable share of agent solutions, while experts favor rule‑based construction (T3) and structured decomposition (S2).
**Table 7.** Codex-harness results on OR-License.
**Table 6.** Mean agent resource usage under Claude Code. API costs are calculated using official list prices.
**Figure 7.** Objective handling (a) and problem-structure exploitation (b) across LLMs and the expert reference. Each cell reports adoption as a percentage of the 31 tasks, using one selected solution scheme per model–task pair. O1 and O2 are mutually exclusive and sum to 100% within each row; S1–S4 may coexist. Both panels use a 0–100% color scale.
Related Benchmarks Comparison
Comparing OR‑LICENSE to related benchmarks across task sources, client interaction, and reference solutions.
This appendix expands the comparison across benchmarks for operations research, combinatorial optimization, and professional‑work settings, focusing on task sources, client‑material preservation, multi‑step evaluation, client interaction, and reference‑solution provenance.
Key related benchmarks include NL4Opt, Mamo, OptiBench, IndustryOR, OptMATH‑Bench, ConstraintBench for modeling; GraphArena, CO‑Bench, RoutBench, HeuriGym, ALE‑Bench for algorithm engineering; OR‑Space and ORAgentBench for multi‑stage agents; and GDPval, APEX‑Agents, AgencyBench, $\tau$‑bench for long‑horizon professional work.
The table displays the percentage of usage for different solver categories (T1 through T7) across various AI models, including a row for "Expert" performance.
Agent Execution Harnesses
We describe the agent harness used to evaluate OR workflows and list model settings.
Our benchmark evaluates the full OR workflow, requiring agents to inspect briefs, edit solver code, run it, and revise deliveries. To standardize this process we adopt Claude Code as the primary harness, exposing a common set of shell‑based and file‑based interaction primitives.
Agents invoke the ORSIM simulator via the Bash command ask, while the harness handles file inspection, editing, solver execution, and feedback loops. Crucially, the harness provides only execution mechanics—it does not embed any OR modeling expertise or hidden evaluator state.
**Table.** Model configuration settings including Temperature, Top-p, Top-k, Max output tokens, Max context, and Thinking strength for various AI models.
Execution Framework Details
Details on execution limits, semantic validation, and prompts for the evaluation framework.
We run experiments on Harbor 0.18.0, using Claude Code 2.1.278 and Codex 0.155.1 as harnesses. The interaction policy caps each episode at $500$ rounds, and Harbor‑native tasks receive a $12$‑hour ($43,200$ s) wall‑clock budget, with each solver call limited to $3,600$ s, covering subprocess and feedback overhead.
Bike‑sharing Tasks 26–31 need extra semantic validation because they involve scenario structures, risk preferences, and rolling optimization. Beyond feasibility and objective quality, we must ensure the solver implements the intended optimization formulation and decision process, which deterministic checks cannot fully guarantee.
All reported results were manually reviewed by human experts, and we provide a human‑validated LLM‑as‑a‑judge protocol. The judge uses DeepSeek‑V4.1‑Flash with a maximum output length of $393,216$ tokens and high reasoning strength; each task was judged five times, with every judgment independently checked by two experts, achieving full agreement.
Appendix D.5 lists the prompts for candidate agents, the simulator’s fact selector and response rewriter, and the modeling‑review judge. The candidate‑agent environment prompt defines the working directory, file persistence, available tools, and execution constraints, using double‑angle brackets for variable placeholders.
Prompting and Execution Discipline
Lists the prompts that drive LLM agents and judges for the OR‑LICENSE tasks.
D.5.2 defines three hourly‑submission prompts. Task 24 uses a rolling delivery prompt that locks each hour after a successful JSON plan submission and reveals the next data window only after time advances. Tasks 26‑31 use a similar prompt but restrict planning to historically visible data, while the grocery‑delivery prompt governs 18 irreversible batches across three days, requiring a JSON plan with specific fields for each batch.
D.5.3 adds three optional sections to the candidate prompts. The public‑data addition grants read‑only access to a directory for auxiliary files. The development‑backtesting addition lets agents repeatedly run a backtest over the development window without exposing evaluation data. The source‑code‑retention addition requires agents to keep their solving code under a work directory and report the command used to generate final files for later review.
D.5.4 specifies a simulator prompt called “Selector” that asks the LLM to return only those fact identifiers from a supplied catalog that directly answer each subquestion, omitting irrelevant facts and outputting a JSON list.
D.5.5 defines a simulator “Replier” prompt that instructs the LLM to answer a business question using only the Business Brief and Authorized Facts, clearly separating supported claims from unknowns and avoiding speculation or external knowledge.
D.5.6 introduces a suite of LLM‑as‑judge prompts that evaluate modeling choices in submitted solver code. The SSA Base prompt checks five criteria (scenario modeling, scenario weights, planning discount, planning loss terms, rolling optimization) without re‑checking already verified execution results. Additional prompts (Common Formula Reference, SSA Formula Reference, Chance‑Constrained Base, Chance‑Constraint Formula Reference) provide formulaic context and define the correct handling of scenario weighting, risk objectives, and chance‑constraint penalties.
Dataset Curation and Expert Recruitment
Details on how the benchmark instances were sourced, curated, and characterized.
We recruited experts with master‑level or higher degrees in industrial engineering, operations research, computer science, electrical engineering, or data science, each having at least three years of professional OR experience. Experts were compensated commensurately with their time and expertise.
For each selected professional OR project, domain experts de‑identified requirements, data, and deliverables, then created a brief.md file that contains the business background disclosed to the agent at evaluation start. The agent‑facing package includes these de‑identified input files, while experts also annotate client‑held information that can be acquired later through ORSIM.
Hidden constraint‑checking verifiers and objective‑function specifications were supplied for each instance; they reconstruct task‑specific operational requirements and remain unavailable to the agent during execution. Reference solutions come from competitively awarded OR engagements that were accepted by the original clients.
The final dataset comprises 31 instances across 10 industrial scenarios, representing \$3.68 million in total project value, an average duration of $4.7 months, and $2.16$ experts per project. Problem‑size estimates are reported in Appendix G.3.
Independent expert reviewers verified that each human‑authored solution code executes correctly and that hidden checkers implement the original business rules. They also confirmed that de‑identification preserved original information boundaries, ambiguity, and terminology.
Table 9 (see FigureSpotlight) lists approximate model‑size estimates—integer variables, continuous variables, and constraints—for every task, illustrating the potential scale of decision variables and constraints.
**Table 1.** Comparison of existing benchmarks with OR-LICENSE.
Each benchmark instance is evaluated in two logically separate stages: a hard‑feasibility verifier that checks output contracts and recomputes constraint‑relevant quantities, and an independent evaluator that recomputes the reported objective, cost, service level, or profit.
Six non‑exclusive competency labels (K1–K6) summarize task demands, ranging from direct static optimization (K1) to multi‑scenario risk modeling (K5) and secondary tie‑aware reasoning (K6). These labels are not used for scoring but help describe the nature of each task.
**Table 9.** Estimated scale of the optimization models.
**Figure 3.** Overview of the dataset. (a): The two-ring donut summarizes the covered industries (outer ring) and OR problem types (inner ring). Industry categories are based on the three-digit subsectors of the 2022 North American Industry Classification System (NAICS) (Office of Management and Budget, 2022), with selected labels shortened for readability. (b): Source-engagement statistics summarize the financial value and human effort associated with the original client projects. (c): The benchmark covers six core OR expert capabilities reflected in the curated tasks.