LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets
Jun Zhao, Leiming Fu, Yanbo Wen, Yiding Wang, Xuantong Liu, Yang Shu, Yuyang Lu, Xuanran Xing, Jingqi Tong, Hao Xu, Qi Zhang, Xuanjing Huang
LiveMACEBench uses live financial markets to diagnose LLM agent capabilities by separating realized outcomes from process-level mechanism use.
How can we evaluate LLM agent capabilities in evolving environments like live financial markets without relying solely on final outcomes?
Evaluating LLM agents by trading returns alone obscures the specific capabilities—like tool use or memory—that actually produce those outcomes. In evolving environments, realized performance often conflates agent skill with external market conditions. LiveMACEBench introduces a process-aware evaluation framework that records complete decision traces across matched agent configurations. It evaluates agents through three distinct signals: task outcomes, controlled mechanism effects, and trace-derived diagnostics. Across 30 days of live market trading, the authors find a pronounced outcome–capability gap where return rankings frequently diverge from capability-specific measurements. Similar trading returns often mask fundamentally different patterns of mechanism utilization and failure.
Paper Primer
The paper addresses the "attribution problem" in agent evaluation: when an agent succeeds or fails in a dynamic environment, it is unclear whether the result stems from its core reasoning, its available tools, or simply the luck of the market. LiveMACEBench treats the live market as a diagnostic substrate rather than a leaderboard, using a shared ReAct scaffold to isolate the effects of Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration.
Realized trading returns are a poor proxy for specific agent capabilities.
Cumulative return rankings fluctuated significantly over 30 days, showing 19–29 pairwise reversals across configurations, while capability-specific metrics remained temporally consistent with only 0–8 reversals. Return rankings frequently diverged from capability-specific scores; for example, the top-performing agent in memory utilization was not the top-performing agent in cumulative return.
The method hinges on "process-aware evaluation": by recording every reasoning step, tool call, and memory retrieval, the benchmark separates access to a mechanism from the effective use of that mechanism. It is like a flight recorder for an agent: it doesn't just show where the plane landed, but whether the pilot used the navigation tools correctly or ignored the warning lights.
Mechanism availability does not guarantee effective utilization.
Trace diagnostics revealed that models with identical memory mechanisms developed vastly different storage and retrieval profiles, with some models failing to grow their memory store despite frequent access. In tool use, models achieved high information coverage (above 0.8) but consistently struggled with invocation efficiency (0.58–0.73), showing that agents compensate for inefficient tool use by simply calling more tools.
Why use live financial markets instead of a static benchmark?
Live markets provide an evolving, non-resettable environment where agent actions have persistent consequences. This forces agents to handle continuous change and state accumulation, which are necessary to elicit and diagnose long-horizon capabilities like memory and rule following.
What is the "outcome–capability gap" the authors describe?
It is the observation that realized performance (e.g., profit) and agent capability (e.g., rule compliance or tool efficiency) are distinct dimensions. High returns can be achieved through luck or inefficient processes, while strong capabilities can be obscured by poor market timing.
Researchers should stop using aggregate performance metrics as the sole indicator of agent progress. Future agent evaluation must incorporate trace-level diagnostics to distinguish between mechanism access, effective utilization, and realized task benefit.
The Need for Process-Aware Evaluation
We expose why outcome‑only metrics miss agent capabilities in evolving environments.
Outcome‑only metrics tell us whether an agent succeeds, but they hide which capabilities actually produced that success. In evolving environments—such as live financial markets—outcomes conflate agent behavior, persistent state, and external changes, making it hard to attribute performance to specific mechanisms.
Instead of judging an agent solely by its final result, we examine the entire decision trace to see how and whether it accessed, used, and combined its mechanisms while the environment changed.
Outcomes alone are insufficient for evaluating agents in dynamic environments.
LiveMACEBench Architecture
LiveMACEBench defines the experimental protocol, agent variants, and evaluation metrics for long‑horizon market tasks.
LiveMACEBench evaluates agents in a continuously evolving live‑market environment, exposing all agents to identical price streams while each maintains a persistent, non‑resettable account.
It is a benchmark that couples long‑horizon paper‑trading on real‑time crypto and equity markets with process‑aware diagnostics, so we can see not only how much profit an agent makes but how it uses tools, memory, rules, and collaboration.
How does LiveMACEBench differ from other agent benchmarks like AgentBench?
AgentBench evaluates agents on static, episodic tasks; LiveMACEBench instead runs agents in a continuously updating market, measures persistent portfolio effects, and adds process‑aware diagnostics for tool use, memory, rule compliance, and collaboration.
Initialize a shared live‑market feed (crypto + US equities) and a fresh account for each agent.
Schedule decision rounds on a global four‑hour grid; at each grid point all agents start a new round simultaneously.
Within a round, the agent may: (a) query the market state, (b) invoke any enabled tools, (c) update its internal memory, (d) check hierarchical trading rules, (e) produce zero or more trade actions, then terminate.
Persist the resulting portfolio state (positions, cash) for the next round.
Record the full decision trace (model outputs, tool calls, tool outputs, actions) for later metric computation.
After the evaluation window, compute cumulative return and the capability‑specific scores (TCS, MS, RSS, OCS) from the stored traces.
Tool call: price‑forecast(asset A) → forecast = \$105.
Decision: buy 20 shares @ \$100 \to cash = \$10,000 − 20 × 100 = \$8,000; position A = 20 shares.
End of round: portfolio state (cash = \$8,000, position A = 20) is stored for the next round.
This concrete trace shows how a single tool invocation can directly affect the persistent portfolio, illustrating why later rounds inherit the risk exposure created here.
LiveMACEBench defines five controlled agent configurations: a Base ReAct scaffold and four capability‑specific extensions that each add a distinct mechanism on top of the shared environment.
Each capability is evaluated with a dedicated set of metrics: Tool use uses TR, IE, IC, EF, RQ, VCR, HFR and aggregates them into the Tool Call Score (TCS); Persistent memory uses MCQ, MUQ, $\Delta$MDD, $\Delta$TL to form the Memory Score (MS); Rule following combines HPR, SRS into the Rule Satisfaction Score (RSS) and adds RC, CH for auditability (RAS); Collaboration uses Role Diversity (RD) and Evidence Integration (EI) to produce the Observable Collaboration Score (OCS).
**Figure 1** LiveMACEBench overview. Agents interact with a shared, continuously evolving live-market environment along persistent trajectories. Evaluation combines realized outcomes with mechanism-specific process diagnostics to distinguish task performance from how effectively agents use each mechanism.
Experimental Results and Capability Analysis
Live market tests reveal divergent rule‑following despite similar returns.
Rule‑following scores vary widely across models while final returns remain clustered.
RSS spans 0.634 – 0.872, yet final return ranges only 0.511 % – 0.882 %.
**Figure 2.** Capability evaluation in a shared market environment. (a) Daily cumulative-return and Tool Call Score (TCS) rankings for the same Tool-use accounts. Rank 1 is highest; annotations count pairwise reversals among comparable pairs. (b) Full-period TCS, Memory Usage Quality (MUQ), Rule Satisfaction Score (RSS), and Observable Collaboration Score (OCS), all on [0, 1]. Colors identify models; bold marks each dimension's highest score, and dashes indicate unavailable scores. Rankings for the other dimensions are shown in Appendix C.7.
**Figure 11.** Overall trading and rule-following results across models. The left panel reports final return and maximum drawdown (MDD); the right panel reports Hard-rule Pass Rate (HPR), Rule Satisfaction Score (RSS), and Rule Auditability Score (RAS).
Agent Configurations and Metrics
Describes the experimental variations, constants, and measurements for base and advanced agent configurations.
The paper’s central premise is that process‑aware metrics are needed because final outcomes hide the reasoning steps that produced them. This section details how the base ReAct and advanced Multi‑Agent setups are varied and evaluated.
RSS measures how faithfully an agent follows the explicit operational rules that define safe and compliant behavior.
How does RSS differ from simply counting rule violations?
RSS weights violations by rule priority and incorporates soft‑rule penalties, so a single R0 breach can outweigh many minor R2 infractions, whereas a raw count would treat all breaches equally.
TCS quantifies the relevance and efficiency of an agent’s tool invocations during a decision episode.
Why isn’t a higher raw count of tool calls automatically better?
Because the metric rewards relevance; calling a tool that returns no useful information still consumes compute and time, so the score penalizes such wasteful calls.
In the Multi‑Agent setup a manager coordinates several specialist agents (trading, news, coder, analyst, critic) to produce a joint decision.
How does the Multi‑Agent approach avoid simply stacking more agents?
It explicitly evaluates the marginal information gain of each specialist against its coordination cost, calling only those agents that provide a net benefit.
ReAct is a single‑agent loop that interleaves reasoning (thought) and tool calls (action) without external coordination.
Why might a single‑agent ReAct loop miss rule violations that a manager‑driven Multi‑Agent system catches?
The manager explicitly audits rule compliance across agents; a single model must infer all constraints internally, which can lead to overlooked R0/R1 checks.
Select a backbone model (GPT‑5.4, DeepSeek‑V3.2, Gemini‑3.1‑Pro, Qwen3‑Max).
Instantiate two agents per model: one using the ReAct configuration, one using the Multi‑Agent configuration.
Hold constant: evaluation environment, market‑simulation horizon, and the process‑aware metrics (RSS, TCS).
Vary: coordination strategy (manager vs. none) and the set of specialist agents invoked.
Run each configuration for a fixed number of decision episodes (e.g., 100 episodes per model).
Measure equity trajectory, RSS, and TCS for every episode; aggregate results across episodes.
Step 1: Manager calls TradingAgent → receives a high‑conviction long‑BTC recommendation.
Step 2: Manager calls NewsAgent → receives no recent catalyst, so risk assessment stays neutral.
Step 3: Manager calls CoderAgent → runs a volatility check, confirming low volatility.
Step 4: Manager aggregates evidence, decides to open a BTC long with 0.15 × balance, leverage 2.
Step 5: Execution agent trades; equity rises from $10,000 to \$10,450, RSS = 9.8, TCS = 0.92.
This concrete run shows how the manager selectively invokes specialists, yielding a high RSS (few rule breaches) and a strong TCS (efficient tool use).
**Figure 5.** Equity trajectories under Multi-Agent Configuration and Base ReAct Configuration across four backbone models. The dotted line marks the initial capital of \$10,000.
Related Work
Provides the prompts, policies, and evaluation rubrics used for the benchmark.
Section A.1 details hierarchical trading rules and the prompts for the rule‑following agent and its auditor.
Section A.2 introduces the persistent‑memory prompts that evaluate content quality and usage quality.
Section A.3 contains the base‑configuration prompt, the tool‑use configuration prompt, and the LLM‑judge prompt for tool‑use evaluation.
Section A.4 lists the system prompts for multi‑agent collaboration, covering manager, trading, news, coder, analyst, critic, and execution agents.
AgentBench is a benchmark that measures how well large language models perform as autonomous agents across a suite of multi‑step tasks.
WebArena provides a realistic web‑based environment where agents navigate pages, click links, and extract information to complete tasks.
SWE‑bench assesses whether language models can resolve real‑world software engineering issues on GitHub repositories.
Supplementary Experimental Results
Supplementary analyses of market conditions, trading outcomes, tool use, and memory effects.
During the evaluation period both cryptocurrency and U.S. equity markets exhibited alternating positive and negative phases, while the buy‑and‑hold benchmark stayed positive throughout. This non‑stationary backdrop creates a realistic testbed for comparing agent behaviours and realised trading outcomes.
**Figure 7.** Weekly market returns during the evaluation period for the cryptocurrency basket, U.S. equity basket, and buy-and-hold reference.
**Figure 8.** Judge-specific scores for the four tool-use dimensions across Tool-Augmented ReAct agents: Tool Relevance (TR), Invocation Efficiency (IE), Information Coverage (IC), and Evidence Faithfulness (EF). Scores are shown on a 0–1 scale.
**Figure 9.** Cross-judge robustness of tool-use evaluation on the 0–1 scale. (a) Average judge scores by model. (b) Agreement on Tool Relevance (TR), Invocation Efficiency (IE), Information Coverage (IC), and Evidence Faithfulness (EF), measured by mean absolute error (MAE) and the fraction of paired scores within 0.1 points.
**Figure 10** Account-equity trajectories for all ten matched accounts over the full evaluation period. Solid lines denote w/ MEM and dashed lines denote w/o MEM. The largest matched downside differences are visible for Grok-4.20 and DeepSeek-V3.2;
Late-Phase Performance Analysis
Limits of memory, rule adherence, and collaboration on agent performance.
Process‑aware metrics matter because final outcomes hide the reasoning and tool‑use steps that produced them. This limits section examines how memory accumulation, rule adherence, and multi‑agent collaboration shape late‑phase performance and risk profiles.
Table 11 reports late‑phase results from April 23 onward, after roughly ten days of memory accumulation. With memory, GPT‑5.4 improves return to $+0.56\%$ (vs $+2.36\%$ without memory) and reduces tail loss to $-0.31\%$ (vs $-0.43\%$). DeepSeek‑V3.2 shows a modest gain $+0.29\%$ but worsens tail loss to $-0.04\%$. Gemini‑3.1‑Pro gains $+2.89\%$ and reduces tail loss to $-0.51\%$, while Grok‑4.20 and Qwen3‑Max exhibit mixed effects.
In the rule‑following analysis (C.5), audit conclusions are stable across three auditors (Qwen3.5‑397B‑A17B, DeepSeek‑V4‑Pro, GPT‑5.5) with a Spearman correlation of $0.83$. The dominant hard‑rule bottleneck is R1‑02 (single‑asset concentration limit), triggered at $31$ events/100 decisions by GPT‑5.4 and $36$ by Qwen3‑Max. Cash‑utilization (R2‑02) and active‑engagement (R2‑06) failures appear at $81$ and $46$ events/100 decisions for GPT‑5.4, and $51$ and $64$ for Grok‑4.20 respectively.
C.6 explores multi‑agent collaboration. Excluding the Grok‑4.20 multi‑agent run (incomplete trace) and swapping CriticAgent for CoderAgent changes absolute OCS values but preserves the model ordering DeepSeek‑V3.2 > Gemini‑3.1‑Pro > Qwen3‑Max > GPT‑5.4 (Table 14). This shows that OCS rankings are robust to role‑set variations.
The cumulative‑ranking protocol (C.7) extends temporal comparisons to four capability dimensions. Figure 13 shows that tool‑use and collaboration rankings fluctuate frequently, whereas memory and rule‑following rankings remain comparatively stable. Pairwise ranking reversals number 0 for tool‑use capability scores versus 21 for return rankings; memory shows 5 vs 19; rule‑following 8 vs 19; collaboration 2 vs 29.
The table presents the audit scores for various AI models (GPT-5.4, DeepSeek-v3.2, Gemini-3.1-pro, Grok-4.20, and Qwen3-Max) when audited by different agents (Qwen3.5-397B-A17B, DeepSeek-V4-Pro, and GPT-5.5), along with a mean score for each audited agent.
**Figure (a)** Rule-level hard-rule violations and soft-rule deviations. **Figure (b)** Profitability, hard-rule safety, and auditability.
**Table 14.** Uncertainty and dominance threshold. The 95% confidence intervals in Table 14 are obtained from 10,000 trajectory-level bootstrap resamples with a fixed random seed. Varying the role-dominance threshold over {0.7, 0.8, 0.9} also preserves the model ordering and changes model-level OCS by at most 0.016. These checks support the stability of the comparative conclusion while showing that absolute OCS values remain sensitive to metric conventions.
**Figure 13** Daily cumulative-return and capability rankings for tool use, memory, rule following, and collaboration. Capability scores are computed from all available evaluation records from the experiment start through each observation time: Tool Call Score (TCS), Memory Usage Quality (MUQ), Rule Satisfaction Score (RSS), and Observable Collaboration Score (OCS). Each row compares matched accounts and observation times; rank 1 is highest. Annotations count pairwise ranking reversals among comparable pairs.
**Figure 14** Positive tool-use case study. The upper panel shows the GPT-5.4 Tool-Augmented and Base ReAct account-equity trajectories. The lower panel indexes NVDA to the Tool Agent’s April 13 execution price; vertical lines mark the Tool and Base entries. Bars report, for each UTC day, the root mean square of observed hourly returns across the benchmark asset universe.
Section D provides representative decision traces that illustrate strengths and failure modes of the evaluated agents.
D.1 showcases a tool‑augmented ReAct episode where GPT‑5.4 discovers an NVDA opportunity early, executes 62 unique tool calls, and achieves a $221.84$ equity advantage over the base ReAct account.
D.2 examines persistent memory effects. In low‑volume, low‑conviction regimes (Cases 1–2) memory‑driven rules protect capital, while in a trending market (Case 3) the same memory mechanism suppresses upside, leading to lower returns for Qwen3‑Max.
D.3 contrasts execution‑level rule satisfaction with auditability. Cases 1–4 illustrate silent compliance, hallucinated violations, target‑boundary misinterpretations, and correct repair of real violations, underscoring the need for both RSS and RAS metrics.
D.4 presents three multi‑agent collaboration episodes. GPT‑5.4 uses a single specialist effectively; Gemini‑3.1‑Pro revises its plan after specialist disagreement; Qwen3‑Max records extensive evidence yet follows the aggressive technical plan, resulting in poorer performance despite high OCS.
Memory accumulation significantly impacts late‑phase performance and risk profiles.
Experimental Details
Detailed benchmark setup, runtime configurations, and supplementary results for the evaluation.
Section B introduces the full benchmark and experimental setup. It details the multi‑granularity evaluation protocol (B.1), the tradable universe and baselines (B.2), and the runtime and agent configuration specifications (B.3‑B.3.5).
Subsection B.3 breaks runtime into five components: base execution, tool‑use overhead, persistent‑memory handling, hierarchical rule‑following, and multi‑agent coordination. Each component’s timing is reported separately to isolate performance bottlenecks.
Section B.4 presents capability‑specific evaluation details, covering tool‑use (B.4.1), persistent memory (B.4.2), rule following (B.4.3), and multi‑agent collaboration (B.4.4). These metrics isolate the contribution of each capability to overall performance.
Section C aggregates supplementary experimental results. It reports market conditions (C.1), a trading‑outcome summary (C.2), and detailed breakdowns for tool‑use (C.3) and persistent memory (C.4‑C.4.4). Additional analyses for rule following (C.5), multi‑agent collaboration (C.6), and the cumulative ranking protocol (C.7) follow.
Section D provides case studies that illustrate concrete agent behaviors: tool‑augmented ReAct (D.1), persistent memory usage (D.2), domain‑policy‑guided rule compliance (D.3), and multi‑agent collaboration scenarios (D.4). These narratives connect the quantitative metrics to real‑world decision patterns.
Section A compiles all prompts, policy documents, output formats, and evaluation rubrics used by the agents. It includes the full rule‑following prompt (A.1.2) and the auditor prompt (A.1.3), as well as the hierarchical trading rules (A.1.1) that define hard and soft constraints.
Benchmark Implementation Details
Details of benchmark protocols, agent configurations, and evaluation metrics.
The evaluation records account checkpoints every 30 seconds and computes returns at 15‑minute, 1‑hour, 4‑hour, and 1‑day intervals. Returns are derived from consecutive Account Equity values $AE_{t-1}$ and $AE_{t}$ as $\text{Return}_{t}= \frac{AE_{t}-AE_{t-1}}{AE_{t-1}}$, and the overall Return multiplies $1+\text{Return}_{t}$ across the trajectory.
Additional metrics include Slice Win Rate (fraction of positive‑return intervals), Maximum Drawdown (worst peak‑to‑trough decline), Sharpe Ratio (return‑to‑volatility), Calmar Ratio (return‑to‑drawdown), TailLoss (average of the worst 5 % returns), and Rolling Volatility (standard deviation of recent returns). When fewer than two returns are available, Rolling Volatility is set to zero; if the starting account value is non‑positive, the corresponding return is forced to zero.
**Table 1.** Environment variables and their corresponding experimental settings and meanings.
The tradable universe comprises 23 assets: crypto (BTC, ETH, SOL, BNB, XRP, DOGE) and U.S. equities (AAPL, NVDA, GOOGL, META, AMZN, TSLA, PG, JNJ, UNH, JPM, V, BA, XOM, NEE, AMT, PLD, LIN). Agents are confined to this set throughout evaluation.
**Table 5.** Tool definitions for the agent's environment.
The Base runtime executes a decision round as a ReAct‑style loop: the agent observes the environment via the fixed tools, reasons, invokes additional tools as needed, and may call `execute_trade` multiple times before emitting the termination token <`TRADE_DONE`>. All interactions are recorded in a decision trace for later capability evaluation.
In the Tool‑Use runtime the agent can dynamically discover additional tools from a catalog of 295 APIs using the meta‑tool `select_tools`. Retrieved tools are validated (parameter checks, deduplication, schema conformity) before becoming callable; every tool call, successful or not, is retained in the trace.
The Persistent‑Memory runtime adds `memory_search` and `memory_add`. Entries follow a $[\text{CONDITION}]arrow[\text{OBSERVATION}]arrow[\text{RULE}]$ format and are scored by $\text{score}(m)=\alpha\!\cdot\!\text{sim}(q,m)+(1-\alpha)0.5^{d/h}$ with $\alpha=0.8$ and half‑life $h=7$ days; duplicates (similarity ≥ 0.9) are rejected.
The Rule‑Following runtime enforces a hierarchy $R_{0}>R_{1}>R_{2}$, where $R_{0}$ and $R_{1}$ are hard constraints and $R_{2}$ are preferences. Before trading, the agent produces a compliance rationale listing each rule’s status (Pass/Fail/Adjusted) and any conflict resolutions; this rationale is part of the decision trace.
The table outlines the roles, responsibilities, and available tools for various agents in a multi-agent system. The roles include Manager, TradingAgent, NewsAgent, CoderAgent, AnalystAgent, CriticAgent, and ExecutionAgent.
The Multi‑Agent runtime replaces the single Base agent with a Manager coordinating five specialist agents and an ExecutionAgent. Specialists return evidence items $e_{j}=(\text{id}_{j},\text{agent}_{j},\text{summary}_{j},\text{recommendation}_{j},\text{risks}_{j},\text{stance}_{j})$; the Manager aggregates these, forms a plan, and passes it to ExecutionAgent, all within a 100‑step budget per round.
**Table 12.** Per-judge persistent-memory scores. MCQ denotes Memory Content Quality, MUQ denotes Memory Usage Quality, and MS denotes Memory Score. For each model and judge, MS combines that judge's MCQ and MUQ with the model's fixed programmatic $\Delta$`MDD_norm` and $\Delta$`TL_norm` components defined in Appendix B.4.2. Avg denotes the arithmetic mean across the three judges.
Capability‑specific evaluations define composite scores: Tool Call Score (TCS) combines normalized LLM‑judge dimensions (TR, IE, IC, EF) with objective metrics (RQ, VCR, HFR); Memory Score (MS) averages MCQ, MUQ, $\Delta\text{MDD}_{\text{norm}}$, and $\Delta\text{TailLoss}_{\text{norm}}$; Rule Satisfaction Score (RSS) equals Hard‑rule Pass Rate (HPR) times Soft‑rule Satisfaction (SRS); Rule Auditability Score (RAS) averages Rule Coverage (RC) and Conflict Handling (CH); Observable Collaboration Score (OCS) averages Role Diversity (RD) and Evidence Integration (EI).