Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Memory Requirements

Zhi Zheng, Rongsheng Chen, Yunpeng Ba, Zhenkun Wang, Yee Whye Teh, Wee Sun Lee

Agentic ESOpt uses evolution strategies to fine-tune long-horizon LLM agents with inference-level memory.

How can we fine-tune LLM agents for long-horizon tasks without the massive GPU memory overhead of traditional Reinforcement Learning?

Reinforcement learning (RL) struggles to fine-tune long-horizon LLM agents because backpropagation through long, branching trajectories is memory-intensive and makes credit assignment difficult. Agentic ESOpt replaces backpropagation with evolution strategies: it samples full-parameter perturbations, evaluates them with environment rewards, and applies a reward-weighted update that requires only inference-level GPU memory. This approach outperforms RL on long-horizon tasks, enabling full-parameter adaptation of 27B-parameter models on standard hardware.

Paper Primer

Agentic ESOpt is a backpropagation-free framework for fine-tuning LLM agents. It treats the model as a black box, sampling multiple parameter perturbations and updating the model based on the scalar rewards of the resulting trajectories.

The core mechanism is a reward-weighted parameter update that avoids the need to decompose rewards across individual turns. By using a cosine decay schedule for the perturbation scale $\sigma$, the method balances broad exploration early in training with precise exploitation as the agent converges.

Agentic ESOpt achieves superior performance on long-horizon reasoning tasks compared to RL-based baselines.

On 15-turn Sudoku, Agentic ESOpt outperforms the strongest GRPO baseline by 12.5 percentage points.

The method enables full-parameter fine-tuning of large models on limited hardware.

Agentic ESOpt requires only 8.41 GB of GPU memory for a 4B model, compared to 58.88 GB for GRPO, and allows full-parameter adaptation of a 27B model on four H100 GPUs.

Why is this approach better suited for long-horizon agents than standard RL?

Standard RL methods must assign credit across many individual actions, which becomes noisy as trajectories lengthen. Agentic ESOpt attributes the final reward to a single, coherent policy perturbation, avoiding the variance associated with per-turn score accumulation.

Does this method replace the need for prompt-space optimization?

No; it is designed to be complementary. Because the update is lightweight and black-box, it can be composed with existing prompt-space or skill-space optimization methods to enable simultaneous parameter and prompt co-evolution.

For long-horizon agentic tasks, evolution strategies offer a more scalable and memory-efficient alternative to RL, effectively shifting the bottleneck from backpropagation memory to independent environment evaluations.

The Memory Bottleneck in Agentic RL

We expose the memory bottleneck of RL for long-horizon agents and propose ES-based Agentic ESOpt.

Training LLM agents with RL requires storing gradients and activations for long trajectories, which quickly exceeds GPU memory; this makes fine‑tuning large models impractical.

When an LLM must plan over many steps, it interacts with tools, external knowledge, and branching possibilities, while only receiving sparse rewards at the end of the episode.

Instead of back‑propagating through the whole trajectory, we treat the LLM as a black box, perturb its parameters, evaluate the perturbed agents with the environment reward, and update parameters by a reward‑weighted average of the perturbations.

ES brings three key benefits for long‑horizon agents: (1) model scalability—full‑parameter updates need only inference‑level memory; (2) flexibility—black‑box feedback composes easily with prompt‑space evolution; and (3) long‑horizon scalability—parameter attribution occurs at the trajectory level, avoiding the need to decompose rewards across turns.

**Figure 1.** (a) Long-horizon agentic reasoning introduces new challenges for Agentic RL (b), including high GPU memory requirements in training and difficult credit assignment across horizons. Agentic ESOpt (c) addresses these issues through inference-level GPU memory, flexible black-box feedback, and better long-horizon scalability.

The memory cost of backpropagation in long‑horizon agents is the primary barrier that Agentic ESOpt removes.

Agentic LLM Reasoning

Defines multi‑turn LLM agents and the RL baselines GRPO and PPO used for comparison.

This section formalizes the agentic RL setting and introduces the two baseline policy‑optimization methods, GRPO and PPO, that operate on a scalar environment reward.

Agentic RL treats a language model as an interactive policy that receives observations and emits actions; GRPO and PPO are two ways to improve that policy using only a scalar reward signal.

The Agentic ESOpt Mechanism

We detail Agentic ESOpt, a memory‑efficient black‑box ES method for LLM parameter adaptation.

RL‑based LLM training stores full gradients and activations for long trajectories, exhausting GPU memory; Agentic ESOpt sidesteps this by treating the model as a black box and updating parameters directly in the space of perturbations.

**Figure 2** Detailed workflow of Agentic ESOpt. Starting from the current LLM, Agentic ESOpt samples parameter perturbations, evaluates the perturbed agents in the environment, normalizes their scalar rewards, and applies a reward-weighted ES update. Compared with Agentic RL, Agentic ESOpt provides **model scalability**, **optimization flexibility**, and **long-horizon scalability**. Its lightweight black-box interface also allows easy composition with prompt-space optimization methods such as Trace2Skill (LLM-based) and EoH (heuristic-based), enabling on-the-fly parameter adaptation within existing test-time compute procedures.

Agentic ESOpt performs full‑parameter black‑box evolution by sampling Gaussian perturbations around the current model, scoring each perturbed copy with a single scalar reward, and moving the parameters in the direction of higher‑reward perturbations.

Compute the mean reward $\mu_R=(0.6+0.3+0.1)/3=0.33$ and standard deviation $s_R\approx0.25$.

Normalize: $\hat{R}_1=(0.6-0.33)/(0.25+\varepsilon)\approx1.08$, $\hat{R}_2\approx-0.12$, $\hat{R}_3\approx-0.96$.

Form the weighted sum $\sum_i \hat{R}_i\epsilon_i = 1.08\cdot(0.02,-0.01) + (-0.12)\cdot(-0.03,0.04) + (-0.96)\cdot(0.01,0.02) \approx (0.018, -0.018)$.

With $\alpha=0.5$, update $\theta_{t+1}= (0.5,-0.3) + \frac{0.5}{3}\,(0.018,-0.018) \approx (0.503, -0.303)$.

Even with a tiny perturbation radius, the normalized rewards correctly bias the update toward the highest‑reward direction while keeping the parameter change minimal.

How does Agentic ESOpt differ from classic Evolution Strategies?

Classic ES typically evaluates a population of perturbations and computes a gradient estimate using the raw returns; Agentic ESOpt first normalizes the returns (z‑score) and omits the explicit $1/\sigma$ factor, treating $\alpha$ as the effective learning rate. This makes the update scale independent of the perturbation radius and better suited for on‑the‑fly LLM adaptation.

Instead of updating a policy’s action distribution, Parameter‑Space ES directly optimizes the model’s weights by treating the entire parameter vector as the search space.

A scalar reward $R(\tau)$ compresses an entire interaction trajectory into a single number, enabling black‑box ES updates without needing per‑step gradients.

Sample $G$ Gaussian perturbations $\epsilon_i\sim\mathcal{N}(0,I)$ and form perturbed parameters $\theta_i=\theta+\sigma\epsilon_i$.

Run each perturbed model to obtain scalar returns $R_i$.

Compute the population mean $\mu_R$ and standard deviation $s_R$, then normalize each reward to $\hat{R}_i$.

Update the base parameters with $\theta\leftarrow\theta+\frac{\alpha}{G}\sum_i\hat{R}_i\epsilon_i$.

Optionally decay the perturbation radius $\sigma$ according to a cosine schedule.

Core Agentic ESOpt loop – minimal memory footprint.

We start with a large perturbation radius $\sigma$ to explore broadly, then gradually shrink it so later updates focus on fine‑grained exploitation and reduce smoothing bias.

Agentic ESOpt can be interleaved with prompt‑space updates, allowing the model parameters and the prompt embeddings to adapt together toward the same task.

Scalability on Long-Horizon Sudoku

Agentic ESOpt scales to long‑horizon Sudoku while using only inference‑level memory.

Agentic ESOpt outperforms all baselines at the longest horizon ($H^*=15$) with 53.13 % success, a 12.5 % lead over the strongest GRPO variant.

Table 1 and Figure 3 show the crossover of success rates as $H^*$ increases.

**Figure 3.** Agentic Sudoku performance grouped by minimum successful horizon $H^*$. (a) reports final success rate averaged over 3 runs with standard-deviation error bars for PPO, the stronger GRPO-B configuration, and Agentic ESOpt. Red/green annotations below the Agentic ESOpt values report its difference from the stronger Agentic RL result. (b) and (c) show evaluation curves for the methods. Both Vanilla ES and Agentic ESOpt use $G = 32$.

**Table 1.** Agentic Sudoku final evaluation success rate ($\times 100$) and GPU memory requirement, grouped by minimum successful horizon $H^*$. Values are reported as mean $\pm$ standard deviation.

**Compute and wall-clock efficiency.** The memory and compute results characterize two complementary efficiency properties of Agentic ESOpt. The GPU-memory results in Table 1 measure its minimal training-side requirement: Agentic ESOpt requires only 8.41GB, equal to the inference memory of the Qwen3.5-4B backbone and 85.7% below GRPO's 58.88GB requirement.

**Figure 4** Intuition for population scaling: sampled directions around a stronger backbone are more likely to align with a useful Agentic ESOpt direction.

ESOpt maintains performance where RL memory requirements explode.

Fine-Tuning for Tool Usage

Agentic ESOpt lifts average task performance by over thirteen points.

Agentic ESOpt improves average performance across Math and DocVQA by 13.7 points.

Table 3 shows the mean‑@4 gain of 13.7 points over the No Skill baseline.

On the WebArena‑Lite benchmark, Agentic ESOpt raises the Qwen3.5‑27B No Skill success rate by 6.69 points.

Table 4 reports a rise from 29.47 % to 36.16 %.

**Table 3.** As shown in Table 3, Agentic ESOpt consistently outperforms the matched Agentic GRPO baselines across ReAct-style Math and DocVQA. Without evolved skills, Agentic ESOpt improves the Qwen3.5-4B base model by 13.8 and 15.0 percentage points on DAPO and AIME 2026 Mean@4, respectively, and improves DocVQA Mean@4 accuracy by 12.3 points. Averaged across these three metrics, Agentic ESOpt improves the base model by 13.7 points and Agentic GRPO by 8.3 points. Agentic ESOpt also composes effectively with Trace2Skill: the combined method achieves the strongest Qwen3.5-4B Mean@4 results on DAPO, AIME 2026, and DocVQA, showing the flexibility of parameter-space optimization methods to compose with external skill optimization.

**Table 4.** Performance comparison of different models and methods across various datasets, showing the impact of Agentic ESOpt.

Automatic Heuristic Design

Agentic ESOpt boosts test‑time heuristic design, improving most comparisons under equal budgets.

Traditional test‑time compute keeps the LLM policy fixed while searching over heuristics. Agentic ESOpt’s flexibility lets it adapt the model parameters together with heuristic generation, opening a joint search space.

AHD asks an LLM to write small pieces of code that make greedy decisions for combinatorial problems such as TSP, KP, and ASP.

Agentic ESOpt improves 28 of 36 matched method–budget comparisons across constructive and ACO‑style test‑time tasks.

Aggregating results from Table 5 and the ACO‑style ablations shows a net gain in 28 out of 36 method–budget pairs.

**Table 5.** Design constructive heuristics at total evaluation $T \in \{1000, 2000\}$. TSP is minimized; KP and ASP are maximized. The Optimal row shows optimum objective values, not an experimental baseline. Sample and EoH are each paired with their Agentic ESOpt counterpart; the $\Delta$ rows report the gap-ratio gain defined in the text. Baselines and Agentic ESOpt use LLaMA-3.1-8B-Instruct for all runs.

Limitations and Hyperparameters

Limits of Agentic ESOpt and avenues for future work.

Agentic ESOpt sidesteps the memory bottleneck of RL by using black‑box evolution strategies, which only need forward passes at inference time.

Unlike standard Agentic RL, ESOpt adds hyperparameters for the perturbation radius $σ$ and the step size $α$, whose optimal values can vary with the LLM, reward distribution, and environment.

Nevertheless, Appendix D.6 reports that a single configuration ($σ₀ \approx 1\! \times \!10^{-3}$, $α \approx 5\! \times \!10^{-4}$) works well across five experiments, suggesting the hyperparameters are fairly generic.

We leave the design of an automatic schedule for $σ$ to future work.

When environment evaluation itself is costly, the advantage of ESOpt—more trajectories for the same FLOPs—may diminish, because rollout cost can dominate the total budget.

Continual learning remains an open question: ES updates are dense, but most changes stay within the perturbation scale, as Table 7 shows (96.26 % of updates ≤ $σ$, 99.42 % ≤ $2.0\! \times \!10^{-3}$).

**Table 7.** Distribution of parameter-update magnitudes after Agentic ESOpt for WebArena on Qwen3.5-27B with perturbation scale $\sigma_t = 1.5 \times 10^{-3}$.

Looking ahead, scaling Agentic ESOpt to larger LLMs and studying population‑scaling laws could enable full‑parameter adaptation of much bigger agents.

Quantizing the model while applying dense ES perturbations will require scale‑aware noise generation and numerically stable updates, opening a promising systems direction.

Finally, tighter co‑evolution of skills and parameters—allowing external context $c_t$ and model weights $θ_t$ to evolve on comparable timescales—could continuously reshape the data distribution each influences.

Mathematical Derivation of ES Gradient

Derivations and proofs underlying Agentic ESOpt and its computational trade‑offs.

Section C.1 derives the scalar‑score Evolution‑Strategies (ES) gradient that Agentic ESOpt uses.

Section C.2 proves Lemma 1, quantifying the bias introduced by Gaussian smoothing.

Section C.3 compares the variance growth of policy‑gradient (PG) estimators with that of the ES estimator as the horizon $H$ grows.

Section C.4 formulates Automatic Heuristic Design (AHD) as optimization over a space of heuristics.

Section C.5 derives the model‑side FLOPs required by ES, GRPO, and PPO during training.

WebArena Adaptation Stages

Component ablations reveal how each design choice impacts performance across tasks.

This section isolates each architectural component of Agentic ESOpt and measures the performance change when the component is removed or altered.

**Figure 7** Repeated-sampling profiles for DAPO, AIME 2026, DocVQA ANLS, and DocVQA accuracy. The $k = 1$ markers use the reported Mean@4 values as estimates of single-sample performance, while stars show the originally reported Pass@4 or Max@4 values. Curves report the newly estimated best-of-$k$ results for $k \in \{2, 8, 16, 32\}$; the newly computed $k = 4$ curve points are omitted to keep the original four-sample measurements visually distinct.

**Figure 8.** Average realized trajectory length during DocVQA training. Agentic GRPO and Agentic ESOpt use the same rollout decoder; their training schedules are aligned to a common update axis and shown through step 40.

Discussion and Future Directions

We discuss when ES is most useful, summarize findings, and acknowledge contributors.

Population size $G$ is a key ES hyperparameter; Table 6 shows that larger backbones become less sensitive to increasing $G$, with the 4B model gaining +594.5% best‑test accuracy when $G$ doubles, while the 9B model improves only +24.1%.

Takeaway 5 highlights that doubling $G$ dramatically boosts final‑test success for the 4B model (+677.0%) but yields no gain for the 9B model, suggesting stronger models may require fewer sampled directions.

In conclusion, Agentic ESOpt offers a memory‑efficient alternative to RL for long‑horizon LLM agents, scaling favorably with model size and integrating naturally into test‑time compute loops, while acknowledging current study limits and outlining future directions.

We thank Jiaying Wu, Penghui Qi, Zichen Liu, Ziqiao Meng, and Ziang Li for their valuable comments on methodology and writing.

Experimental Settings

Appendix D details experimental setups, hyperparameters, and diagnostics for Sudoku, Math, DocVQA, and WebArena‑Lite.

Section D.1 introduces the Sudoku environment and presents three diagnostics that probe long‑horizon scaling as the minimum successful horizon $H^{*}$ grows.

First, a two‑dimensional slice of the parameter‑space reward landscape is visualized; increasing $H^{*}$ lowers absolute reward levels and contrast while preserving enough variation for evolution‑strategy updates.

**Figure.** (a) Minimum successful horizon $H^* = 5$, (b) Minimum successful horizon $H^* = 10$, (c) Minimum successful horizon $H^* = 15$.

Second, per‑turn correctness $p$ is related to trajectory‑level success via the schematic formula $S_{H^{*}} = p^{H^{*}}$, illustrating exponential decay of success with longer horizons.

**Figure a.** Schematic compounding from per-turn correctness to trajectory success.

Third, realized horizons during training are compared; Agentic ESOpt stays close to the target $H^{*}=15$ whereas GRPO drifts upward, indicating tighter adherence to the minimum successful horizon.

**Figure b.** Measured realized-horizon diagnostics on the $H^* = 15$ setting.

Training configurations for Sudoku are summarized in Table 8; both Vanilla ES and Agentic ESOpt use 100 generations, 32 directions per generation, and an update scale $\alpha = 5\times10^{-4}$ with varying mask counts and sigma schedules.

Vanilla‑ES population‑sensitivity experiments (Section D.1.3) fix $\sigma = 5\times10^{-4}$ and vary the population size $G\in\{8,16\}$ to isolate the effect of population on performance.

Section D.2 shifts to Math Reasoning, where Agentic ESOpt training (Table 9) runs 25 generations on a Qwen3.5‑4B backbone with population size $G=16$ and a cosine sigma schedule from $10^{-3}$ to $5\times10^{-4}$.

Multi‑turn GRPO for Math (Table 10) follows the same sampling regime but uses a learning rate of $10^{-6}$, KL coefficient $10^{-3}$, and a maximum interaction budget of 100 user + 100 assistant turns.

Section D.3 covers DocVQA; Agentic ESOpt training details (Table 11) mirror the Math setup with 40 generations and identical hyperparameters, while the reward is continuous ANLS with a strict $>0.5$ threshold.

Multi‑turn GRPO for DocVQA adopts the same interaction limits (50 turns, 512 tokens per turn) and optimizer settings as the Math baseline, ensuring a fair comparison.

All methods share a common evaluation protocol: 100 held‑out Math problems and 100 DocVQA questions, each evaluated with four samples, temperature 1, top‑p 1, and top‑k 40.

Repeated‑sampling profiles (Section D.3.4) extend the four‑sample results to budgets up to $k=32$; Agentic ESOpt consistently outperforms matched GRPO baselines across both Math and DocVQA metrics.

Training‑stage turn diagnostics for DocVQA (Section D.3.5) reveal that GRPO trajectories lengthen later in training, whereas ESOpt trajectories remain shorter and more stable, echoing the Sudoku pattern.

Section D.4 introduces WebArena‑Lite, a goal‑conditioned browser task where each episode consists of up to 30 actions and a 2,048‑token response budget.

The rollout protocol (Section D.4.2) uses temperature 0.7, top‑p 0.8, top‑k 20, and identical sampling for all frozen and Agentic ESOpt rows; three seeds are evaluated on 165 held‑out tasks.

Parameter and skill adaptation stages (Section D.4.3) separate the frozen model updates from the skill‑distillation pipeline, ensuring that WebArena‑Lite tasks never contribute to parameter‑update rewards.

Sudoku Environment Prompts

Provides the prompt format and example for the multi‑turn Sudoku environment used in evaluation.

Sudoku is presented as a multi‑turn action environment where the agent must fill the board one cell at a time. At each turn it sees the original puzzle, the current board, the list of empty cells, and optional feedback from the previous action, then emits a single command of the form set <row> <col> <value>. Rows and columns are 1‑indexed, the episode ends when all masked cells are filled or the action budget runs out, and the final reward is binary.

Turn 0 of the evaluation split contains 15 masked cells. The prompt shown below includes the original puzzle, the current board (identical at turn 0), and the explicit list of empty cells.

Turn‑0 prompt for the Sudoku environment (`mask_count` = 15)

The environment applies the action, inserts a feedback line such as “Last environment feedback: Filled r1c5 with 6.” before the puzzle block, updates the board, increments turn to 1, decrements `remaining_empty`, and removes the filled cell from the empty‑cell list.

**Figure 9** Full WebArena-Lite evaluation success rate during the No Skill Qwen3.5-27B Agentic ESOpt run. Evaluation is performed every 10 ES updates; the curve starts from the 29.50% base checkpoint and reaches 35.76% after 70 updates.

Math Reasoning Prompts

Provides ReAct prompts for math reasoning and outlines the trajectory‑to‑skill composition pipeline.

E.2.1 introduces a ReAct‑style prompt that treats the AIME math task as an agentic environment where the model can interleave thoughts, bash actions, and observations, receiving a binary exact‑match reward for the final boxed answer.

The example problem asks for the distance to the park given staggered start times and speeds; the model issues a Python command to solve the resulting equations, observes the output “$252/25$ $277$”, and reports the final answer $\boxed{277}$, yielding $m + n = 277$.

E.2.2 describes the trajectory‑to‑skill composition pipeline: from 25 ES generations it retains at most one failed trajectory per training problem, discards successful traces, runs a post‑hoc Trace2Skill pass with GPT‑5.4‑nano (temperature 1) across three seeds for each model, selects the best distilled skill, and evaluates by injecting this skill into the system context without further parameter updates.

DocVQA ReAct Prompts

Shows the ReAct prompt used for DocVQA and how the skill‑distillation pipeline composes traces.

DocVQA is treated as an agentic ReAct task: the agent receives a document image path, may invoke command‑line or Python utilities to inspect, crop, or OCR the image, and finally returns a short answer that is scored with ANLS.

The prompt requires a Thought → Action → Observation loop, with actions expressed as a JSON block invoking a bash command; an example sequence queries the image size, crops the chart region around the 1975 tick, observes the saved crop, and extracts the “actual” value 0.28 per 1000 as the final answer.

E.3.2 explains how the final 50 task occurrences are filtered to keep at most one failed and one successful trace per occurrence, then a single post‑hoc Trace2Skill pass with GPT‑5.4‑nano (temperature 1) distills a skill for three seeds (base, Agentic RL, Agentic ESOpt); the best skill is injected into the system context without further parameter updates.

WebArena Skill Distillation

This appendix details the WebArena skill used by Trace2Skill agents.

Trace2Skill agents follow a prescribed WebArena skill that relies solely on visible page evidence and the current WebRL IDs. The policy iterates: observe, act, re‑read, and repeat, only advancing when the page visibly changes.

The workflow mandates using the most direct visible path—search, filters, or built‑in navigation—before resorting to long scrolling or guessing controls. After each click the agent re‑reads the page and aborts the step if nothing changed.

When an action repeats (e.g., typing), the agent may switch to a different action type if needed; before typing it verifies the target text to avoid overwriting.

For each form field the agent reads the label, current value, and input type, then clears or selects all existing content. Dropdowns and autocomplete fields are inspected first, the visible matching option is chosen, and the selection is confirmed.

Tasks that require matching all items trigger exhaustive pagination and scrolling until no entries remain. The agent maintains a running tally of unique items and verifies counts directly from visible rows.

Before answering a status query the agent opens the item’s detail page, confirming the review title or metadata rather than relying on snippets. Extraction requests copy the exact visible span verbatim.

When creating or editing resources the agent locates an explicit writable composer, fills every required field, and clicks the page’s own save or submit control. Success is confirmed by a visible persisted record, not by a transient toast.

For reporting tasks the agent first uses the site’s dedicated report page, sets filters, and generates the output. Map‑related queries are handled in a separate directions mode, and the answer is taken from the route summary.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers