EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments

Zhilin Wang, Han Song, Runzhe Zhan, Jusen Du, Jiacheng Chen, Tianle Li, Qingyu Yin, Yulun Wu, Zhennan Shen, Tong Zhu, Yanshu Li, Guanjie Chen, Derek F. Wong, Yafu Li, Yu Cheng, Yang Yang

EvoPolicyGym evaluates how autonomous agents iteratively refine executable policies under bounded feedback.

How can we systematically evaluate whether autonomous agents are actually improving their decision-making policies through feedback, rather than just churning code?

Autonomous agents are increasingly expected to improve their own decision-making policies, but current benchmarks often conflate this process with open-ended software engineering or collapse it into a single final score. EvoPolicyGym formalizes this as a controlled optimization loop where an agent repeatedly edits an executable policy system under a fixed interaction budget, receiving trajectory-level feedback from sandboxed rollouts. GPT-5.5 achieves the strongest aggregate performance on the Core16 suite, demonstrating consistent top-two placement across diverse environments where other models struggle with cross-task reliability.

Paper Primer

The benchmark treats policy evolution as a budget-constrained search problem. An agent maintains a persistent workspace and submits candidate policy revisions to a server, which returns rollout summaries and trajectory diagnostics until the episode budget is exhausted.

The core mechanism is a feedback-driven edit loop: agents perform structural synthesis (creating new control machinery like memory or planning) or parametric tuning (adjusting gains and thresholds). Success hinges on the agent's ability to translate visible rollout failures into targeted code revisions rather than blind retries.

GPT-5.5 demonstrates superior cross-environment reliability compared to other frontier models.

Aggregate rank scores on the Core16 suite, which spans Gym, MuJoCo, MiniGrid, and robotics tasks. GPT-5.5 achieved a 0.891 aggregate score with top-two placement on all 16 environments.

Trajectory-level diagnostics reveal that high-performing agents are distinguished by their ability to identify effective structural abstractions early. While weaker agents churn through code changes without traction, successful agents use feedback to diagnose perception or control failures and preserve useful policy checkpoints.

Why is a fixed interaction budget central to this benchmark?

The budget forces agents to make explicit decisions about exploration, exploitation, and the efficiency of their feedback-to-revision loop, preventing the "blind retry" behavior that often inflates scores in unconstrained settings.

How does this differ from standard software engineering benchmarks like SWE-bench?

Unlike repository-level benchmarks that focus on binary unit-test success, EvoPolicyGym evaluates continuous improvement of decision-making policies where the goal is to maximize cumulative return across held-out environment instances.

Researchers should shift from measuring final task success to analyzing the trajectory of policy evolution, specifically how agents allocate budget to transition from structural synthesis to parametric refinement.

Introduction and Motivation

We define Autonomous Policy Evolution and introduce EvoPolicyGym to evaluate agents under bounded feedback.

Existing evaluations collapse autonomous policy improvement into a single final score, obscuring the iterative process and mixing it with open‑ended software‑engineering progress. To isolate an agent’s ability to turn bounded feedback into functional policy upgrades, we propose a controlled setting that separates structural synthesis from parametric tuning.

An agent repeatedly edits an executable policy, receives feedback, and decides the next edit within a fixed interaction budget.

How does the Policy Evolution Loop differ from a standard reinforcement‑learning training loop?

In RL the agent updates a policy via gradient steps while interacting with the environment, whereas the Policy Evolution Loop treats the policy as an editable program; each iteration can rewrite control flow or constants, and the budget caps the number of edit‑feedback cycles rather than raw environment steps.

The key gap is that software‑engineering benchmarks measure code churn, while functional policy improvement requires evaluating how agents refine policies under limited feedback.

The EvoPolicyGym Framework

We describe the budget‑constrained agent‑driven loop that powers EvoPolicyGym experiments.

We now describe the EvoPolicyGym experimental setup, which operationalizes autonomous policy evolution as a budget‑constrained agent‑driven loop over executable policies.

It treats a coding agent as an optimizer that edits a persistent policy workspace, submits train episodes under a fixed budget, and receives server‑generated feedback while validation and held‑out evaluations remain hidden.

How does the visibility boundary differ from a typical RL training loop where validation data is accessible?

In EvoPolicyGym the agent only sees feedback from train episodes; validation and held‑out pools are hidden until the run ends, so the agent cannot tune its policy against validation returns, unlike standard RL where validation metrics are often monitored continuously.

Policy‑system entry point required by the EvoPolicyGym server.

**Figure 1.** **EvoPolicyGym framework.** (a) **Interaction loop**: agents edit policies, submit episodic rollouts under a finite budget, and receive platform-mediated feedback. (b) **Visibility boundary**: training feedback is visible, while validation-based checkpoint selection and held-out evaluation are hidden. (c) **Environment suite**: a unified interface spanning control, navigation, driving, and robotics tasks under a shared evaluation protocol. (d) **Measured aspects**: feedback utilization, budget efficiency, and policy improvement dynamics, captured via the evolution of best-so-far performance over time.

**Figure 2.** Minimal runtime boundary for one EvoPolicyGym episode. The server owns the Gymnasium-style environment, which maps actions to observations and rewards. The submitted policy system maps observations to actions through this entry point, but may internally contain helper modules, planners, memory, diagnostics, learned parameters, or other decision logic.

The fixed visibility boundary separates online train feedback from hidden validation and held‑out evaluation, shaping the agent’s learning dynamics and ensuring that final scores reflect genuine generalization rather than overfitting to observable metrics.

Evaluation Protocol

How the Core16 benchmark splits data, selects checkpoints, and scores agents.

This section details the three‑way split of cases, the hidden‑validation checkpoint selection, and the rank‑based scoring used on the Core16 leaderboard.

The benchmark partitions each environment into three disjoint sets: train cases the agent can see and edit, a hidden validation set used only for checkpoint selection, and a hidden held‑out set used for the final reported performance.

Why is the validation set kept hidden from the agent?

Because the agent could otherwise tailor its edits to the validation cases, inflating the selected checkpoint’s performance without genuine generalization. Hiding validation forces the agent to rely solely on train‑time feedback, making the later held‑out evaluation a true test of policy quality.

Agent receives the 16 train case handles and interacts for up to 128 episodes, submitting checkpoints after each batch of submissions.

Server stores every checkpoint that finishes without error (status‑ok).

After the budget is spent, the server evaluates each stored checkpoint on the hidden 16‑case validation set and records the mean return.

The checkpoint with the highest validation mean return (later checkpoint wins ties) is selected.

The selected checkpoint is evaluated on the hidden 32‑case held‑out set; the raw mean return is recorded for that environment.

Across the four environments the held‑out means are turned into ranks; the normalized score $s_{m,e}$ is computed and finally averaged to obtain the Core16 leaderboard score.

Rank the five returns descending: 1 → GPT‑5.5, 2 → Claude Opus, 3 → MiniMax‑M3, 4 → DeepSeek‑V4‑Pro, 5 → random.

Apply the score formula $s_{m,e}=1-\frac{\text{rank}-1}{5-1}$: GPT‑5.5 gets $1-\frac{0}{4}=1.0$, Claude Opus $1-\frac{1}{4}=0.75$, MiniMax‑M3 $0.5$, DeepSeek‑V4‑Pro $0.25$, random $0.0$.

Average the four agent scores (excluding random) to obtain the environment score: $(1.0+0.75+0.5+0.25)/4 = 0.625$.

The rank‑normalized score compresses raw return differences into a uniform [0, 1] scale, making agents comparable even when environments have wildly different reward magnitudes.

**Figure 6.** Bipedal code-phase timeline, rendered with the same synthesis-edit and parametric-edit phase rules as Figure 5. The environment is tuning-dominant, but successful tuning still depends on first reaching a viable gait topology; same-topology source-bundle edits then expose whether an agent can improve that structure by adjusting constants and thresholds.

Experimental Results

Core16 experiments reveal GPT‑5.5 dominates the leaderboard while budgets and harnesses stay fixed.

GPT‑5.5 attains the highest Core16 score of 0.891, winning nine environments and placing in the top two on all sixteen tasks.

Table 2 shows GPT‑5.5’s aggregate rank of 0.891, the most wins (9) and top‑two placements (16) among the evaluated agents.

All agents were evaluated under identical conditions: a 128‑episode training budget per environment, the same split of 16 validation and 32 held‑out cases, and inclusion of each model’s coding harness as part of the system.

Core16 aggregates performance across four heterogeneous environment families to measure a policy’s reliability rather than raw return.

How does Core16 differ from a standard RL benchmark that reports average return?

Core16 ranks agents per environment and then averages those ranks, so it measures consistency across diverse tasks rather than raw reward magnitude, which can be incomparable across environments.

**Figure 3.** Score evolution over each run's episode-budget trajectory. Each curve tracks the post-hoc best-so-far hidden-validation score across candidate evaluations. Vertical jumps indicate improvements in the selected policy, while plateaus correspond to budget spent without improvement.

**Table 2.** Aggregate Core16 leaderboard. Family and Core16 scores are macro-averages of per-environment rank scores over the four agents and the uniform random-policy reference. Wins counts first-place environments, Top-2 counts first- or second-place environments, and rows are sorted by Core16 score.

Mechanisms of Policy Evolution

We dissect how agents evolve policies by separating structural changes from parameter tweaks.

Beyond leaderboard rankings, we ask how agents explore the policy space: do they invent new control mechanisms (structural synthesis) or fine‑tune existing constants (parametric tuning)? The following diagnostics answer that question.

Structural synthesis rewrites the policy’s code skeleton (adding or removing modules, branches, loops), while parametric tuning keeps the skeleton intact and only adjusts numeric constants such as gains and thresholds.

Why isn’t “tuning” just a subset of “synthesis” since both modify the policy?

Because synthesis edits alter the shape of the program (the AST), introducing or removing code constructs, whereas tuning edits only change the values of already‑present parameters. The two operations affect orthogonal dimensions of the policy.

**Table 3.** Realized computational structure in validation-selected policies. Values are group means over the same synthesis/tuning split as Figure 4. Columns report deterministic AST features of the selected policy source bundle (policy.py plus reachable local Python modules).

**Figure 4.** Relative held-out performance under different dominant task demands. Scores are macro means over synthesis-dominant and tuning-dominant environments, using the random-to-best normalization described in the text. Each row follows the same model and harness identity as the main leaderboard. Per-environment values are reported in Appendix Table 8.

The table shows where those improvements come from. On synthesis-dominant tasks, GPT-5.5 and Claude Opus 4.7 turn synthesis edits into new validation bests at high rates (41% and 48%), while MiniMax-M3 and DeepSeek-V4-Pro mostly churn structure without traction (10% and 3%). Same-topology edits rarely rescue a wrong mechanism, but they become useful on tuning-dominant tasks once the controller family is close enough.

**Figure.** Agent-saved visible diagnostics: red car, yellow road samples, cyan/magenta guide lines, and action bars. Hidden validation and heldout cases are not shown.

Supplementary Diagnostics

Supplementary diagnostics detail per‑environment performance, token usage, and edit‑size effects.

This section expands the quantitative diagnostics introduced in Section 5. It reports per‑environment held‑out scores, token traffic breakdowns, and the relationship between edit size and performance gain.

**Figure 7.** CarRacing feedback-utilization traces. Each row links evidence, attribution, policy revision, and outcome across agents. The submit column reports the submission index (s00k denotes the k-th submission) and the cumulative episode budget consumed prior to that submission. Labels are derived from logs, feedback summaries, and checkpoint diffs. The figure provides qualitative evidence of how feedback is translated into policy updates and is not an aggregate metric.

Related Work

We situate our work among benchmarks for coding agents, self‑improvement, and bounded‑feedback policy evolution.

Repository-level benchmarks such as SWE-bench evaluate coding agents on execution‑grounded software‑engineering tasks, emphasizing repository navigation, tool use, and interactive debugging.

Long‑horizon benchmarks go beyond single‑edit success, studying software evolution under repeated modifications and exposing quality degradation, multi‑file consistency challenges, and hidden‑test mismatches.

In contrast, policy evolution under bounded feedback iteratively refines executable policies from limited rollout signals and evaluates progress via continuous return rather than binary test outcomes.

Feedback‑driven self‑improvement research such as Reflexion and Self‑Refine shows language models can improve through iterative feedback, while Voyager, Eureka, FunSearch, and AlphaEvolve extend this paradigm to executable artifacts.

Interactive benchmarks place agents in web, operating‑system, database, and workplace environments to assess tool use, state tracking, and multi‑step decision making, but they remain episodic and do not capture persistent policy improvement.

Iterative system‑development benchmarks let agents design, execute, and refine models or training pipelines under repeated feedback, yet they focus on task‑specific system construction rather than general policy evolution.

Frontier‑Eng studies generative engineering design under bounded optimization, improving executable artifacts with explicit limits on feedback and computation, while our setting uses standard reinforcement‑learning environments with episode‑level budgeting.

Episode‑level budgeting and a platform‑controlled evaluation protocol enable finer‑grained trajectory‑level analysis, revealing intermediate behaviors such as retry patterns and verification issues that aggregate success rates hide.

Policy Mechanism Case Studies

Illustrative policy snippets reveal how structural synthesis and parametric tuning combine in practice.

This section showcases concrete policy programs that emerged from the autonomous evolution loop, highlighting how agents combine structural synthesis with parametric tuning to solve distinct tasks.

**Figure 8.** Observed association between policy edit size and improvement probability across adjacent submissions. Edit bins are computed from checkpoint code diffs; circle size indicates the number of transitions in each bin. The plot is diagnostic: large edits can represent useful mechanism synthesis, destructive rewrites, or interface repair depending on the surrounding run context.

Compute

When centers exist, split them into

Form a raw steering command by weighting deviations of

Clamp the blended steering to the range [‑1, 1] and update

Derive a speed command from the absolute steering magnitude via

This pattern turns raw vision into a closed‑loop controller with an explicit recovery path when the road cannot be detected, exemplifying structural synthesis (adding the road‑mask pipeline) plus parametric tuning (the blend weights and speed schedule).

Measure

Compute the gait phase as

Assemble the full action vector (hip, knee, ankle, and mirrored negative torques for the opposite limbs).

Increment the internal time counter

Return the clipped action, which the environment executes.

The policy demonstrates parametric tuning (amplitude and frequency parameters) wrapped around a structurally simple open‑loop gait, with safety scaling providing a corrective structural element.

Extract

Update the internal world representation from the current view using

Identify the cell directly in front of the agent; if a goal object (ball, key, door) is visible, select the corresponding primitive action (e.g.,

If no immediate primitive applies, assemble a set of candidate targets (balls, keys, doors, blockers, frontier cells) and run BFS on the world graph to obtain a path.

Follow the path with

Record the chosen action as

The agent combines a structurally synthesized symbolic mapper with parametric tuning of BFS heuristics, yielding a versatile planner that can switch between exploration, object manipulation, and obstacle clearing.

Parse

Determine whether a clearance maneuver is needed based on distance, lateral offset, and forward progress (

Branch on the situation: if the object is very close, set $a$ short‑range target; if clearance is required and the gripper is low, move to a high clearance height; if the gripper is far behind the object, move to an approach pose; otherwise, compute a direct push target slightly beyond the goal.

For each branch, select a target position and corresponding XY/Z gain pair (e.g.,

Invoke

The policy illustrates structural synthesis (adding a clearance branch) together with parametric tuning of gain values, showing how agents embed safety checks into otherwise straightforward geometric controllers.

`CODEBLOCK_0`

Benchmark Details and Conclusion

The benchmark defines the EvoPolicyGym run protocol and the Core16 suite of tasks.

EvoPolicyGym provides agents a live workspace and a strict interaction budget of 128 episodes per run. The server exposes /info, /task, and /submit endpoints; agents edit the policy entry point (system/policy.py) which defines a top‑level Policy class receiving observation and action specifications. Submissions are executed, their results stored under feedback/`submit_NNN`, and the system enforces live no‑rollback semantics so any valid edit persists until explicitly overwritten.

The Core16 benchmark assembles 16 Gymnasium‑compatible scenarios spanning four families—Gym/Box2D, MuJoCo, MiniGrid, and Robotics/Driving—to stress diverse control, visual, symbolic, and geometric capabilities. The four‑by‑four organization gives each family equal weight, and the suite includes environments such as Acrobot‑v1, CarRacing‑v3, MiniGrid‑DoorKey‑16x16‑v0, and FetchPickAndPlace‑v4. Table 5 summarizes the category distribution, ensuring balanced policy‑interface heterogeneity while raw returns are reported per environment before within‑environment ranking.

**Figure 5.** CarRacing code-phase timeline. Phase bands are inferred mechanically from the same policy source-bundle rule as Table 4: synthesis-edit phases denote new AST topologies after numeric constants are stripped, and parametric-edit phases denote changed source bundles under the same topology. Symbols mark validation outcomes and candidate-management events; rollback/retest are event types, not additional edit types.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers