Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition
Kaisen Yang, Qingle Liu, Kejin Wang, Yicheng Zhao, Jieming Li, Shenghan Zheng, Ruize Yang, Bojun Yang, Heng Gong, Xiang Gao, Lanyue Zhang, Kaiyu Zhong, Zhuo Liu, Shaoxuan Li, Chengxi Li, Yong Yan, Weixuan Zhang, Tianwei Luo, Situ Wang, Youjie Zheng, Sihan Zhao, Shengyuan Wang, Huan-ang Gao, Jiazheng Xu, Xiaohui Xie, Wentao Han, Hongning Wang
AAArena formalizes Adversarial Heuristic Learning, using AI agents to iteratively refine game-playing code against human-archived opponents.
Can current LLM-based agents effectively learn and adapt game-playing strategies through iterative self-improvement in a competitive, long-running environment?
AI agents struggle to adapt strategies in adversarial environments where opponents constantly shift and feedback is limited. The authors formalize Adversarial Heuristic Learning (AHL), a paradigm where an AI agent acts as a learning engine, reading game logs and replays to iteratively revise an executable policy's code and supporting tools while keeping the base model's weights fixed. On the AAArena benchmark, the strongest configuration earns gold medals in six of twelve games, though complex rule specifications remain a significant barrier to achieving top-tier performance.
Paper Primer
AHL treats the game agent as a "heuristic system" consisting of both the executable policy and a development state that stores tools, tests, and memory. The AI agent operates in a loop: it acquires experience through matches, interprets feedback from logs or replays, and directly edits the source code to improve competitive rank.
AI agents can reach human-level performance in adversarial games through iterative code revision.
The Opus5.5 configuration achieved rank-1 finishes (gold medals) in 6 out of 12 games in the AAArena benchmark. 50% of the evaluated game ladders were topped by the best-performing AI agent.
Performance is strongly correlated with rule complexity: agents consistently struggled to top ladders in games with larger abstract syntax trees (ASTs) and more rule atoms, suggesting that translating complex rule specifications into effective code remains a primary bottleneck.
Why use this "heuristic learning" approach instead of standard reinforcement learning?
AHL leverages the agent's ability to perform explicit, interpretable code revisions and tool management, allowing it to learn from limited samples and diagnostic replays without requiring the massive interaction budgets typical of gradient-based reinforcement learning.
What is the role of the "small-match" versus "full-pool" evaluation budgets?
Small matches allow the agent to perform targeted diagnosis and iterative testing against specific opponents, while full-pool evaluations provide a standardized, periodic measure of the agent's Elo and rank against the entire human-archived ladder.
Introduction and Motivation
We expose the limits of heuristic learning when agents refine game policies under adversarial constraints.
Adversarial games have long driven AI progress, but learning robust strategies from limited samples remains difficult.
AI agents try to improve game strategies by repeatedly revising executable code based on game logs and feedback, yet it is unclear how far this heuristic learning can scale toward human‑level strategic adaptation.
We formalize this setting as Adversarial Heuristic Learning (AHL) and introduce AAArena, a benchmark of twelve authentic games with 1,920 archived human programs, evaluated under a competition‑style protocol.
Agents must interpret game rules, select opponents, analyze replays, and revise their game agents within fixed match and evaluation budgets.
Our experiments with seven model‑harness configurations show that Opus5.5 paired with Claude Code reaches the top of six human ladders, while no system tops the remaining six, especially those with more complex rule specifications.
Further analysis reveals that careful opponent selection and dense feedback accelerate policy improvement, and that leveraging both on‑policy and off‑policy replays helps overcome learning plateaus.
The gap between current LLM‑based agent learning and human‑level strategic adaptation remains substantial, especially in games demanding deep rule comprehension and long‑horizon planning.
The Heuristic Learning Framework
Describes the experimental setup and the adversarial heuristic learning loop.
The loop repeatedly gathers game experience, interprets feedback, and updates the executable policy while the underlying model parameters $θ$ stay unchanged.
How does this heuristic learning loop differ from standard reinforcement learning?
Standard RL updates the policy parameters $θ$ directly from reward signals, whereas the heuristic loop keeps $θ$ fixed and instead modifies the surrounding executable code (tests, configuration, memory) that defines the policy. The AI agent therefore learns to edit the program rather than to adjust neural weights.
Fix the base model parameters $θ$ and load the initial policy $P_0$ and development state $C_0$.
Choose a set of feedback sources (e.g., game replays, test suites) and an evaluation schedule (e.g., every 10 iterations).
While the stopping criterion (e.g., max iterations or Elo plateau) is unmet, repeat:
Acquire game experience by playing matches against opponents from the frozen pool.
Interpret the feedback using $A_θ$ together with resources $R$, current policy $P$, and state $C$.
Revise $P$ and $C$ according to the interpreted feedback.
Measure competitive performance (Elo) after each revision to track progress.
Initialize $P = P_0$, $C = C_0$.
Acquire one match replay where the agent loses 2–0; the replay is stored in $R$.
Interpret feedback: $A_θ$ detects that the opponent exploited a missing defensive test, so it adds a new test to $C$ and patches $P$ to include a defensive check.
Revise: $P$ becomes $P_1$ (adds a conditional branch), $C$ updates to $C_1$ (records the new test).
Evaluate $P_1$ against the frozen pool, obtaining an Elo gain of +15 points.
This single iteration shows how the heuristic loop can improve performance by editing code rather than adjusting neural weights.
**Figure 2** Agent for agent iteration loop. An AI agent uses game resources, replays, and retained notes to revise an executable game-agent policy. Validated snapshots enter small matches against selected opponents or full-pool evaluations against the frozen pool. Small matches return dense replays; full-pool evaluations return outcomes, Elo, and rank. The agent retains development history across revisions.
The AAArena Benchmark
The AAArena Benchmark defines games, opponent pools, and evaluation budgets for measuring heuristic learning.
The AAArena Benchmark provides a unified experimental platform for evaluating heuristic‑learning agents across a diverse set of twelve games.
It bundles a curated suite of games, frozen opponent pools, and a two‑tier evaluation protocol so that progress can be measured both on targeted diagnostics and on full competitive standings.
How does AAArena differ from a typical single‑game benchmark?
Most benchmarks evaluate on one fixed environment, whereas AAArena combines twelve distinct games, each with its own frozen opponent pool, and offers both fine‑grained (small‑match) and coarse‑grained (full‑pool) feedback. This prevents agents from exploiting idiosyncrasies of a single game and forces them to learn more general strategies.
**Figure 1.** **AAArena overview.** Left: the 12 games in ascending rule-description size (abstract syntax tree node count), with player counts, role symmetry, and information structure (Table 1). Right: retained-champion Elo for seven models and human SOTA. Each spoke reports native Elo, with game-specific tick intervals; higher Elo lies farther outward. Human SOTA forms the dashed regular dodecagon, with its Elo labeled in red. Model colours match Figure 3.
Allocate budgets $B_s$ (small‑match units) and $B_\ell$ (full‑pool evaluations), defaulting to $B_s=128$ and $B_\ell=16$.
Run up to $B_s$ small‑match games against selected opponents from the frozen pool, collecting per‑opponent outcomes.
Run up to $B_\ell$ full‑pool evaluations, where the agent plays against the entire frozen pool and receives an Elo rating.
Gather all officially evaluated snapshots $V$, compute their ranks $r(P)$, and select the maximum rank as the final score.
Retain the policy that achieved the maximal score as the “champion” for possible further revisions.
Small‑match 1: agent defeats Opp‑A, earning a local win.
Small‑match 2: agent loses to Opp‑B, recording a loss.
Full‑pool evaluation: agent plays against Opp‑A, Opp‑B, and Opp‑C, obtaining Elo scores 1500, 1450, and 1520 respectively.
Snapshot ranks are computed (higher Elo → higher rank); the best rank among the three is 1520.
The final score $\text{Score}(2,1)$ is 1520, and the policy achieving it is retained as the champion.
Even with a tiny budget, the protocol can surface the strongest policy by combining targeted diagnostics with a single full‑pool check.
The benchmark’s twelve‑game diversity and frozen opponent pools give a stable, multi‑facet yardstick for tracking heuristic‑learning progress.
Performance Overview
AI agents reach top rank in most games while using a uniform budget.
AI agents achieve rank 1 in eight of the twelve evaluated games.
Table 2 shows a green check (✓) for rank 1 in eight rows, indicating at least one configuration reached the top spot.
All configurations were evaluated with the same 128‑small/16‑full budget and max reasoning effort; only Opus5.5 uses Claude Code while the other six models rely on Codex.
The Elo of the best policy that survived the training run and is kept for final evaluation.
Ablations and Learning Dynamics
Analyzes how policies change over learning, budget extensions, and feedback choices.
Heuristic learning lets AI agents iteratively revise code based on game logs and performance feedback; this section examines how the resulting policies evolve under different interaction budgets and feedback choices.
GLM‑5.3 reaches the top‑six human ladders in six games but does not achieve rank 1 in the other six.
Performance overview (previous section) reports the per‑game Elo and rank for all twelve games.
Under the default 128‑small/16‑full budget we observe three trajectory types across the 12 runs: Success (rank 1 achieved), Progress (overall gains with occasional regressions), and Stagnation (extended plateaus after early adaptation).
Targeted code revisions raise Dorado’s Elo by 147.7 points and improve its rank from 41 to 2.
Case study in Section 4.2.2 details three successive revisions that produce the final rank‑2 policy.
**Figure 4.** GLM-5.3 on *Dorado*. Full-pool Elo and rank, small-full interaction order, and token use under the 128/16 budget. Appendix B.5 specifies budget-slot placement, batch markers, rank annotations, and token accounting.
**Table 4.** Threefold-budget continuation for GLM-5.3. Elo and rank of the retained champion at each budget. The extension adds 256 small units and 32 full-pool evaluations to the same run. A positive $\Delta$Rank indicates an improved position.
**Table 4.** Tripling the budget leaves two retained champions unchanged and improves two; none reaches rank 1. Generals and LOTA keep their earlier best scores, while Miracle moves from rank 26 to 19 and AntWar2 from rank 9 to 6. Figure 5 shows flat intervals and fluctuations through the extended runs. These outcomes describe a finite continuation of four GLM-5.3 runs under the same development protocol.
Tripling the interaction budget yields a maximum overall Elo increase of +428.3 points, with only modest improvements for individual games.
Table 4 shows the Elo progression from 1782.0 to 2210.3 across the extended run.
We compare four opponent‑selection strategies—model‑chosen, ladder, random, and top‑four—on Pacman, AntWar, and Miracle while keeping all other settings constant.
Dense replay feedback (full small‑match logs) consistently raises retained‑champion Elo compared with binary win/loss feedback in all three games.
Batch size experiments reveal the best performance with four opponents per small match for Pacman and Miracle, and two opponents for AntWar; larger batches exhaust the small‑match allowance too quickly.
Off‑policy replays enable breakthroughs: Pacman reaches rank 1, Miracle improves from rank 27 to 15 ($\Delta$ ‑12), while on‑policy learning stalls at rank 26.
Figure 10 contrasts on‑policy and off‑policy Elo and rank outcomes for the three games.
**Figure 5.** Policy performance under a threefold budget. Full-pool Elo, interaction order, and token use for four continued runs. Ranks appear at every third full-pool evaluation and local Elo peaks. Plot conventions appear in Appendix B.5.
**Figure 6.** Opponent selection. GLM-5.3 retained-champion Elo and rank with model, ladder, random, and top-four selection; all use dense feedback and batch size 4. Plot conventions appear in Appendix B.5.
**Figure 7. Replay feedback.** Dense replays versus binary win/non-win feedback under the same 128/16 budget. Labels give Elo and rank. Dense feedback improves performance in all three representative games.
**Figure 8** Small-match batch size. Number of opponents per submission, with a fixed total allowance of 128 opponent tests and 16 full-pool evaluations. Labels give Elo and rank. The best size is 4, 2, and 4 for Pacman, AntWar, and Miracle, respectively.
**Figure 9.** On-policy and off-policy learning trajectories. Full-pool Elo and ranks, cumulative tokens, and token increments for GLM-5.3 with self-generated and other players’ replays. Both conditions receive full-pool feedback on their own policies. Appendix B.5 specifies rank rows, reference levels, and token accounting.
**Figure 10.** On-policy versus off-policy replay learning. Retained-champion Elo and rank for GLM-5.3 under the same 128/16 budget ceilings. On-policy scores are the main-table results. Labels give Elo and rank; each panel uses the same zero-based Elo scale. Both Pacman runs terminate early upon reaching rank 1.
**Figure 13.** Stagnation. Full-pool Elo and ranks, interaction order, and token use for games that plateau after initial adaptation. Plot conventions appear in Appendix B.5.
Related Work
We compare prior game‑agent evaluation protocols and position AAArena among them.
This section surveys existing evaluation protocols for game‑agent programs, highlighting how they differ in output type, opponent modeling, feedback availability, and evaluation protocol.
**Table 1.** Comparison of evaluation frameworks for AI agents.
Compared to prior work, AAArena retains the “Programs” output and “Programs” opponents but deliberately removes feedback, enabling a stricter test of autonomous code synthesis under a fixed evaluation budget.
Game Descriptions and Future Work
Appendix A lists the twelve games evaluated in AAArena, summarizing their core mechanics.
Pacman is a symmetric competitive maze game where players collect beans, avoid monsters, and place mines that affect movement. Scoring depends on bean collection, collision avoidance, and opponent disruption.
SnakeGo is a territory game where snakes move, collect items, claim cells, and can split or use special actions. Immediate safe moves can be costly later due to long‑horizon territory growth.
Rollman is an asymmetric maze game; one side gathers beans and uses temporary skills, while the other coordinates ghosts to intercept. The differing action spaces and objectives create distinct policy problems.
MoneCraft is a symmetric mining game where players capture mines that generate currency and can teleport, place mines, or purchase protection. Policies balance movement and disruption against sustained resource production, with final currency determining the winner.
AntWar combines tower defence and economic management; players build and upgrade towers, improve bases, and time super‑weapons while ants follow environment‑driven paths. The policy must balance present defence with spending that influences future pressure.
LostSpace is a multiplayer survival game on a layered map that contracts over time; players collect supplies and keys, use tools, fight or evade rivals, and aim to escape. Trade‑offs involve immediate combat, survival, and timing of escape attempts.
AquaWar is a hidden‑identity tactical game where players select fish, make assertions about opponent identities, and choose attacks or type‑specific skills. Success depends on roster design, inference, and timing of information‑sensitive actions.
Generals combines territorial expansion, army movement, neutral assets, production, technology, and tactical abilities; effective policies coordinate economic development with attack timing rather than optimizing either alone.
Dorado blends resource competition with hero and army development; players choose heroes, produce units, mine resources, and invest in upgrades. Policies must coordinate movement, combat, harvesting, and abilities to threaten bases while sustaining production.
Miracle is a hex‑grid strategy game with heterogeneous units and a drafting stage; players select creatures and an artifact, then summon, move, attack, and use abilities to destroy the opponent’s miracle. Resource limits and unit interactions make tactical coordination central.
LOTA is a lane‑based combat game where each player chooses a hero and controls movement, attacks, abilities, skill development, and revival while units advance along fixed lanes. Victory requires coupling immediate fights with sustained pressure on defensive structures.
AntWar2 extends tower‑defence with ordered operation lists whose accepted prefix changes subsequent decisions; agents must coordinate tower placement, economic development, and weapon timing while respecting sequence‑dependent legality.
Evaluation and Resource Details
Details on resources, evaluation, metrics, and analysis for the appendix.
Public resources and opponents are frozen across runs; the opponent pool size counts programs, not distinct participants. The workspace contains rules, SDKs, replay documentation, a leaderboard, and a single reference program (rank‑40 entry). The agent cannot read opponents’ source code or internal evaluator state.
Each model‑game run uses a persistent writable session; Opus5.5 runs in Claude Code while other models run in Codex. The agent decides when to read rules, edit code, test selected opponents, and submit a full‑pool evaluation. Small matches return replays; full‑pool evaluations return Elo, rank, and per‑opponent outcomes, and the highest officially evaluated Elo is reported.
Elo is estimated with a fixed‑pool Bradley–Terry model using match outcomes $s_m\in\{0,\tfrac12,1\}$ and opponent ratings $r_j$. A virtual draw against the pool’s mean rating $\bar r_H$ keeps the estimate finite for perfect win or loss records. Rank is defined as one plus the number of opponents with higher rating.
Gold medals count rank‑1 finishes across the twelve games. For configuration $m$, Gold$(m)=\sum_{g=1}^{12}\mathbf{1}\{q_{m,g}=1\}$, where $q_{m,g}$ is the reported rank. A gold medal reflects first place under the rating protocol, not a direct head‑to‑head win against the human leader.
Rule‑description size is measured by two language‑agnostic metrics. AST counts all structural and expression nodes in the canonical abstract syntax tree of the game description. RA (rule atoms) counts independently changeable propositions across seven classes (state, action, observation, setup, condition, transition, outcome). Both metrics ignore identifier names, comments, and formatting.
Trajectory plots have three panels. Panel (a) shows full‑pool Elo over evaluations with rank annotations; panel (b) displays interaction order across budget slots, marking wins (green), losses (magenta), and draws (grey) for small matches and diamonds for full‑pool rank; panel (c) reports token usage with cumulative lines and incremental bars.
**Figure.** Success metrics for four agents: Rollman, MoneCraft, Pacman, and AquaWar. Each agent's performance is visualized through three panels: (a) Full-pool Elo over evaluation iterations, (b) Opponent selection ladder rank over interaction slots, and (c) Token usage (cumulative and incremental) per evaluation.
**Figure.** Progress of five different agents (AntWar, Dorado, LostSpace, SnakeGo, AntWar2) across 16 full-pool evaluations. Each agent's progress is shown in a row containing three panels: (a) Full-pool Elo, (b) Opponent selection, and (c) Token usage.
Section C reports detailed ablation results. Table 6 evaluates opponent‑selection strategies, Table 7 varies feedback types, and Table 8 varies small‑match batch sizes, all using the 128/16 budget.
Section D quantifies token consumption. Table 9 shows that total tokens vary over fivefold across models, with GPT6‑sol using only 260 M tokens versus Qwen 3.8’s 1 454 M, yet both achieve four gold medals.
Section E classifies the decision code of retained champions into six algorithm families. Table 10 displays the proportion of non‑comment lines belonging to each family. Rules and heuristic scoring dominate (≈ 79 %), while lookahead search appears in only seven champions.