SPADE ♠: Self-Play in Adaptive Synthetic Executable Environments

Bo Liu, Simon Yu, Yiding Jiang, Ao Qu, Andrew Zhao, Zichen Liu, Junsu Kim, Zijian Zhou, Seungone Kim, Tongzheng Ren, Mickel Liu, Hanfei Yu, Zhaorun Chen, Weiyan Shi, Paul Pu Liang, Luke Zettlemoyer, Yejin Choi, Natasha Jaques

SPADE co-evolves an LLM as both environment designer and agent, using hint-based regret to sustain open-ended learning.

How can a single LLM simultaneously generate diverse, adaptive training environments and learn to solve them to achieve continuous self-improvement?

Language agents stop improving once they exhaust their fixed training environments, as human-curated or static synthetic pools cannot adapt to the agent's growing capabilities. SPADE solves this by training a single LLM to play two roles: an Environment Designer that writes executable Python code for new training tasks, and a Reasoning Agent that learns to solve them. The designer is trained via reinforcement learning to maximize a "hint-based regret" signal, which targets environments that are solvable with a hint but difficult without one. This co-evolutionary loop allows the environment distribution to shift alongside the agent's frontier, yielding significant gains on held-out math, science, and tool-use benchmarks compared to static baselines.

Paper Primer

The core mechanism is a dual-role self-play loop where the Environment Designer emits full Markov Decision Processes (MDPs) as Python programs. By representing environments as code with a standard `reset()`/`step()` interface, the system unifies single-turn reasoning and multi-turn tool use into a single, learnable training pipeline.

SPADE significantly outperforms fixed-environment baselines on held-out reasoning and tool-use benchmarks.

At 30B-parameter scale, SPADE improves the suite average by +5.3% on games and +13.9% on ACEBench-Agent compared to the strongest fixed-environment baselines. +13.9% gain on ACEBench-Agent; +5.3% average gain across eight held-out benchmarks.

The Environment Designer uses two key inputs to avoid mode collapse: corpus grounding (sampling from pretraining documents to seed new tasks) and environment memory (a buffer of past tasks to maintain difficulty). This prevents the designer from re-generating mastered tasks or collapsing into repetitive, trivial outputs.

Why use hint-based regret instead of just rewarding the agent for solving harder tasks?

Purely rewarding difficulty can lead to unsolvable environments, while rewarding success can lead to trivial ones. Hint-based regret targets the "learning frontier"—environments where the agent fails without help but succeeds with it—ensuring tasks remain both challenging and feasible.

Does this approach require a separate model for the environment designer?

No, a single LLM plays both roles using role-specific system prompts. The policy parameters are shared, meaning updates to the Reasoning Agent's capabilities immediately inform the Environment Designer's next generation cycle.

By turning environment design into a learnable RL component, SPADE demonstrates that agents can generate their own curricula, effectively bypassing the bottleneck of finite human-curated training data.

Introduction: The Need for Adaptive Environments

Static training environments stall language agents, and SPADE proposes a self‑play framework to keep them improving.

Language agents quickly exhaust fixed pools of hand‑curated or synthetically generated environments, causing their performance to plateau as they scale. Existing pipelines either keep the environment set static or rely on generators that do not adapt to the agent’s growing capabilities, creating a mismatch between task difficulty and agent skill.

SPADE lets a single LLM wear two hats—an Environment Designer that writes full, executable training worlds, and a Reasoning Agent that learns to solve them—so the training distribution evolves together with the agent.

**Figure 1.** SPADE designs and solves its own training environments in both settings. Left: a single LLM $\pi_\theta$ plays both roles, an Environment Designer that writes an executable environment $e$ with a privileged hint $h$, and a Reasoning Agent that solves $e$ with and without $h$; the return gap rewards the Environment Designer (hint-based regret), task completion rewards the Reasoning Agent, and both update the same weights. Middle: average relative improvement over the untrained base across the eight games-setting evals ((score - base)/base), versus Fixed-env RLVE (orange) and Fixed-env GRPO (gray). Right: per-benchmark relative improvement of SPADE-30B-A3B on the three tool-use evals ($\tau^2$-bench, BFCL v4 multi-turn, and ACEBench-Agent; Table 2).

**Figure 2** SPADE generates an adaptive, multi-turn curriculum. Four environments the Environment Designer produces over one 30B-A3B run, from step 0 (early) to step 384 (late); each card shows the agent's first observation, the generated Python environment, and the designer-written hint. Every environment is a complete MDP with a reset()/step() interface, and the tasks shift toward state-gated, multi-turn interaction as the Reasoning Agent improves. Unlike a fixed human-curated pool or a frozen synthetic generator, this curriculum keeps moving with the learner.

The core limitation of static training environments is that they cannot keep pace with a language agent’s growing reasoning and tool‑use abilities.

Related Work

We situate SPADE among prior self‑play, environment‑design, and synthesis works.

Prior work on self‑play, unsupervised environment design, and large‑scale environment synthesis provides the backdrop for SPADE’s unified curriculum generation.

A fixed‑environment baseline trains the Reasoning Agent on a static set of environments, offering no adaptation as the agent improves.

Early self‑play systems (TD‑Gammon → AlphaZero) inspired a lineage of LLM self‑play methods, ranging from task‑generation to dual‑role single‑LLM setups.

UED research treats environment creation as a learnable process, often using evolutionary or adversarial search to drive agent improvement.

Scaling LLM training has motivated systems that synthesize environments at massive scale, often as multi‑turn, verifiable MDPs.

Preliminaries

Defines the MDP interface and RLVR training pipeline that power SPADE.

Static, hand‑crafted environments quickly saturate an LLM’s reasoning ability, forcing the training set to become a bottleneck. SPADE needs a lightweight contract that lets it spin up fresh worlds on demand and query them step‑by‑step. The MDP Interface provides exactly that minimal programmatic bridge.

The MDP Interface is a tiny API that lets a learning agent start a new episode and then advance it, exposing the current state, reward, and explicit termination flags at each step.

How does this MDP Interface differ from the classic OpenAI Gym API?

Classic Gym returns a single “done” flag that conflates natural episode termination with time‑limit truncation. SPADE’s interface splits them into terminated (task solved) and truncated (forced stop), which is crucial for curricula that adapt episode length.

Call

The explicit terminated flag tells the agent it has achieved the goal, while truncated would signal a time‑limit stop; this distinction lets SPADE know whether a curriculum needs to be lengthened.

Training proceeds with RLVR: for each prompt the policy samples a group of $G$ responses, computes group‑normalized advantages $\hat{A}_i$, and updates $\theta$ via a clipped policy gradient with KL regularization (GRPO). This pipeline turns verifiable rewards into stable policy improvement steps.

The SPADE Framework

How a single LLM alternates between creating environments and solving them to keep learning moving.

Static training sets quickly saturate a model’s abilities, causing learning to stall. SPADE solves this by letting the same LLM generate new, solvable environments and then practice on them, keeping the curriculum aligned with the agent’s current skill.

**Figure 3.** The SPADE framework. Top: the Environment Designer conditions on the environment memory $M$ and pretraining corpus $C$ to emit an executable environment $e$ and a privileged hint $h$. Bottom: the Reasoning Agent plays $e$ with and without $h$; the return gap is the Environment Designer's hint-based regret $r_D(e)$ (Eq. 3) and task correctness is the Reasoning Agent reward. Both rewards update the shared policy $\pi_\theta$ via GRPO.

The Designer is a LLM prompt that writes a tiny Python program (a Gym‑style reset/step) and attaches a short, task‑relevant hint $h$.

The Agent is the same LLM prompted to act as a player, issuing sequential actions against the Python environment produced by the Designer.

Sample a passage from the external corpus $C$ and retrieve the memory buffer $M$.

Prompt the shared policy $\pi_\theta$ (Designer role) to emit an executable environment $e$ and a privileged hint $h$.

Validate $e$ for syntactic correctness and executability; discard if it fails.

Run $G$ rollouts of the Reasoning Agent on $e$ without the hint, record average return $\overline{r}_A(e)$.

Run $G$ rollouts on the same $e$ with the hint $h$, record average return $\overline{r}_A(e\mid h)$.

Compute the Hint‑Regret Reward $r_D(e)=\overline{r}_A(e\mid h)-\overline{r}_A(e)$ and feed it to the Designer update (after a delay of $k$ rollouts).

Update the shared policy $\pi_\theta$ with gradients from both the Agent’s task reward and the Designer’s regret reward.

Rollout 1 without hint: agent fails → win indicator 0.

Rollout 2 without hint: agent fails → win indicator 0.

Average return without hint $\overline{r}_A(e)=0$.

Rollout 1 with hint $h$: agent succeeds → win indicator 1.

Rollout 2 with hint $h$: agent succeeds → win indicator 1.

Average return with hint $\overline{r}_A(e\mid h)=1$.

Hint‑Regret Reward $r_D(e)=1-0=1$, indicating a high‑regret environment at the learning frontier.

This toy example shows that a positive regret signals an environment that is solvable with a hint but not without, exactly the kind of curriculum SPADE seeks.

Because the same policy is used for two different objectives, we keep their learning signals on comparable scales.

**Figure 4** How a privileged hint changes Reasoning Agent play. Two positive-regret examples from the canonical 30B games run. Left: each environment's task prompt and privileged hint, quoted verbatim (ellipses mark elided text; the standardized answer-format sentence is omitted from the hint). Right: one logged Reasoning Agent rollout per arm, condensed while preserving action order, feedback, and values; elided turns are marked and named. The two arms are independent plays with independently seeded resets, so board layouts and probe outcomes differ across arms. The dashed box on each environment card reports the two displayed rollout returns and their single-pair gap; the Environment Designer reward in Equation 3 is instead the difference of the arm means over all logged rollouts (0.00 → 1.00 for the fiber task, 0.30 → 0.65 for the audio task). An expanded task-hint set appears in Appendix G.1.1.

Experimental Setup

SPADE post‑training yields a +13.9 ACEBench‑Agent boost, demonstrating its advantage across benchmarks.

SPADE post‑training improves the ACEBench‑Agent average score by +13.9 over the same backbone without SPADE.

Table 2 shows the SPADE‑enhanced Qwen3‑30B‑A3B‑Instruct‑2507 achieving a +13.9 increase in the ACEBench‑Agent average column.

We train three Qwen3 backbones (4B, 8B, 30B) using GRPO for 400 rollouts of 24 environments each. The Environment Designer refreshes the environment set every $k$ rollouts, and its update is delayed by the same $k$, so the Reasoning Agent sees a fixed curriculum for $k$ steps before the next shift.

In the games domain, each rollout contains 24 games covering three of six cognitive‑skill categories, with $k=4$ governing the regeneration interval. The Environment Designer writes each game as a self‑contained Python script grounded in a 15 k‑document math‑science corpus.

**Table 1.** Performance comparison of various agents across BFCL v4, $\tau^2$-bench, and ACEBench-Agent benchmarks.

For tool‑use environments, the Designer emits OpenAI‑function‑calling specifications and a sequence of user instructions; the Reasoning Agent must call tools over multiple turns, receiving a privileged hint that outlines a step‑by‑step plan. Generation uses a larger curriculum interval $k=8$ because tool environments are costlier to produce.

Experimental Results

SPADE continuously improves the Reasoning Agent by co‑evolving adaptive environments.

Recall that SPADE lets a single LLM generate and solve its own training environments, keeping the Reasoning Agent in step with an ever‑evolving curriculum.

SPADE improves the suite‑average score by +8.1 % over the base model and +5.3 % over the strongest fixed‑environment baseline (Fixed‑env RLVE).

Table 1

In the tool‑use setting, SPADE yields the largest gains where the benchmark structure mirrors the generated environments: ACEBench‑Agent sees +13.9 %, BFCL v4 multi‑turn +5.7 %, and $\tau$ 2‑bench +3.6 % at 30B‑A3B (BFCL v4 also gains +10.3 % at 4B).

Procedural‑reasoning skills benefit most: every cognitive‑skill category improves because the synthetic games expose many problem structures, unlike a static task set.

**Table 1.** Training the Reasoning Agent on diverse synthetic games improves held-out reasoning and code benchmarks at every backbone scale. The eight held-out benchmarks probe four capability families: competition math (AIME 2025/2026, Avg@32), science reasoning (GPQA-Diamond, accuracy), code generation (LiveCodeBench-v6, Pass@1), and procedural reasoning across four cognitive skills (Reasoning-Gym, win rate at HARD). Both fixed-environment baselines are retrained per backbone from the same base model for 400 training iterations; Fixed-env RLVE follows the official RLVE sampling and curriculum settings. Avg is the unweighted mean over the eight benchmarks; green subscripts denote absolute pp gain over the same-model base. Best per column within each backbone block in bold, second best underlined.

**Figure 6.** Full SPADE raises the learnable share of its environment budget to roughly a third by the end of training; component ablations decline or collapse. Share of each rollout's 24 environments that is learnable, defined as Reasoning Agent win rate in [0.2, 0.8] and weighted by the number of valid environments generated in each 16-step window. Matched 30B-A3B settings over a common 400-step budget; unfilled rollout capacity contributes zero by construction.

**Figure 8.** Trained Environment Designer environments stop revealing the solution method in the prompt. Physics environments from one 30B-A3B run at steps 20, 192, and 384; the step-20 environment retains the scaffold's default class name. The formula-reveal rate (percentage on each panel) falls from 25% to 5% over 473 environments. Rightmost (red): over steps 290–312 the no-corpus ablation emits the same RotatingMazeEnv task 41 consecutive times. The complete source of the step-384 environment, together with one further exemplar, appears in Appendix G.3.2.

Additional quantitative signals support the qualitative picture: Vendi/n diversity stays high (0.68) with corpus grounding but falls to 0.04 without it; reward granularity rises from 3.7 to 5.8 distinct levels per environment, and partial rewards increase from 2.2 to 4.0.

The Reasoning Agent’s behavior evolves over training: early on it reasons entirely up‑front, later it tests hypotheses and finally gathers evidence before deriving a solution, a shift that only appears when the task rewards inference.

Ablation Studies

We isolate each Environment Designer component and reward, measuring how their removal degrades performance.

Freezing the self‑play Environment Designer and dropping its memory makes the model worse than no training.

Average score drops to 40.5, 9.7 points below the untrained base (50.2).

A fixed GPT‑5.5 Environment Designer that retains corpus grounding and memory outperforms the untrained base.

Average score 53.0 versus 50.2, recovering about 35 % of SPADE’s gain.

Using hint‑based regret as the Environment Designer reward adds +8.1 points to the eight‑benchmark average.

Average rises from 50.2 to 58.3.

EMA‑based learning‑potential reward achieves +5.7 points, reaching an average of 55.9.

Performance climbs more slowly and stays below the hint‑regret variant.

**Table 3.** The full adaptive configuration outperforms every partial and frozen-designer control. Games setting, Qwen3-30B-A3B-Instruct-2507. Best checkpoint per variant on suite average; Avg is the unweighted mean over the same eight benchmarks as Table 1; full trajectories in Figure 10. Best in bold, second best underlined.

**Figure 9.** From front-loaded derivation to evidence-first interaction. Reasoning Agent episodes from one 30B-A3B run at steps 0, 200, and 300. At step 0 the agent derives in advance and cannot recover from format errors; by step 200 it tests short hypotheses and revises on evidence; by step 300 it probes first and derives once. Benchmark gains of late checkpoints (Table 1) confirm the model keeps its long-form derivation ability. Transcripts verbatim (math glyphs transliterated to ASCII; environment feedback abridged); token counts use the backbone's tokenizer.

**Figure 10.** Removing Environment Designer training and memory together drops self-play below base; removing either one alone has an above-base selected checkpoint but peaks early and can fall below base late. One curve per variant of Table 3 (Qwen3-30B-A3B-Instruct-2507, games setting). Top row: AIME 2025/2026 Avg@32, GPQA-Diamond accuracy, and LiveCodeBench-v6 Pass@1; bottom row: the four Reasoning-Gym categories; the dashed line marks the untrained base model.

**Table 6.** Full ablation breakdown (games setting, Qwen3-30B-A3B-Instruct-2507). Best checkpoint per variant on the suite average. AIME reports Avg@32; GPQA-D accuracy; LCB-v6 Pass@1; Reasoning-Gym (RG) win rate at HARD; GEM the overall win rate across the GEM game suite (Liu et al., 2025c). Best in bold.

Scaling Analysis

Scaling shows larger models gain far more from adaptive environments than static baselines.

Scaling model size yields larger gains from SPADE’s adaptive environments, reaching +8.1 over base at 30B‑A3B versus only +5.2 at 4B.

Average gain over each backbone’s base grows from +5.2 at 4B and +5.7 at 8B to +8.1 at 30B‑A3B, while Fixed‑env GRPO stays near +1.2.

The curriculum’s diversity drives the gains: the full six‑skill set outperforms a reduced two‑skill version, which captures only about half of the Reasoning‑Gym improvements and yields smaller GPQA‑Diamond and LiveCodeBench‑v6 gains. This shows that adding more skill categories, rather than any single game family, is responsible for the scaling effect. The best checkpoint on the eight‑benchmark suite rises from 53.7 to 58.3 as curriculum diversity increases.

Theoretical Analysis

We formalize the incentive structure of hint‑regret and characterize equilibrium behavior.

Recall that SPADE lets a single LLM generate and solve its own training environments, avoiding static curricula.

We model the interaction between the Environment Designer and the Reasoning Agent as a two‑player game: the Designer picks a distribution over environments, the Agent picks a policy, and payoffs are expected verifier returns.

A pure Nash equilibrium $(D^{\circ},\pi^{\circ})$ satisfies that neither player can improve its payoff by unilaterally changing its distribution or policy.

We now state the three technical assumptions that underlie the analysis.

Assumption B.1 (Sound generation): the Designer samples only from the executable set $M$ (i.e., $D\in\Delta(M)$).

Assumption B.2 (Articulated hints): for any policy $\pi$ and environment $e$, the hinted return reaches the optimal unhinted value, i.e., $R^{h}_{\pi}(e)=R^{\ast}(e)\ge R_{\pi}(e)$.

Assumption B.3 (Internalizability): for every policy $\pi$ there exists a policy $\pi'$ that attains the hinted return without the hint, i.e., $R_{\pi'}(e)=R^{h}_{\pi}(e)$ for all $e\in M$.

For any policy $\pi\in\Pi$ and any environment $e\in M$, $R^{h}_{\pi}(e)-R_{\pi}(e)=R^{\ast}(e)-R_{\pi}(e)$. Consequently the hint‑regret is non‑negative and strictly positive iff $R_{\pi}(e)<R^{\ast}(e)$.

Under Assumptions B.1–B.3, every pure Nash equilibrium $(D^{\circ},\pi^{\circ})$ satisfies $u_D(D^{\circ},\pi^{\circ})=0$ and $R_{\pi^{\circ}}(e)=R^{\ast}(e)$ for all $e\in M$, i.e., the Reasoning Agent is optimal on every environment and hints become vacuous.

Define $\rho_{\text{reg}}(e)=R^{h}_{\pi^{\circ}}(e)-R_{\pi^{\circ}}(e)=R^{\ast}(e)-R_{\pi^{\circ}}(e)\ge0$ for all $e\in M$ (Lemma B.4).

By Assumption B.3 there exists $\pi'\in\Pi$ with $R_{\pi'}(e)=R^{h}_{\pi^{\circ}}(e)=R^{\ast}(e)$ for all $e$.

Since $\pi^{\circ}$ is a best response to $D^{\circ}$, $u_A(D^{\circ},\pi')\le u_A(D^{\circ},\pi^{\circ})$.

Equality must hold, implying $\mathbb{E}_{e\sim D^{\circ}}[\rho_{\text{reg}}(e)]=0$.

Because the Designer can deviate to $\delta_e$, Nash equilibrium requires $\rho_{\text{reg}}(e)\le0$ for every $e$, and together with $\rho_{\text{reg}}(e)\ge0$ we get $\rho_{\text{reg}}(e)=0$.

Thus $R_{\pi^{\circ}}(e)=R^{\ast}(e)$ and $R^{h}_{\pi^{\circ}}(e)=R_{\pi^{\circ}}(e)$ for all $e\in M$, yielding $u_D(D^{\circ},\pi^{\circ})=0$.

Extended Ablations

Ablation results confirm SPADE’s benefits across backbones and reveal early‑training dynamics.

We evaluate SPADE on a non‑Qwen backbone (Nemotron‑30B‑A3B‑BF16) and on the full 6‑skill curriculum, extending the component‑controlled ablations of Table 3. All four Reasoning‑Gym categories improve over the untrained base, allowing us to isolate the impact of each component.

**Figure 14.** The full 6-skill curriculum lifts held-out benchmarks more than the restricted 2-skill variant; curriculum breadth drives the gains. Qwen3-30B-A3B-Instruct-2507, games setting. Top row: AIME 2025/2026 Avg@32, GPQA-Diamond accuracy, and LiveCodeBench-v6 Pass@1. Bottom row: the four Reasoning-Gym categories; the dashed line marks the untrained base model. Discussed in Section 8.

On the Nemotron backbone, SPADE raises the RG‑Cognition win rate by +9.6% over the untrained base.

Figure 15 reports RG‑Cognition +9.6%.

On the same backbone, the RG‑Algorithmic win rate improves by +9.3%.

Figure 15 reports RG‑Algorithmic +9.3%.

The full 6‑skill SPADE configuration attains the highest GEM overall win rate of 73.5%.

Table 6 lists a GEM overall win rate of 73.5% for the SPADE (6‑Skills) variant.

Extended Related Work

We survey self‑play, unsupervised design, synthetic generation, scaling, and memory systems for LLMs.

Self‑play has been a cornerstone of AI since TD‑Gammon (1995) and later AlphaGo, AlphaZero, OpenAI Five, and AlphaStar demonstrated its power on games, while Cicero extended it to strategic language tasks; asymmetric self‑play (Sukhbaatar et al., 2017) introduced the teacher‑student template that underlies many modern LLM curricula.

Recent LLM‑focused self‑play methods such as SPIN, Self‑Rewarding Language Models, SPAG, ReSTEM, Prover‑Verifier Games, ReMA, R‑Zero, PopuLoRA, G‑Zero, Tool‑R0, Self‑Questioning Language Models, Language Self‑Play, PasoDoble, SeRL, and many others generate tasks or environments without external data, but they rely on frozen generators or heuristic proxy rewards that can lead to reward hacking and distributional drift.

Two themes emerge across this body of work: (1) data‑free self‑play methods generate only a problem statement and a sparse terminal reward, and (2) the generator is either frozen or trained with heuristic proxy rewards that risk reward hacking and distributional drift.

Unsupervised environment design (UED) builds on curriculum learning, Quality‑Diversity (QD) algorithms such as MAP‑Elites, and POET’s co‑evolution of terrains and agents, with later works like PAIRED, PLR, Replay‑Guided Adversarial Environment Design, ACCEL, DISCOVER, and CENIE refining the regret‑based curriculum signal.

Open‑endedness research emphasizes that unbounded environment spaces are essential for continual improvement, with works such as Hughes et al., Baker et al., OMNI, OMNI‑EPIC, Imagined Autocurricula, SIMA 2, Goldfeder et al., and PAPRIKA exploring novelty, learnability, and multi‑agent autocurricula.

A growing body of work uses LLMs to programmatically generate training environments, from Agent World Model (AWM) and ScaleEnv to TermiGen, Nemotron‑Terminal, SkillSynth, Eurekaverse, EvoCUA, DreamGym, Simia, and many others, often relying on prompting heuristics rather than joint RL optimization.

Consensus is emerging that scaling the number and diversity of environments, rather than algorithmic tweaks, drives the next leap in agentic RL; seminal analyses by Silver and Sutton, AgentRL, SCALER, RLVE, WebScale‑RL, Endless Terminals, Self‑Evolving Curriculum, TTCS, and related surveys underscore this trend.

Memory‑centric approaches such as MemRL, ALMA, and HyperAgents explore self‑evolving memory architectures and self‑modifying code, complementing SPADE’s focus on generating adaptive environments for LLMs.

Extended Quantitative Analysis

Headline improvements show win‑rate and learnability gains across training.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers