Self-Improving Agents: A Survey

Zhe Ren, Yimeng Chen, Dandan Guo, Guowei Rong, Tonghui Li, R. B. Xiong, Qingfeng Lan, Wenyi Wang, Li Nanbo, Yibo Yang, Mingchen Zhuge, Jürgen Schmidhuber

A unified taxonomy and formal framework for autonomous agents that modify their own parameters or operational structures.

How do modern autonomous agents systematically improve their own capabilities through either parameter updates or scaffolding modifications?

Autonomous agents often struggle to adapt to new tasks without constant human intervention, as their underlying logic and capabilities remain static after deployment. The authors formalize self-improvement as a self-induced update operator that modifies either the agent's foundation model parameters or its surrounding operational scaffold—such as prompts, memory, and tools—to durably improve performance. This survey categorizes existing research into these two distinct pathways, tracing their evolution from classical AI concepts to modern, language-native agentic systems.

Paper Primer

The paper defines an agent as a configuration of a foundation model $\theta$ and an operational scaffold $\Sigma$. Self-improvement is the process of applying an update operator $U$ to these components based on signals generated during the agent's own execution, such as interaction trajectories, critiques, or verification outcomes.

Foundation model improvement targets the neural parameters $\theta$ to consolidate capabilities over long time scales, while scaffolding improvement targets the structural components $\Sigma$ to rapidly reshape the agent's observation and action semantics without retraining the core model.

Modern foundation models have fundamentally shifted the self-improvement paradigm by providing a unified natural-language interface for reasoning and self-modification.

The authors demonstrate that this language-native substrate drastically reduces the search space for viable modifications compared to historical attempts that required searching through raw synaptic weights or assembly-like code. High; this shift enables the scaling of self-improving agents from narrow, hand-engineered prototypes to open-ended, real-world applications.

Why is it necessary to distinguish between foundation model improvement and scaffolding improvement?

These pathways operate on different time scales and computational costs: foundation model updates provide stable, global capability gains, whereas scaffolding updates offer fast, reversible, and targeted structural adaptations.

What is the primary risk identified in achieving autonomous self-improvement?

The authors note that agents interacting with limited resources and irreversible actions face significant risks, including self-deception, reward tampering, and the generation of fatal errors if the self-modification process lacks robust internal evaluation and safety constraints.

Researchers should view self-improvement as a nested loop across both parameters and scaffolding, moving away from treating these as isolated problems toward a unified, self-referential architecture.

The Evolution of Self-Improving Agents

Self‑improving agents evolve their configuration by updating model parameters or scaffolding.

Self‑improving agents aim to autonomously evolve their configuration $A_t = (\theta_t, \Sigma_t)$ by updating either the foundation‑model parameters $\theta$ or the external scaffolding $\Sigma$ to boost performance from experience. Early theoretical work—Schmidhuber’s self‑referential learning (1987), the Gödel Machine (2003), fast‑weight learning (1992), and meta‑learning systems (2001)—established the possibility of agents that rewrite themselves. Modern foundation models provide a language‑native interface that dramatically shrinks the modification search space, giving rise to two distinct pathways: Foundation Model Improvement (updating $\theta$) and Scaffolding Improvement (updating $\Sigma$ such as prompts, memory, or tools).

**Figure 1.** Overview of self-improvement paradigms for modern AI agents. We categorize existing methods into two primary pathways according to what is modified. The first pathway is Foundation Model Improvement, where the model parameters are updated from $\theta_t$ to $\theta_{t+1}$ using intrinsic generative demonstrations $\mathcal{D}_t$, intrinsic evaluative feedback $e_t$, or extrinsic exploratory experience $\tau_t$. The second pathway is Scaffolding Improvement, where the operational scaffold is updated from $\Sigma_t$ to $\Sigma_{t+1}$ through non-parametric changes Across scaffold components, a generic update signal $\mathcal{S}_t$ is instantiated to drive improvements in prompts $p_t$, memory $m_t$, tools $\mathcal{T}_t$, or the full scaffolding $\Sigma_t$.

**Table.** Summary of notation used in the paper, categorized into "General agent formalism" and "Foundation model improvement".

**Table.** Summary of symbols and notation used in the paper.

The field is moving from static, hand‑crafted prototypes toward agents that autonomously evolve their own configuration.

Formalizing the Agent State

Clarifies the three main ways self‑improving agents are upgraded.

Surveying the literature reveals a clear split: most work either fine‑tunes the underlying foundation model or reshapes the external scaffolding that surrounds it. A smaller but growing strand treats the agent as a joint configuration, tweaking both sides in concert.

This camp treats the agent’s core neural network (the foundation model) as the primary lever. Methods fine‑tune or adapt the parameters $\theta$ of a pre‑trained model, often using task‑specific data or reinforcement signals. The survey reports that such approaches dominate benchmarks in language modeling, vision‑language tasks, and planning, typically yielding modest but consistent gains (≈1–3 % absolute improvement) while incurring high compute cost. They differ from other camps by keeping the scaffolding (prompts, memory, tools) fixed.

This camp focuses on the external components $\Sigma$ that interface with the foundation model: prompts, memory stores, toolkits, and retrieval mechanisms. Rather than altering $\theta$, researchers redesign or augment these modules to steer the same model toward better behavior. The survey finds that prompt‑engineering, memory‑augmented retrieval, and tool‑use modules achieve comparable or larger gains than FM‑level tuning on low‑resource tasks, with negligible compute overhead.

This camp treats the agent as a joint tuple $A_t = (\theta_t, \Sigma_t)$ and updates both components jointly or in alternating phases. The survey records several systems that coordinate FM‑level fine‑tuning with scaffold adaptation, aiming for synergistic gains. Reported results show that joint updates can surpass the sum of isolated FM and scaffold improvements, especially in long‑horizon interactive tasks.

Think of an agent as a two‑part machine: a core engine (the foundation model) and a set of accessories (prompts, memory, tools). Changing either part changes the agent’s behavior, and the paper’s formalism captures this with the tuple $A_t = (\theta_t, \Sigma_t)$.

A Unified Taxonomy of Improvement

We map the landscape of self‑improving agents and highlight the split between model‑level and scaffold‑level updates.

The field converges on two orthogonal update axes: tweaking the foundation‑model parameters versus reshaping external scaffolding such as prompts, memory, or tools.

A learning signal is any observable that tells the agent whether its recent action improved performance, guiding the next update.

This camp treats the foundation model itself as the mutable substrate. Researchers inject new data, fine‑tune on task‑specific corpora, or apply self‑supervised objectives that reshape $\theta$ while keeping the external scaffolding fixed.

This camp leaves $\theta$ untouched and instead optimizes the surrounding “scaffolding” $\Sigma$: prompts, memory stores, tool‑use policies, or full external pipelines. The goal is to extract more capability from a fixed foundation model.

**Figure 3.** A unified taxonomy of self-improving agents spanning foundation-model updates, scaffold updates, and evaluation benchmarks.

Updating Foundation Model Parameters

Survey of how agents upgrade their core model versus external scaffolding.

The community converges on two complementary routes: (1) directly fine‑tune the foundation‑model parameters, and (2) enrich the surrounding scaffolding (prompts, memory, tools). Most methods excel at one side, leaving a clear gap for hybrid approaches that jointly adapt both.

Changing the internal weights $\\theta$ of a large language model so that it performs better on downstream tasks, without altering any external components.

How does “Foundation Model Improvement” differ from “Scaffolding Improvement”?

FM improvement rewrites the model’s internal parameters $\\theta$, affecting every downstream inference pass. Scaffolding improvement only tweaks the surrounding context $\\Sigma$ (prompts, memory stores, tool selectors), leaving $\\theta$ untouched and thus limiting the scope of change to the specific interaction pattern.

Methods that attach an external memory module to a frozen or lightly‑fine‑tuned foundation model, enabling the agent to store and retrieve episodic information across long horizons. The memory is queried by the model during generation, effectively extending its context window.

Architectures that focus on how stored memories are selected, filtered, and integrated before being fed back to the model. Rather than adding capacity, they improve the quality of the signal extracted from existing memory stores.

Approaches that equip agents with external tools (APIs, code interpreters, vision modules) and learn when and how to invoke them. The scaffolding $\\Sigma$ is enriched with a tool‑selection policy.

Fine‑grained routing mechanisms that decide at inference time which tool to invoke for each sub‑goal, often using a learned classifier or a planning graph.

Loops that repeatedly invoke a tool, evaluate its output, and feed the result back into the model for further refinement, akin to self‑debugging.

Systems that synthesize new tool implementations on the fly, often by generating code or API wrappers, and then immediately integrate them into the agent’s toolbox.

End‑to‑end pipelines that jointly evolve prompts, memory stores, tool libraries, and even meta‑learning strategies, treating the entire scaffolding $\\Sigma$ as a learnable object.

**Algorithm 1: Foundation-Model Improvement** **Input:** Initial agent state $\mathcal{A}_0 = (\theta_0, \Sigma)$; maximum iterations $T$ **Output:** Improved agent $\mathcal{A}_T = (\theta_T, \Sigma)$ 1. **for** $t \leftarrow 0$ to $T - 1$ **do** 2.   // 1) Construct or collect learning signal $\mathcal{S}_t$ 3.   $\mathcal{S}_t \leftarrow \emptyset$ 4.   **if** subcategory includes intrinsic generative demonstrations **then** 5.     $\mathcal{S}_t \leftarrow \mathcal{S}_t \cup \text{GENERATEDEMONSTRATIONS}(\mathcal{A}_t)$ 6.   **if** subcategory includes intrinsic evaluative feedback **then** 7.     $\mathcal{S}_t \leftarrow \mathcal{S}_t \cup \text{GENERATEEVALUATIVEFEEDBACK}(\mathcal{A}_t)$ 8.   **if** subcategory includes extrinsic exploratory experience **then** 9.     $\mathcal{S}_t \leftarrow \mathcal{S}_t \cup \text{COLLECTEXPERIENCE}(\mathcal{A}_t, \text{env})$ 10.   // Optional: quality control / weighting of signals 11.   $\mathcal{S}_t \leftarrow \text{FILTERORWEIGHT}(\mathcal{S}_t)$ 12.   // 2) Parameter update 13.   $\theta_{t+1} \leftarrow \text{UPDATE}(\theta_{1:t}, \mathcal{S}_t)$ 14.   // 3) State update 15.   $\mathcal{A}_{t+1} \leftarrow (\theta_{t+1}, \Sigma)$ 16.   **if** $\text{CONVERGED}(\mathcal{A}_{t+1})$ **then** 17.     **break** 18. **return** $\mathcal{A}_T$

Updating Agent Scaffolding

Scaffolding Improvement upgrades prompts, memory, and tools while keeping the foundation model fixed.

Across recent work, the field converges on treating prompts, memory, and tool interfaces as mutable scaffolding that can be iteratively refined, yet a unified framework for jointly optimizing all three remains lacking.

Improving an agent’s external scaffold—its prompts, memory structures, and tool bindings—while leaving the underlying foundation‑model parameters unchanged.

**Algorithm 2: Scaffolding Improvement** Input: Initial agent state $\mathcal{A}_0 = (\theta, \Sigma_0)$; max iterations $T$ Output: Improved agent $\mathcal{A}_T = (\theta, \Sigma_T)$ 1. for $t \leftarrow 0$ to $T - 1$ do 2. $\quad$ // 1) Collect scaffolding learning signal $\mathcal{S}_t$ 3. $\quad$ $\mathcal{S}_t \leftarrow \text{INTERACTANDEVALUATE}(\mathcal{A}_t, \text{env})$; // traces, critiques, success/failure, cost 4. $\quad$ // Optional: quality control / weighting of signals 5. $\quad$ $\mathcal{S}_t \leftarrow \text{FILTERORWEIGHT}(\mathcal{S}_t)$ 6. $\quad$ // 2) Scaffolding update (keep $\theta$ fixed) 7. $\quad$ if subcategory = Full scaffolding then 8. $\quad$ $\quad$ $\Sigma_{t+1} \leftarrow \text{IMPROVE}_{\Sigma}(\Sigma_{1:t}; \mathcal{S}_t)$; // § 6.4 9. $\quad$ else 10. $\quad$ $\quad$ $p_{t+1} \leftarrow p_t; m_{t+1} \leftarrow m_t; T_{t+1} \leftarrow T_t$ 11. $\quad$ $\quad$ if subcategory includes Prompt-based then 12. $\quad$ $\quad$ $\quad$ $p_{t+1} \leftarrow \text{IMPROVE}_p(p_t; \mathcal{S}_t)$; // § 6.1 13. $\quad$ $\quad$ if subcategory includes Memory-based then 14. $\quad$ $\quad$ $\quad$ $m_{t+1} \leftarrow \text{IMPROVE}_m(m_t; \mathcal{S}_t)$; // § 6.2 15. $\quad$ $\quad$ if subcategory includes Tool-based then 16. $\quad$ $\quad$ $\quad$ $T_{t+1} \leftarrow \text{IMPROVE}_T(T_t; \mathcal{S}_t)$; // § 6.3 17. $\quad$ $\quad$ $\Sigma_{t+1} \leftarrow (p_{t+1}, m_{t+1}, T_{t+1})$ 18. $\quad$ // 3) State update 19. $\quad$ $\mathcal{A}_{t+1} \leftarrow (\theta, \Sigma_{t+1})$ 20. return $\mathcal{A}_T$

Methods that treat the prompt or system instruction $p_t$ as a learnable artifact. By feeding interaction traces $\mathcal{S}_t$ into a meta‑learner, the prompt is rewritten to better align model outputs with task objectives, all without touching $\theta$.

Approaches that augment or rewrite the agent’s episodic or external memory $m_t$. The memory stores retrieved facts, plans, or tool results, and improvement functions $\text{IMPROVE}_m$ modify retrieval policies or insert new entries based on $\mathcal{S}_t$.

Techniques that expand or refine the set of external tools $T_t$ the agent can invoke. Improvement functions $\text{IMPROVE}_T$ adjust tool descriptors, API wrappers, or selection policies based on interaction outcomes.

Jointly optimizes prompts, memory, and tools in a single loop. The improvement function $\text{IMPROVE}_{\Sigma}$ consumes the full learning signal $\mathcal{S}_t$ and produces an updated scaffold tuple $(p_{t+1}, m_{t+1}, T_{t+1})$.

Application Domains and Loops

We map how agents improve themselves via model updates or scaffold edits and highlight the field’s consensus and gaps.

Self‑improving agents repeatedly adjust their configuration $A_t = (\theta_t, \Sigma_t)$, either by tuning the foundation‑model parameters $\theta$ or by revising the surrounding scaffolding $\Sigma$. These updates are driven by signals the agent generates during its own execution.

Across surveyed work, the community agrees that model‑level updates yield broad, durable capability gains, while scaffold‑level edits offer rapid, reversible adaptation—but the field lacks systematic benchmarks to compare their long‑term trade‑offs.

This camp targets the parameters $\theta$ of the underlying foundation model while keeping the agent‑level scaffolding $\Sigma$ fixed. The agent repeatedly executes its policy $\pi_{\theta_t,\Sigma_t}$, collects self‑generated signals (rewards, preferences, synthetic labels), and feeds them to a learning algorithm $U_{\theta}$ that updates $\theta$. The process operates on long time‑scales, incurs substantial compute, and produces global changes to the model’s representations and capabilities.

This camp keeps the foundation‑model parameters $\theta$ fixed and instead modifies the surrounding scaffolding $\Sigma$, which includes prompt templates, memory organization, tool interfaces, and execution constraints. The agent’s own execution $E$ produces meta‑level artifacts (new prompts, reordered memory slots, tool‑call specifications) that $U_{\Sigma}$ commits as durable edits. Changes are fast, often reversible, and can be highly context‑dependent.

Researchers diverge on which dimension matters most: some prioritize the broad, lasting capability gains of parameter updates, while others emphasize the low‑cost, quickly reversible nature of scaffold edits.

Evaluating Self-Improvement

Self‑improving agents split into two dominant camps: updating the foundation model versus updating the external scaffold.

Across the surveyed literature, agents either rewrite their neural weights or reconfigure their external scaffolding, and the field lacks a unified benchmark that jointly evaluates cost, speed, and transparency of these two routes.

This camp updates the parameters $\theta$ of the underlying foundation model while keeping the scaffold $\Sigma$ fixed. Updates are driven by self‑generated signals such as synthetic demonstrations, internal critiques, or interaction trajectories, and are applied with gradient‑based or fine‑tuning procedures that embed new capabilities directly into the model.

This camp modifies the explicit, inspectable components $\Sigma$ (prompts, memory entries, routing rules, or full orchestration logic) while keeping the foundation model frozen. The updates are usually rule‑based, search‑driven, or generated by lightweight policies, enabling rapid, reversible adaptation without touching the large neural weights.

Extended FM Improvement Analysis

Foundation‑model improvement clusters into three signal types, each with distinct trade‑offs.

Self‑improvement via foundation models follows three dominant pathways: (i) generating its own demonstrations, (ii) critiquing its own outputs, and (iii) learning from interaction‑derived trajectories.

**Figure 6.** Overview of foundation model improvement under agent-induced learning signals. The agent improves the foundation model by generating intrinsic demonstrations, producing intrinsic evaluative feedback, or collecting extrinsic exploratory experience, each forming a distinct parameter-update loop.

**Figure 7.** Prompt refinement as a self-improvement loop and its four paradigms, organized by the learning signal $S_t$.

**Table 2.** Comparison of prompt optimization paradigms in prompt-based self-improvement. As learning signals become more structured and informative, optimization becomes less heuristic and more automated. (1): Scalar-feedback optimization; (2): Qualitative-feedback refinement; (3): Population-based evolution; (4): Textual-gradient optimization.

This camp treats the foundation model as both learner and data synthesizer. The agent samples a generative distribution $P_{\text{gen}}(x,y\mid A_t)$, filters the resulting set $\tilde D_{\text{gen}}$, and fine‑tunes on the union $D_t = D_{\text{base}}\cup\tilde D_{\text{gen}}$. The loop is fully self‑contained: no external annotations are required, and updates are performed by standard gradient‑based fine‑tuning.

This group converts the model’s own judgments into learning signals. Candidates $Y_t(x)$ are produced, then an internal evaluator $\phi_t$ produces a signal $e_t$—scalar scores, preferences, confidence estimates, or natural‑language critiques. The signal $S_t\approx e_t$ drives reinforcement‑learning, preference‑optimization, or critique‑conditioned fine‑tuning updates.

Here the learning signal $S_t\approx\tau_t$ is derived from trajectories collected by acting in an environment (grounded or simulated). The agent updates $\theta$ using reinforcement learning, preference learning, or supervised fine‑tuning on the observed outcomes. Both direct interaction (e.g., code execution with unit tests) and world‑model simulation are covered.

Extended Scaffolding Analysis

Scaffolding improvements span memory, structure, processing, tools, and full code, each with distinct trade‑offs.

Scaffolding research converges on five axes—memory objects, structural organization, processing policies, tool governance, and full‑code mutation—while a persistent gap remains: implicit memory offers speed but lacks interpretability.

**Figure 8.** Overview of memory for self-improving agent.

Table 3 presents a qualitative scorecard of memory‑object types, rating fidelity, interpretability, compactness, write cost, and auditability on a 1‑to‑5 scale and listing their most common failure modes.

Memory objects are the atomic units an agent writes to, reads from, and updates. They fall into two families: explicit objects that retain human‑readable content (processed trails, curated raw artifacts, integrated external knowledge) and implicit latent embeddings that reside in model‑native tensors. Explicit objects excel at auditability and debugging, while implicit objects provide constant‑time associative access at the cost of transparency.

Table 4 compares 24 memory‑augmented agent frameworks across object type, structural topology, CRUD capabilities, and governance mechanisms, highlighting the diversity of design choices.

Beyond the objects themselves, a memory structure defines how those objects are indexed, linked, and retrieved. Four paradigms dominate current designs, each trading scalability, expressiveness, and latency.

Memory processing comprises the signal‑driven operations that evolve the memory state $m_t = (\text{object}_t,\text{structure}_t)$. Four core primitives—Creation, Reading, Updating, Deletion—are orchestrated by the learning signal $S_t$.

The figure illustrates a conceptual framework centered on "Tool Governance Metacognition," represented by an icon of a toolbox containing various tools. This central concept is framed by three pillars forming a triangle: "Autonomous Tool Creation" on the left, "Iterative Tool Refinement" on the right, and "Dynamic Tool Routing" at the base.

Tool governance treats the agent’s toolbox as a mutable resource. Three intertwined capabilities—Dynamic Tool Routing, Iterative Tool Refinement, and Autonomous Tool Creation—enable agents to select, improve, and invent tools without external supervision.

**Figure 10.** Full scaffolding self-improvement across iterations.

Full scaffolding treats the entire operational code $Σ_t$ as mutable. Updates are expressed as patches $Δ_t$ applied to a serializable program representation $⟨Σ_t⟩$, with verification $V( \cdot )$ ensuring safety before adoption.

Table 5 synthesizes application‑arena self‑improvement loops (software engineering, web automation, games, science, embodied AI, general computer control), showing how sandboxed feedback shapes dominant bottlenecks, improvement targets, and iteration modes.

Discussion and Future Directions

We outline the limits of structural versus scaffolding updates and detail the improvement algorithm.

Structural updates modify the agent’s intrinsic configuration ($\theta_t$) and are permanent, whereas changes to working memory are transient and disappear after a single episode.

Scaffolding improvement is broken into four targeted operations: Prompt Optimization ($p_t arrow p_{t+1}$) edits prompts; Memory Evolution ($m_t arrow m_{t+1}$) reshapes storage and retrieval; Tool Governance ($T_t arrow T_{t+1}$) expands or refines the tool set; and Full Scaffolding Update ($\Sigma_t arrow \Sigma_{t+1}$) reconfigures the entire operational architecture.

Algorithm 1 iterates over $T$ steps, each time constructing a learning signal $S_t$ (by generating demonstrations, evaluative feedback, or collecting experience), optionally filtering it, updating the foundation model parameters $\theta_{t+1} \leftarrow \text{UPDATE}(\theta_{1:t}, S_t)$, and checking for convergence before proceeding.

Extended Applications and Benchmarks

Survey of domains where self‑improving agents are applied and how they are evaluated.

Self‑improving agents are deployed across a spectrum of domains, each exposing distinct feedback signals and improvement pathways.

Web navigation and automation confront dynamic pages, long horizons, and weak verification, making one‑shot prompt engineering brittle and motivating both FM improvement (updating $\\theta$) and scaffolding improvement (updating $\\Sigma$).

Evaluation substrates such as Mind2Web, WebArena, VisualWebArena, WorkArena, and BrowserGym provide reproducible environments, demonstrations, and execution‑based success criteria that serve as the data and measurement backbone for studying improvement loops.

Scaffolding improvement in web settings is exemplified by WebCoach, which augments cross‑session memory to avoid repeated mistakes, and ReAP, which stores and reuses reflections over trajectories, thereby updating memory and retrieval policies without touching the foundation model.

FM improvement leverages interaction feedback: Patel et al. fine‑tune on model‑generated data in WebArena, OpenWebVoyager iterates on real‑world browsing policies, and reinforcement‑learning systems like WebRL and WebAgent‑R1 use outcome‑supervised reward modeling to boost capability.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers