Toward Recursive Self-Improvement in Physical AI
Meng Luo, Zechen Bai, Shanqing Xu, Yanlin Li, Kaixin Li, Mengchen Qian, Weifeng Lu, Ran Zhang, Yingtian Zou, Ziyang Luo, Wei Chen, Hongzhan Lin
A framework for evaluating whether robots can recursively improve their own learning procedures through physical experience.
How can we systematically evaluate whether a physical robot is actually improving its own learning process, rather than just its immediate behavior?
Robots often improve their performance through trial and error, but they typically rely on fixed learning procedures designed by humans. This limits their ability to adapt their own improvement process as they encounter new physical constraints. The authors define a ladder of self-improvement, culminating in recursive self-improvement (RSI), where a system uses its acquired repair knowledge to improve the very procedures that generate its future updates. They propose a two-round experimental protocol to isolate these recursive gains from simple behavioral learning. While existing systems demonstrate behavioral improvement and procedure refinement, the authors find that no current physical study has yet isolated gains specifically attributable to recursion.
Paper Primer
The paper positions RSI as a progression: from improving a robot's behavior, to improving the learning procedure, to finally improving the improver itself. The core move is to shift the focus from "what the robot does" to "how the robot learns to learn," using physical experience to refine the data collection, supervision, and update rules that govern future iterations.
Physical RSI requires two specific links to be established: repair knowledge must improve the improver, and the improved improver must then help produce its own next improvement.
The authors define these as R1 and R2, requiring a comparison against a frozen-knowledge control to isolate the contribution of acquired knowledge. Current studies support behavioral and procedure improvement, but none have yet passed the controlled two-round test required to confirm recursive gains.
Why is it difficult to prove that a robot is "recursively" improving rather than just getting better at a task?
A system can repeatedly improve its behavior using a fixed procedure without any recursion occurring. Recursion requires that the system's internal improvement mechanism itself changes in a way that makes subsequent revisions more effective, which must be verified by comparing matched learners with and without the acquired repair knowledge.
What are the primary physical constraints that make RSI harder for robots than for digital agents?
Physical systems face costly trials, irreversible hardware wear, calibration drift, and the need for reliable sensing to judge outcomes. Unlike digital agents that can reset to a clean state, robots must account for the fact that practice changes the physical conditions of the environment.
Researchers should move beyond reporting simple success rates and adopt the authors' two-round experimental protocol to isolate whether improvements in learning procedures are truly recursive or merely the result of a fixed, human-designed optimization loop.
Abstract
We map physical robot learning to recursive self‑improvement, outlining criteria for safe, scalable gains.
Physical robot learning is framed as recursive self‑improvement, where each improvement can tighten the process that generates it. Gains only propagate when feedback fills remaining learning gaps, the receiving component can apply the change, the value endures across task or hardware shifts, and practice stays safe and affordable. Existing manipulation work demonstrates behavioral, learning, and procedural gains, yet no physical study has isolated recursion‑driven improvements.
Introduction to Physical RSI
Framing the challenge of recursive self‑improvement for physical agents.
This section frames the problem: can a robot improve the very process by which it learns, not merely its current behavior?
RSI is a system that upgrades the learning procedure itself, so each new version of the learner benefits from the improvements made to the process that produced it.
Early ideas trace back to I. J. Good’s 1966 speculation and Schmidhuber’s Gödel machine (2003). Recent digital agents—STOP (Zelikman et al., 2024) and the Darwin Gödel Machine (Zhang et al., 2026c)—run code that rewrites its own source and evaluate candidates with executable tests.
Physical RSI asks the same question of robots that learn by acting. Four experimental factors sharpen the problem:
First, outcome checks rely on sensors, human judges, or learned detectors, each of which can miss failures and propagate errors. Second, each candidate incurs robot time, resets, and wear, so improvements must pay off within a few trials. Third, practice changes the environment: objects move, grippers wear, and calibration drifts, so a software rollback does not fix physical damage. Fourth, the scope of improvement expands beyond prompts and code to include controllers, dynamics models, and body‑interface software.
Table 1 contrasts digital and physical self‑improvement along these dimensions, highlighting that physical systems face additional constraints on sensing, cost, reversibility, and non‑stationarity.
**Figure 1. From digital to physical recursive self-improvement.** Numbers mark where experience leaves a change: policy 1, world model 2, and external knowledge 3 improve what the robot does; the learning procedure 4 changes how it learns; and the improver, drawn as the loop around the procedure, revises it. RSI further requires that repair knowledge improve the improver and that the improver, once improved, help produce its own next improvement.
Consider a robot arm that repeatedly misses a grasp. The failure can be addressed in four places: (1) the policy that issues motor commands, (2) the world model that predicts contact dynamics, (3) external knowledge such as a saved recovery routine, and (4) the learning procedure that decides how data are collected and how updates are applied.
**Figure 3. The review at a glance.** Illustrations in (b) show the four grasp breakdowns from Section 1.
The central argument is that physical experience can improve (i) task competence, (ii) the learning procedure, and (iii) the improver that revises the procedure. Four recurring conditions—informative feedback, usability, robustness to distribution shift, and continued practice—determine whether these gains propagate. Each condition can fail: sensors may miss critical errors, hardware may not accept the new knowledge, improvements may not survive later changes, or safety/cost constraints may halt further practice.
The review contributes three things: (1) it positions RSI within robot learning by separating method paradigms and defining a ladder of claims; (2) it distinguishes mechanisms that change an improver’s knowledge from evidence that the improver has improved; (3) it proposes a two‑round experiment to isolate true recursion and estimates its hardware cost.
Four research questions guide the discussion: R Q 1 – what limits a robot’s next improvement? R Q 2 – which changes address that limitation? R Q 3 – when can improvements build on one another to achieve recursion? R Q 4 – how do physical constraints reshape self‑improvement?
The key distinction is that behavioral improvement changes what the robot does, whereas process improvement changes how the robot learns; only the latter can sustain recursive self‑improvement.
Foundations of Physical RSI
We map physical systems onto a three‑stage loop to assess recursive self‑improvement.
Physical RSI studies converge on a three‑stage loop: experience → retained change → reuse, but few demonstrate that the loop closes recursively.
Methods that treat the robot controller as the primary learned component. Policies are neural networks trained via imitation, reinforcement learning, or fine‑tuning, often using large pretrained models (RT‑1, RT‑2, Octo). They output actions directly from observations and are evaluated by success rates on grasping or manipulation tasks.
Techniques that learn a dynamics or contact model predicting the consequences of actions. The model can be a hand‑built simulator or a learned predictor used for planning, imagined experience generation, or as a critic for policy updates.
Approaches that store artifacts beyond network weights: episodic memories, skill libraries, or generated programs. These artifacts are later retrieved to guide action selection, data collection, or to modify the learning procedure itself.
Methods that search over or learn components of the learning procedure itself—optimizers, loss functions, architecture choices, or hyper‑parameter schedules. The search is typically outer‑loop optimization that produces a better inner‑loop learner.
Techniques where a system trains on its own outputs (self‑training) or maintains performance across a sequence of tasks (continual learning). The learner’s own predictions or generated data become the training signal for the next iteration.
The review tracks every system along a loop: an experience produces a retained change, and that change is later used to influence actions, predictions, or learning.
Is the Physical RSI Path just a standard training pipeline with an extra storage step?
No. The key distinction is that the stored artifact must be reused to affect a later learner or improver, and the paper explicitly tests this reuse with matched‑learner controls (R1, R2). A plain pipeline that discards the artifact does not satisfy the recursion criteria.
**Figure 2. Improving what the robot does and how it learns.** On the left, the learner improves under a fixed recipe and retains the change in its policy, world model, or external knowledge. On the right, the improver revises the learning procedure using retained repair knowledge; the dashed return asks whether that knowledge improves how the next revision is made (R1), and whether the improver, once improved, then helps produce its own next improvement (R2). The bottom row lists the physical limits behind the four conditions that let gains carry forward.
Mechanisms of Improvement
Survey of physical self‑improvement mechanisms across policy, world‑model, and model‑assisted approaches.
Policy‑update signals dominate physical self‑improvement, yet later‑learning benefits are rare; world‑model loops add planning depth, while model‑assisted pipelines bridge simulation and reality.
This camp groups works that modify a robot’s policy by attaching a learning signal to already‑collected trajectories. Four recurring signals appear: (1) Relabeling reinterprets past attempts with new goals; (2) Reward scores outcomes to guide optimization; (3) Correction injects missing actions from human intervention; (4) Replay reuses earlier experience to preserve behavior. Studies range from hindsight experience replay (Andrychowicz et al., 2017) to SI‑EFM (Ghasemipour et al., 2025) and HIL‑SERL (Luo et al., 2025). Reported gains vary: relabeling often yields modest BLEU‑style improvements, while correction‑heavy methods achieve up‑to 99 % success on real manipulation tasks.
**Figure 5. Four teaching signals from physical experience.** Each column follows the grasp example: what was experienced, what the signal changes in the training data, and representative systems from this section.
This camp comprises methods that learn a predictive model of the physical environment and then use it for planning, policy training, or feature provision. Generative video models (UniPi, UniSim, Genie) serve as both planners and data generators. Classical model‑based approaches (Dyna, MBPO, PETS) interleave real and imagined experience, while recent latent dynamics (DreamerV3, DayDreamer) train compact policies inside learned dynamics. Success hinges on the interface: a predicted video must be converted into robot commands (e.g., SILVR’s inverse‑dynamics translator) before execution.
**Figure 6.** Where models, policies, and physical data change. Learning loops of SILVR, RISE, PlayWorld, and VLAW (Luo et al., 2026a; Yang et al., 2026b; Yin et al., 2026b; Guo et al., 2026).
This camp focuses on systems that combine a learned model with a policy‑learning loop, explicitly distinguishing which components are updated, which signals drive learning, and how outcomes are checked. Examples include SILVR (updates planner, freezes inverse dynamics), RISE (fixed dynamics/value, policy advantage), PlayWorld (fixed base policy, noise policy learns inside model), VLAW (updates model and policy iteratively), and DexTouch‑WM (model‑generated observations for policy training).
**Table 7.** Model-assisted learning. Columns follow Table 6.
The Learner is the component that directly maps observations to actions, typically a policy or a world‑model‑conditioned controller.
How does the Learner differ from a generic policy network?
Both map observations to actions, but the Learner in this survey is explicitly tied to a learning signal (e.g., relabeling or correction) and may be frozen, adapted, or combined with a world model. A generic policy network lacks that explicit coupling to the signal that drives its updates.
The Improver modifies the learning procedure itself, changing how the Learner is updated or how signals are generated.
Why not treat the Improver as just another meta‑learner?
Because the Improver in this survey explicitly targets the *process* of learning (e.g., how data are collected or how losses are constructed), whereas a generic meta‑learner typically only adapts parameters of a fixed learning algorithm. The Improver can change the algorithm itself, enabling true recursion.
Externalized Knowledge and Skills
This section maps external knowledge into memory, skill, and runtime categories and highlights their comparative impact.
External knowledge in robotics clusters into three dominant artifact types—memory, skill, and runtime—each offering distinct pathways for improvement, yet the field lacks unified mechanisms for validating and transferring these artifacts across embodiments.
Systems that store experience as persistent records, ranging from raw scene snapshots to distilled principles, and retrieve them later to inform decision‑making. Memory entries are often versioned, verified, and abstracted to different degrees, affecting how reliably a later planner can exploit them.
Callable routines extracted from demonstrations, simulations, or code‑generation agents and packaged into libraries. Skills are defined by an implementation, a callable interface, and explicit pre‑conditions/effects, allowing planners to compose them into new behaviors.
Logic that selects context, monitors progress, interrupts execution, and routes control to recovery modules while keeping the underlying planner and controller weights fixed. Runtime revisions can reshape the effective agent without altering its learned policy.
**Figure 7. Three forms of external knowledge in the grasp example.** Each panel shows one saved form of the slip repair and representative systems from this section; reuse on another robot first checks the interface.
Improving Learning Procedures
Survey of how altering learning procedures fuels recursive self‑improvement in robotics.
Improving the learning procedure is the core lever for recursive self‑improvement, organized into four decision dimensions—what to practice, how outcomes are judged, which data feed the next update, and how the update itself is generated.
This camp groups methods that decide which experiences the robot should collect next. Approaches range from classic active‑learning (Cohn 1996) to task‑generation via large language models (GenSim, RoboGen, OMNI, Eurekaverse). Concrete systems such as SAC‑X schedule auxiliary behaviors, automatic domain randomization widens simulated conditions, and Enhanced POET co‑evolves environments with policies. Recent VLM‑driven selectors (FIND, InSight, Agentic Skill Discovery, EmbodiedRSI) report success lifts of 12–20 percentage points on real‑world tasks, demonstrating that a signal tied to the learner’s current capability can dramatically reshape the data distribution.
This camp comprises techniques that update the reward or evaluator after the policy changes. Preference‑based loops (PEBBLE) relabel replay buffers whenever the learned reward model changes; vision‑language models (RL‑VLM‑F) provide automatic labeling. Hardware‑centric refinements (FIERCE, RLER) periodically refresh evaluators and report 8–12 percentage‑point gains on real peg‑in‑hole and cup‑stacking tasks. EvoHIL and VeriFine demonstrate that human‑in‑the‑loop or classifier‑based updates can raise success from 64 % to 86 % and improve judge correlation from 0.55 to 0.85, respectively, under matched training budgets.
This camp focuses on how the retained dataset for the next learner is curated or generated. Methods such as RoboCat and PLD transfer specialist data to a generalist or distill recoveries back into the generalist. World‑model‑augmented pipelines (VLAW, ASPIRE, CIDER) reuse or refresh models to synthesize additional rollouts while keeping physical rollouts fixed. Empirical reports show that adding model‑generated data can raise success from 13.3 % to 50 % (World‑VLA‑Loop) and from 82 % to 56 % (WorldSample) on real tasks, illustrating the impact of data‑centric loops.
This camp studies learned updaters that modify the learning rule itself. AdaRSI decides which sub‑steps to fine‑tune, achieving 13.8 % unseen composite‑task success versus 12.6 % for naïve fine‑tuning while using only 28 % of GPU time. PRACTICE learns an editor that adds, merges, or removes skills in an external library, improving simulated household task success from 42.7 % to 45.3 % when training on failure examples. ENPIRE builds a full research loop where coding agents generate, test, and retain hypotheses, reporting procedural knowledge transfer across tasks. HARBOR and AURA similarly stage curriculum generation, reward design, and policy tuning through gated sub‑agents, achieving 0.885 success on a stacking benchmark where prior methods reach 0.0 under the same budget.
Improving the Improver
Survey of improver mechanisms and their empirical evidence across domains.
Most surveyed systems improve only parts of the learning pipeline; full recursive self‑improvement remains unproven, especially for physical agents.
Systems that write structured experience documents after each trial and later retrieve them to guide subsequent revisions. AdaRSI records sub‑step benefits and training budgets; HARBOR distills recurring patterns into an experience memory; AURA stores curricula, metrics, and human feedback in a database; EmbodiedRSI learns a value function for which memories to keep; ENPIRE writes a summary of one search to seed a related task. All operate on the same task or closely related tasks, and their ablations show substantial drops when the memory component is removed (e.g., ENPIRE’s success on a new task falls from 99 % to 38 %).
Approaches that reuse earlier guidance or templates but keep the improver’s update rule fixed. ASPIRE retrieves repair guidance from prior program edits; LMPC trains a reward‑code model on past teaching sessions, cutting chat turns from 2.4 to 1.9 on unseen tasks; HarnessPAI’s library of failure‑to‑repair notes lets a program writer reach full success on three LIBERO‑Object tasks within two rounds. These methods improve the construction of later controllers but do not revise the improver itself.
Systems that let the improver edit its own tools, diagnostics, or acceptance rules. PhysEvo lets a meta‑agent revise both task agents and its own diagnostic tools, achieving 62 % average success on held‑out RoboDojo layouts versus 47 % for the strongest published baseline; RegenHarness’s champion–challenger rule prevents degradation when revising a robot’s harness; MEMENTO evolves a rollout evaluator before policy search; VeriFine revises its judge’s rubric with human calibration; STOP applies a language‑model‑driven improver to itself, showing gains for GPT‑4 but not for weaker models; SICA merges meta‑agent and target agent so that a single agent edits its own codebase, raising SWE‑bench accuracy from 17 % to 53 % in fifteen iterations; VERSE revises prompts, skills, and tools, achieving 41.05 % versus 39.20 % when the reviser is frozen; Predict‑Before‑You‑Modify adds an editable predictor that, when frozen, yields bootstrap intervals that include zero.
Algorithms that treat the improver as an object of search. Expert Iteration and STaR repeatedly apply a fixed loop of search‑then‑train; Promptbreeder co‑evolves task prompts and mutation prompts; ADAS searches over agent designs written as code; ALMA searches over memory designs; AlphaEvolve couples code proposals with executable evaluators; the Darwin Gödel Machine (DGM) archives self‑modified code and shows sustained improvement only when the improver itself can modify; Gödel Agent rewrites its own improvement code; SelfSearch modifies itself using records of prior self‑modifications; HyperAgents let the meta‑agent edit itself, achieving higher test scores in simulated quadruped reward design (0.372 vs 0.213) and in math grading (0.630 vs 0.000); AIDE2’s evolved outer‑loop agent produces comparable grades to a human‑designed driver (0.780 vs 0.782). These systems provide the closest analogues to the second recursion link, but most lack frozen‑improver baselines that would isolate the contribution of self‑modification.
Physical or simulated robot platforms that claim recursive self‑improvement. The table in the survey lists D‑World (digital twin revision), MobiAgent (skill learning from verified rollouts, 7.5 %→27.5 % on RoboCasa), Embodied Turing Machines (stateful decision code, 70.24 % on 42 tasks), Astribot S1 (model accuracy 32.5 %→57.5 %), RoboRSI (branch‑local patches, +2.7 to 11.0 points over a fixed agent), Criticality model (policy‑level RSI, collection rule only), EmbodiRSI (demos and corrections, 50.4 %→83.5 % on banana placement), ALOHA (fixed‑rule procedure, 83.1 % vs 75.0 %), EmbodiedRSI (memory selection, 71.3 % vs 40.1 % on unseen tasks), Local Recovery Supervision (88 vs 78 successes), RE‑0 (admission rule, 4/50→31/50), RIWANav (offline navigation +7.94 points), AdaRSI (experience document, 4.6 %→13.8 %), EndoRSI (human‑governed coding, 2/21→17/21), PhysEvo (tool revision, 62 % vs 47 %), RegenHarness (harness configuration, qualitative gains). None of these provide a frozen‑knowledge comparison that would satisfy the R2 recursion link.
Evaluating Recursive Improvement
Physical RSI shows incremental gains but lacks controlled recursive evidence, highlighting a need for isolated component tests and standardized evaluation.
Physical RSI studies report modest improvements, yet none isolate a recursive gain; the community agrees on the necessity of component‑level controls and a shared reporting standard, while the conspicuous gap is the absence of physical experiments that demonstrate true recursion.
**Table 6.** Policy updates and selected within-study comparisons. Outcome checks give the setting, result, and scorer where identified; unresolved scorers are listed in Section 4.1. ↻ updated; ⬆ held fixed.
**Figure 8. Does a revised component help the next learner?** Two matched learners differ only in the component under test, here an evaluator. Their common outcome check stays fixed and separate from that evaluator; comparing the learners’ outcomes tests procedure improvement.
**Table 12.** The open problems of this section at a glance, in the order of the subsections that discuss them. ⦿ marks the main focus of each direction among the four conditions: informative feedback (IF), usability (U), robustness to distribution shift (DS), and continued practice (CP). Starting benchmarks are proposals from the corresponding discussions.
**Table 14.** Comparisons that bear most directly on each condition.
**Figure 9. A proposed test of the two links of recursive self-improvement (R1 and R2).** In R2, the updated improver revises itself with frozen or acquired knowledge, producing the control or next improver, respectively. Each then revises a learning procedure, which is tested through fresh matched learners under a common, protected outcome check; the next improver must beat both the control and the updated improver; for the updated improver, its learner from R1 serves as the baseline. The R2 control assumes that the acquired knowledge can be withheld without changing the parent improver, as with a memory snapshot; the count of four hardware-trained learners per seed assumes that one candidate per improver survives screening.
R1 measures whether a changed improver yields a better learner; R2 measures whether that learner, when used as an improver, produces an even better next improver.
How does the R2 link differ from simply re‑evaluating the same learner with a new evaluator?
R2 requires that the learner itself becomes the next improver; the test freezes the improver’s knowledge while the learner is updated, then checks whether that learner can produce a better improver. Simply swapping evaluators does not close the recursion loop because the improver’s internal state never changes.
This camp studies how data collection, evaluator updates, and feedback mechanisms affect the amount of useful learning signal. The surveyed works show that autonomous data collection quickly saturates, whereas targeted demonstrations or refreshed evaluators continue to provide informative gradients.