Dream-RSI: Recursive Self-Improvement through Evolving Worlds
Tong Zheng, Xidong Wu, Zheng Zhang, Zhankui He, Chaoyi Zhang, Benjamin Coleman, Ruoqiao Wei, Di Bai, Haolin Liu, Rui Liu, Xue Wang, Yue Zhuan, Wang-Cheng Kang, Renkai Xiang, Heng Huang, Xinwu Cheng, Yunsong Guo
Dream-RSI turns historical discovery logs into replay simulators to recursively optimize exploration policies.
How can AI agents improve their own exploration strategies for scientific discovery by treating historical execution traces as a replay simulator?
Recursive self-improvement (RSI) for AI agents is bottlenecked by the high cost of optimizing exploration strategies, which typically requires expensive, long-horizon online rollouts to evaluate new policies. Dream-RSI addresses this by treating completed discovery histories as structured replay simulators, allowing alternative exploration policies to be evaluated offline via "dreaming" over recorded branches without rerunning the underlying agent. This framework achieves competitive or superior discovery quality across algorithm, math, and kernel engineering tasks while significantly reducing the total number of discovery-agent calls.
Paper Primer
Dream-RSI implements a closed-loop meta-exploration system that alternates between online discovery and offline policy refinement. The core move is to organize past discovery trees into a replayable simulator where a policy-development agent can test new exploration strategies against recorded outcomes, selecting the best-performing policy for the next online deployment.
Dream-RSI significantly improves discovery efficiency compared to fixed-exploration baselines.
In algorithm engineering (Lasso path solver), Dream-RSI reduced discovery-agent calls by 1.7× compared to fixed-exploration baselines while outperforming standard solvers. Up to 162× fewer agent calls than the SimpleTES baseline in specific settings.
The framework generalizes across diverse scientific domains, including GPU kernel optimization.
On KernelBench tasks, Dream-RSI reached target execution speeds using 1.79×–2.43× fewer generations or improved performance by up to 2.09× under identical budgets.
Why is it difficult to optimize exploration policies online without this framework?
Evaluating an exploration policy requires observing its impact over long-horizon rollouts, making feedback delayed and expensive. Additionally, the meta-policy space is vast, requiring many trials that are prohibitively costly to run online.
Does Dream-RSI change the underlying discovery agent or the task evaluator?
No. The framework uses a lightweight orchestration layer to control branching and parallel exploration, leaving the underlying coding agent, evaluator, and execution interfaces fixed.
By conceptualizing discovery history as a replay simulator, Dream-RSI transforms meta-exploration from an expensive online trial-and-error process into a fast, simulation-based optimization loop, enabling scalable recursive self-improvement.
Introduction to Recursive Discovery
We expose why static exploration stalls RSI and how Dream‑RSI turns discovery history into a cheap simulator.
Current RSI pipelines rely on fixed exploration policies that cannot leverage accumulated discovery experience, while online meta‑policy optimization demands expensive, delayed feedback over long rollouts.
RSI is the process by which an autonomous agent repeatedly refines its own discovery mechanisms, using the outcomes of prior iterations to guide future ones.
Dream‑RSI treats the recorded discovery tree as a replay simulator, allowing many candidate exploration policies to be evaluated cheaply by “dreaming” over the stored branches instead of re‑executing the costly underlying agent.
The essential shift is from a static, one‑shot exploration policy to a recursively evolving meta‑policy that continuously learns from its own history.
The Dream-RSI Mechanism
Dream‑RSI iteratively refines its exploration policy by replaying past discovery trees.
Online exploration repeatedly revisits the same high‑value regions, wasting compute on already‑known outcomes. Dream‑RSI solves this by turning the accumulated discovery tree into a replayable world, letting the agent “dream” many alternative policies without extra execution.
The loop alternates between an online rollout that expands a discovery tree and an offline “dreaming” phase that evaluates many policy variants inside the tree‑based replay simulator.
Round 1: batch $\{r\}$ → child $v_1$ added; tree now $\{r,v_1\}$.
Round 2: batch $\{v_1,r\}$ → children $v_2$ and $v_3$ added; tree now $\{r,v_1,v_2,v_3\}$.
Replay phase: the fixed tree $\{r,v_1,v_2,v_3\}$ is used to evaluate two policy variants without any new generation calls.
The replay phase can explore many “what‑if” branches on the same tree, turning a single costly rollout into a cheap policy‑search laboratory.
How does Dream‑RSI differ from simply logging past runs and re‑using the logs?
Logging only provides raw data; Dream‑RSI structures that data into a searchable discovery tree and defines a decision interface ($A(T;W)$) that lets a policy query the tree as if it were a live environment, enabling systematic batch exploration and policy‑code revision.
Think of the discovery tree as a video‑game replay: the recorded states let the agent replay any past branch instantly, while still being able to choose new branches to explore.
Why not just run the policy on the full tree instead of batching with $A(T;W)$?
Batching respects the real‑world parallelism constraint: each worker can issue only one generation‑evaluation request at a time. Without $A(T;W)$ the policy would assume unlimited parallelism, over‑optimistic and unimplementable.
Initialize $T_0^{t}=\{r\}$ and set round counter $k=0$.
While $k < K_1$ and the selected batch $C_k^{t}$ is non‑empty:
Select $C_k^{t}\in A(T_k^{t};W)$ using policy $\pi_t$.
Each worker expands its assigned node, generating a new child and receiving score $s_v$.
Attach all new children to $T_k^{t}$, forming $T_{k+1}^{t}$.
Increment $k\gets k+1$.
Terminate when $k=K_1$ or $C_k^{t}$ is empty; record the final tree $T_t$ and append to history $H_t$.
Fix the history $H_t$ (all trees $T_1,\dots,T_t$).
Initialize policy version $\pi_0^{t}=\pi_t$.
For $m=0$ to $M-1$:
For each replay world $T_i$:
Reset policy state, set $T_{m,0}= \{r\}$.
Iteratively select batches $C_{m,k}\in A(T_{m,k};W)$ until $k=K_2$ or no new nodes remain, observing recorded children.
Accumulate replay score $\text{score}_{m,i}$ using the objective.
Compute average replay score $V_m$ over all $t$ worlds.
Use the development agent to revise the policy code, producing $\pi_{m+1}^{t}$.
Select $\pi_{t+1}$ as the version with maximal $V_m$.
**Figure 1 | Overview of DREAM-RSI.** The system operates in a recursive self-improvement loop via three core stages: ① **Online Explore**, where the current exploration policy guides a coding agent to expand a discovery tree and log historical traces; ② **Construct Replay Simulator**, where the generated discovery tree is converted into a reusable simulator pool; and ③ **Dreaming-based Policy Improvement**, where the agent "dreams" up a massive pool of alternative policies in its mind. It then feeds these candidate policies into the replay simulator to simulate executions and derive rapid feedback, continuously refining its strategy (detailed in the Zoom-in box). The updated policy then redeploys for the next round of online exploration.
**Figure 2.** Discovery history as a replay simulator. A deployed policy first explores online to generate a structured discovery tree containing historical execution traces (each node denote an attempt with its full observation). Thousands of candidate policies can then be tested within this simulator—evaluating alternative choices of search branches, exploration orders, concurrency levels, and stopping rules. Since all node outcomes are pre-stored, a single costly online run enables thousands of rapid, zero-execution-cost off-policy evaluations. This enables policy improvement through historical replay: the agent can “dream” over many alternative exploration strategies before redeploying the improved policy online.
Empirical Evaluation
Dream‑RSI cuts average runtime and improves solution quality across algorithm, math, and kernel tasks.
Dream‑RSI reduces average downstream runtime compared to Recursive Fixed Exploration across three domains.
Gemini‑3.7‑Flash average runtime 2350.6 ms vs 2516.7 ms (1879 vs 3200 discovery‑agent calls); Gemini‑3.1 Pro 2931.0 ms vs 3587.1 ms.
It keeps the same exploration policy unchanged across all recursive discovery rounds.
How does Recursive Fixed Exploration differ from Dream‑RSI?
Recursive Fixed Exploration never updates its policy after the first round, while Dream‑RSI refines the policy via a replay simulator and redeploys the improved policy in each subsequent round.
**Figure 3.** Lasso regularization-path discovery results. (a) Final wall-clock runtime on six held-out downstream tasks; lower is better. Compute denotes the cumulative number of discovery-agent calls. (b) Recursive discovery dynamics. Average downstream runtime across six held-out tasks versus cumulative discovery compute for Gemini-3.1-Pro and Gemini-3.7-Flash. Numbers next to markers denote recursive rounds (iterations). Lower is better.
**Figure 4 | GPU kernel engineering results.** Discovery performance of DREAM-RSI and Recursive Fixed Exploration as a function of the number of generations. On VGG16 and LayerNorm, DREAM-RSI reaches comparable performance with 2.43× and 1.79× fewer generations, respectively. On ConvDiv and ConvMax, it achieves 2.09× and 1.44× higher performance under comparable discovery budgets. Higher is better for all tasks.
**Discovered Solver Analysis.**
Analysis of Discovery Dynamics
We revisit how replaying past discoveries guides recursive search and examine component ablations.
We first probe how the historical inductive bias shapes long‑horizon discovery and then track the policy’s adaptive behavior across recursive rounds.
Past discovery traces are replayed to bias future search toward regions that have previously yielded high‑value discoveries.
How does this bias differ from simply providing a fixed guidance direction?
Fixed guidance injects a static semantic cue into the prompt, whereas the historical inductive bias draws from a replay of actual outcomes, allowing the policy to re‑weight regions based on observed success rather than a predetermined direction.
Removing explicit semantic guidance improves discovery performance.
Figure 5 shows Dream‑RSI reaches ≈1.9 while Dream‑RSI + Guidance caps at ≈1.5 under the same discovery budget.
**Figure 5.** Discovery performance on ConvDiv. Using history as an interactive replay simulator outperforms using it only as guidance.
The exploration policy conserves compute early, then ramps up effort when progress stalls.
Figure 6(a) shows round‑best performance rising from 0.427 to 1.898, while Figure 6(b) shows evaluated attempts dropping from ~110 to ~50 before increasing again.
**Figure 6.** Evolution of exploration behavior on ConvDiv. (a) Round-best performance across recursive execution rounds. (b) The number of evaluated attempts in each round.
Related Work
We situate Dream‑RSI among prior discovery and self‑evolving systems.
AlphaEvolve treats discovery as a guided evolutionary loop where candidates are generated, evaluated, and iteratively refined using feedback from prior attempts.
Iteratively generates code snippets, evaluates them on benchmark tasks, and steers the next generation cycle using the results.
Uses large language models to propose program candidates, then refines them through gradient‑based updates guided by execution outcomes.
Combines evolutionary search with neural architecture search to evolve both model structure and parameters for scientific tasks.
Prioritizes candidate evaluation by estimating the potential payoff of each branch, pruning low‑value paths early.
Tracks semantic deltas between successive discoveries and uses them to guide future search directions.
Leverages machine‑learning‑based predictors to evaluate candidate solutions without full execution, accelerating the evolutionary loop.
Provides an adaptive infrastructure that dynamically allocates compute to promising exploration branches.
Orchestrates multiple concurrent search branches, sharing information to improve collective discovery efficiency.
Optimizes the search strategy itself as a learnable policy, rather than focusing solely on candidate solutions.
These works collectively illustrate a shift from static candidate generation toward meta‑level exploration optimization, a trend that Dream‑RSI extends by turning discovery histories into replay simulators.
Task Definitions
Precise definitions of the benchmark tasks used to evaluate the system.
Problem 1 asks for the full Lasso regularization path on a dataset with feature matrix $X\in\mathbb{R}^{n\times p}$ and response $y\in\mathbb{R}^{n}$, minimizing $F_{k}(w)=\tfrac{1}{2n}\|y-Xw\|_{2}^{2}+\lambda_{k}\|w\|_{1}$ for a decreasing sequence $\lambda_{1}>\dots>\lambda_{K}$.
The benchmark measures runtime by summing the compute time $t_{i}$ required to obtain the optimal coefficients $w_{k}^{\star}$ for each $\lambda_{k}$ on a set of timing instances $I$, and reports the score $R_{\text{search}}=-\frac{1}{|I|}\sum_{i\in I}t_{i}$.
Problem 2 seeks a finite integer set $A\subset\mathbb{Z}$ that maximizes the ratio $\Gamma(A)=\frac{\log\bigl(|A+A|/|A|\bigr)}{\log\bigl(|A-A|/|A|\bigr)}$, where $A+A$ and $A-A$ denote the sumset and difference set respectively.
Problem 3 asks for the placement of $n\in\{26,32\}$ circles inside the unit square, choosing centers $(x_{i},y_{i})\in[0,1]^{2}$ and radii $r_{i}\ge0$ that satisfy $r_{i}\le x_{i}\le1-r_{i}$, $r_{i}\le y_{i}\le1-r_{i}$, and non‑overlap $(x_{i}-x_{j})^{2}+(y_{i}-y_{j})^{2}\ge(r_{i}+r_{j})^{2}$ for all $i<j$, while maximizing $\sum_{i=1}^{n}r_{i}$.
Problem 4 defines the autoconvolution $(f*f)(t)=\int_{\mathbb{R}}f(t-x)f(x)\,dx$ for an integrable function $f:\mathbb{R}\to\mathbb{R}$ and poses three related optimization tasks.
The first autocorrelation inequality constrains $f$ to be non‑negative, supported on $[-\tfrac14,\tfrac14]$, and normalized $\int_{-1/4}^{1/4}f(x)\,dx=1$, then minimizes $\max_{t\in[-1/2,1/2]}(f*f)(t)$.
The second inequality keeps the same constraints on $f$ but maximizes $\Phi_{2}(f)=\frac{\|f*f\|_{2}^{2}}{\|f*f\|_{1}\,\|f*f\|_{\infty}}$, where $\Phi_{1}(f)=\max_{t}(f*f)(t)$.
The third inequality relaxes the sign constraint on $f$, still supported on $[-\tfrac14,\tfrac14]$ with the same normalization, and minimizes $\Phi_{3}(f)=\max_{t\in[-1/2,1/2]}|(f*f)(t)|$.
System Prompts
Provides the exact prompts used for online exploration and replay‑based policy improvement.
This appendix lists the literal prompts that drive both the online discovery phase and the replay‑based policy‑improvement phase of Dream‑RSI.
Listing 1 | Prompt used for online exploration (Section B.1).
Listing 2 | Prompt used for replay‑based improvement of the exploration policy (Section B.2).
Discovered Solver Implementation
Provides the full C++ implementation of the Lasso‑path solver discovered by Dream‑RSI.
This listing presents the full C++ implementation of the Lasso‑path solver that Dream‑RSI discovered. It combines strong‑rule screening with adaptive Cauchy‑Schwarz KKT (Karush‑Kuhn‑Tucker) pruning, disjoint active‑set bookkeeping, lazy Gram‑matrix construction, and several hardware‑aware optimizations.
Complete Lasso‑path solver discovered by Dream‑RSI.
Solver Logic Details
Implements the second phase of the solver, handling active‑set updates and KKT checks.
The solver proceeds in a perpetual outer loop that alternates between expanding the active set, performing coordinate‑descent updates, and verifying Karush‑Kuhn‑Tucker (KKT) conditions.
This block implements the full solver iteration: it expands the active set with KKT‑violating screened features, runs a SIMD‑accelerated coordinate‑descent sweep, and then performs a two‑stage KKT verification that may trigger another CD pass or a full reset.
Solver Parallelization
Implementation details for parallel gradient updates and KKT screening using SIMD and OpenMP.
This appendix spells out the low‑level solver operations that drive the Dream‑RSI loop, focusing on SIMD‑blocked matrix‑vector products and OpenMP parallelism.
Parallel reset of gradients for unscreened indices (`run_parallel_reset`)
Copy residual vector $r$ to a padded reference
Parallel copy of gradient reference array
Parallel exact‑gradient computation for the tiny unpruned subset (`run_parallel_comp`)
Serial KKT screening loop (moves violators to screened set)
Standard KKT check without pruning (`run_parallel_uns_std`)
Convergence break when all KKT conditions hold
Adding unscreened violators to the active set and cleaning up
Solver Entry Point
The main reads data, pads the matrix for alignment, computes column stats in parallel, then solves the active‑set.