Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks
Haoyu Zhao, Zhengxu Yu, Zhiyuan He, Meng Fang, Rasul Tutunov, Haitham Bou-Ammar, Weilin Luo, Jun Wang
Memento 3 uses a persistent natural-language rulebook and executable code to learn environment dynamics through recursive self-improvement.
How can an agent recursively improve its world model by maintaining a natural-language "rulebook" that it continuously revises based on environmental feedback?
Agents in unfamiliar environments often struggle to generalize because limited interaction data supports multiple, conflicting hypotheses about how the world works. Memento 3 addresses this by maintaining a "Code as Model" framework: a frozen LLM continuously updates a natural-language rulebook and its corresponding executable code to represent the agent's evolving understanding of environment dynamics. On the ARC-AGI-3 benchmark, this approach clears every level of all 25 public games with a mean Relative Human Action Efficiency of 100.0, using less than half the human action count.
Paper Primer
The core mechanism is a reflective loop where the agent treats its world model as an object of revision. The LLM proposes rulebook updates and compiles them into executable Python code, which is then verified against the entire interaction history via exact replay; only models that both match the rulebook and reproduce every observed transition are accepted for planning.
Memento 3 achieves a perfect success rate on the ARC-AGI-3 benchmark.
The agent cleared all 25 public games, reaching the maximum possible Relative Human Action Efficiency (RHAE) of 100.0. 100.0 mean RHAE, outperforming the next best baseline (99.0) while using 44% of the human action budget.
To mitigate the risk of committing to a single, potentially incorrect hypothesis, the system supports a population extension. This maintains multiple rulebook-executable pairs in parallel, sharing the same interaction history but allowing different models to guide exploration until they are falsified by new evidence.
Why use both a natural-language rulebook and executable code instead of just one?
The rulebook provides a semantic, inspectable space for the agent to express hypotheses about environment rules, while the executable code provides a rigorous, testable implementation for prediction and planning. Together, they allow the agent to distinguish between a misunderstanding of the rules (rulebook change) and a failure to implement those rules correctly (code repair).
What happens when the agent's model makes a prediction error?
The agent uses the discrepancy as a concrete counterexample to trigger the reflective loop. It revisits the interaction history to diagnose whether the error stems from an incorrect rule or a flawed implementation, then proposes a revision that must satisfy both semantic fidelity to the rulebook and exact replay consistency with all past observations.
By externalizing the world model into version-controlled code and natural language, Memento 3 enables agents to perform recursive self-improvement without updating the underlying LLM parameters, effectively turning the agent's memory into a self-correcting simulator.
Introduction
We frame the core problem: limited observations admit many world models, motivating Memento 3.
Learning to act in unfamiliar environments forces agents to infer how the world works from sparse observations. Such limited data often admit many plausible world models that agree on past interactions but diverge on unseen states, creating a fundamental ambiguity for planning.
The agent keeps a natural‑language rulebook that it continuously rewrites from experience and turns into executable code for planning.
The challenge of reconciling limited observations with multiple world models.
The Memento 3 Framework
The section shows how agents keep a natural‑language rulebook and compiled executable, revising them for recursive self‑improvement.
Agents must learn an unknown environment while respecting a strict action budget. Naïvely rebuilding a model from scratch after each observation wastes the limited budget and stalls progress. Recursive self‑improvement solves this by keeping a persistent hypothesis that can be updated incrementally.
A rulebook is a natural‑language collection of hypotheses describing the environment’s entities, actions, dynamics, and goal conditions, stored as persistent semantic memory.
Step 1: The LLM parses the two sentences and generates a Python function move(state) that decrements state.budget.
Step 2: It generates a predicate `is_goal`(state) that returns True iff state.`badge_colour` == “red”.
Step 3: The compiled program is stored as $\theta$₀ and linked to the rulebook $\psi$₀.
Step 4: Replay of the initial observation o₀ initialises $\theta$₀’s internal state s₀^$\theta$ with state.budget = 3 and state.`badge_colour` = “blue”.
Step 5: After the agent moves twice, the replay updates s₂^$\theta$ to budget = 1; the goal predicate still returns False.
The rulebook’s textual form lets the LLM reason about abstract constraints (budget, colour) while the executable provides concrete state updates for planning.
How does a rulebook differ from a simple checklist of observed transitions?
A checklist records only what has been seen; a rulebook abstracts those observations into generalizable hypotheses (e.g., “any move consumes one unit”) that can predict unseen situations and be compiled into executable code.
An executable world model is a program that implements the rulebook’s hypotheses, exposing a transition function $f_θ$, an observation function $ρ_θ$, and a goal set $G_θ$ for planning.
Step 1: $ι_θ(o₀)$ creates state = {budget: 3, `badge_colour`: “blue”}.
Step 2: $f_θ(state, move)$ decrements budget → 2.
Step 3: $f_θ(state, move)$ decrements budget → 1.
Step 4: $ρ_θ(state)$ returns the observation {o₁ = (budget=1, colour=“blue”)}.
Step 5: $is_goal(state)$ evaluates False because the colour is still “blue”.
The executable turns abstract textual rules into concrete state transitions, enabling the planner to simulate future actions without further interaction.
Why compile the rulebook into code instead of planning directly from the textual rules?
Code provides a deterministic, fast simulator: the planner can query $f_θ$ and $ρ_θ$ repeatedly to evaluate many action sequences, something infeasible with on‑the‑fly natural‑language inference.
Use the current executable $θ_t$ and reconstructed state $s_t^θ(H_t)$ to plan an action sequence with the control operator $P$.
Execute the first action $a_t$, observe the new environment observation $o_{t+1}$ and the completion signal $c_{t+1}$.
Append $(a_t, o_{t+1})$ to the interaction record, forming $H_{t+1}$.
Prompt the frozen LLM with $H_{t+1}$ and the current rulebook $ψ_t$ to propose a revised rulebook $ψ_{t+1}$.
Compile $ψ_{t+1}$ into a new executable $θ_{t+1}$ and verify that replay of $H_{t+1}$ under $θ_{t+1}$ reproduces all recorded observations.
Replay $H_{t+1}$ under $θ_{t+1}$ to obtain the updated internal state $s_{t+1}^θ(H_{t+1})$.
Form the point‑mass belief approximation $\hat b_{t+1}=δ(h_{t+1}, s_{t+1}^θ(H_{t+1}))$ and repeat.
**Figure 1.** From a rulebook to an executable world model. Numbered rules map to corresponding branches of a simplified transition function. A planner uses the model to predict a route that changes the badge's pattern and colour, collects a refill, and reaches the matching door within the move budget.
**Figure 3.** A graphical model of recursive self-improvement in Code as Model. The underlying environment state $s_t$ produces observation $o_t$, which enters the external learning state $Z_t = (H_t, \psi_t, \theta_t)$. The agent selects $a_t$ from $Z_t$, and the environment advances to $s_{t+1}$. The next observation $o_{t+1}$, together with $Z_t$ and $a_t$, determines the information available for updating $Z_{t+1}$. Rectangles contain the history, rulebook, and executable; white circles denote hidden states and grey circles observations and actions. $P$ selects actions, and $U$ updates the rulebook-executable pair. Ellipses indicate continued interaction. Completion feedback also enters the interaction history but is omitted from the diagram for clarity.
By maintaining a mutable rulebook and its compiled executable, the agent continuously improves its predictive model without discarding past experience, enabling efficient planning under tight action budgets.
Reflective Rule Revision
This section details how the agent revises its rulebook and executable via reflective belief updates.
After taking action $a_t$, the agent receives feedback $y_{t+1}$ that must be folded into its internal belief over rules and states. Naïvely treating each piece of feedback in isolation quickly leads to inconsistent hypotheses.
The agent treats its rulebook–executable pair as a hypothesis and repeatedly reflects on new observations, revises the rulebook, recompiles a simulator, and verifies against the full interaction history.
How does this belief‑update differ from a standard Bayesian update over a static model space?
Standard Bayesian updates keep the model fixed and only adjust state probabilities. Here the update simultaneously propagates candidate dynamics $T_m$, revises the rulebook $\psi$, and recompiles a new executable $\theta$, so the hypothesis space itself evolves after each feedback.
Observation: execute $a_t$, receive $o_{t+1}$, and append $(a_t,o_{t+1})$ to the history $H_{t+1}$ (Eq. 15).
Reflection: the frozen LLM $\pi_\phi$ samples a revision decision $d_{t+1}$ from $\{\,\text{retain},\text{revise\_rulebook},\text{repair\_code}\,\}$ conditioned on $H_{t+1}$, $\psi_t$, and $\theta_t$ (Eq. 16).
Rule revision: if $d_{t+1}=\text{revise\_rulebook}$, apply $\Delta_\phi$ to obtain a new rulebook $\tilde{\psi}_{t+1}$; otherwise keep $\psi_t$ (Eq. 17).
Compilation: invoke the compiler $C_\phi$ to produce an executable $\tilde{\theta}_{t+1}$ consistent with $\tilde{\psi}_{t+1}$ (Eq. 18).
Verification: replay the entire history through $\tilde{\theta}_{t+1}$; accept the pair only if every predicted observation matches the recorded one, yielding the admissible set $V_{t+1}$ (Eqs. 20–21).
Action $a_0$ is taken; feedback $y_1=(o_1,c_1)$ arrives with $o_1$ matching $\rho_{m_A}(s_1)$ and $c_1=1$ indicating $s_1\in G_{m_A}$.
Compute deterministic likelihoods: $\ell_{m_A}(y_1\mid s_1)=1$, all other $\ell$ values are $0$.
Propagate each state through its model dynamics $T_m(s,a_0)$ (identity in this toy example), then apply Eq. 13 to obtain $b_1$: only the pair $(s_1,m_A)$ retains non‑zero mass, with $b_1(s_1,m_A)=1$.
Since the belief is now concentrated, the LLM selects $d_1=\text{retain}$, the compiler keeps the existing executable, and verification succeeds trivially.
This tiny example shows how a single matching observation can collapse the joint belief to a single hypothesis, triggering immediate acceptance without further revision.
**Figure 2.** Continual world-model learning with a frozen LLM. The agent follows a continual loop of observation, reflection, rule revision, compilation, and verification. In this example, an observed badge change contradicts a mirror hypothesis. Reflection yields a clockwise-rotation rule, which is recorded in the rulebook and compiled into code. Verification checks the executable against the interaction history and the rulebook. The accepted model then guides further interaction, producing evidence for the next revision. Here $H_t$, $\psi_t$, and $\theta_t$ denote the interaction record, rulebook, and executable after $t$ real actions; $\pi_\phi$ denotes the LLM's output distribution with fixed parameters $\phi$.
**Figure.** How one rulebook is built: game tr87, one run, three of its eleven rulebook versions.
Model Selection and Planning
Choosing the simplest rulebook‑executable pair and planning with its deterministic model.
After verification the agent may hold many rulebook–executable candidates, but only one can drive action. The pain is picking a single hypothesis without exploding combinatorial cost.
Choosing the shortest combined description of rulebook $\psi$ and executable $\theta$ is like picking the lightest suitcase that still fits all your clothes – it keeps the agent’s world model compact while preserving fidelity.
Compute total description lengths: (5+7)=12, (4+9)=13, (6+5)=11.
The minimal sum is 11 bits for pair ($\psi$₃,$\theta$₃).
SimplifyLLM therefore selects ($\psi$₃,$\theta$₃) as the hypothesis for planning.
Even though $\psi$₃ is longer than $\psi$₂, its executable is much more compact, so the combined description wins – the selection balances rulebook and implementation size.
From the chosen pair ($\psi$,$\theta$) compute the internal state Ŝ_$\theta$(Hₜ) using the replay function.
Evaluate V_$\theta$(x) for the current state x = Ŝ_$\theta$(Hₜ) via the recursion in Eq. 26.
If V_$\theta$(x) < ∞, follow the minimizing action a = argminₐ V_$\theta$(f_$\theta$(x,a)) to obtain the first step of plan_$\theta$(x).
Execute a; observe outcome y and update belief $\tau$(b,a,y) for the next decision.
If V_$\theta$(x)=∞ (no goal‑reachable path), invoke the LLM to propose an exploratory sub‑goal and run an auxiliary planner to reach it.
How does SimplifyLLM differ from a naïve maximum‑likelihood model selection?
Maximum‑likelihood would pick the hypothesis that best fits the data regardless of its description length, potentially favouring overly complex rulebooks. SimplifyLLM incorporates the description‑length prior, so it prefers the shortest hypothesis that still explains the observations, balancing fit and simplicity.
Population Extension
Extending a single model to a population of hypotheses broadens exploration and avoids self‑confirming traps.
When a single model plans, it commits to one explanation of the rules and the current state, so a mistaken hypothesis can persist indefinitely if its actions never expose its errors (see Eq 28‑29).
Instead of keeping just one rulebook‑executable pair, the agent retains a small set of $N$ distinct hypotheses in parallel, letting each propose its own plan and thereby forcing the agent to gather evidence that distinguishes them.
Hypothesis 1 plans actions $[a_0, a_2]$; Hypothesis 2 plans $[a_1, a_3]$; Hypothesis 3 plans $[a_0, a_1]$.
All three are checked against $H_t$. Hypothesis 1’s first action $a_0$ matches the record, but its second action $a_2$ predicts a state not observed yet, so no falsification occurs.
Hypothesis 2’s first action $a_1$ mismatches the recorded $a_0$, yielding a prediction error; it is marked falsified.
Hypothesis 3’s plan exactly reproduces the recorded actions, so it remains unfalsified.
During the next planning episode, the agent uniformly samples from the surviving classes $\widehat{\Phi}_t = \{[\,\theta_1\,], [\,\theta_3\,]\}$, picking either hypothesis 1 or 3 with equal probability.
Even with only three hypotheses, uniform sampling forces the agent to try distinct action sequences, exposing errors that a single‑model approach would never encounter.
How does Population Extension differ from a standard ensemble of models?
Ensembles typically aggregate predictions after each step, but they do not share a common interaction record nor coordinate planning. Population Extension keeps all hypotheses synchronized on the same history $H_t$, samples a single behavioural class for execution, and uses falsification + repair to actively prune wrong models, rather than merely averaging outputs.
ARC-AGI-3 Realisation
We instantiate ARC-AGI-3, detailing its environment, rulebook, and continual learning pipeline.
The ARC‑AGI‑3 environment is a 64 × 64 colour grid POMDP where the agent only sees a local window; fog of war hides unseen cells, and actions are issued through a key‑click interface with initially unknown semantics. Completion is measured by the RHAE score, which compares the agent’s action count to a human baseline on each level.
ARC‑AGI‑3 is a deterministic, goal‑directed partially observable MDP whose state is a 64 × 64 pixel grid, and whose reward is defined by a human‑scaled efficiency metric (RHAE).
**Figure.** A schematic representation of a game state labeled "THE STATE" at level 3, time step 27-28. The diagram illustrates various components including an agent, an outline, a pinch point, a socket, a legend, a keypad, and a step bar.
**Figure 6.** The 25 public ARC-AGI-3 games.
The environment definition for ARC‑AGI‑3 couples a high‑resolution visual state with partial observability and a human‑scaled efficiency metric.
How does ARC‑AGI‑3 differ from the original ARC benchmark despite sharing a similar visual format?
ARC‑AGI‑3 adds a deterministic POMDP formulation, fog‑of‑war masking, and an explicit action‑efficiency score (RHAE). The original ARC tasks are static input‑output pairs without an interactive environment or a quantitative efficiency metric.
Experimental Results
CODE‑AS‑MODEL attains perfect RHAE on all ARC‑AGI‑3 games while using far fewer actions than humans.
CODE‑AS‑MODEL attains a perfect RHAE of 100 on every ARC‑AGI‑3 game.
Table 1 reports RHAE = 100 for all 25 games, yielding a mean of 100.0, with all 25 games and 183 levels cleared.
Ablation Studies
Removing the rulebook shows how much it speeds up learning.
To assess the contribution of the natural‑language rulebook we remove it from the update loop, keeping the code world model, harness, replay verifier, plan executor, backbone, and reasoning‑effort byte‑identical. We then run one game from each difficulty band—ft09 (easy), ka59 (medium), and cn04 (hard)—so that interaction difficulty is separated from the number of levels.
Ablating the rulebook reduces total actions by 9 %.
Across the three games the rulebook variant uses 617 actions versus 677 without it.
Ablating the rulebook reduces agent turns by 18 %.
The variant requires 678 agent turns versus 830 without the rulebook.
**Table 3.** Ablating the rulebook, one game per difficulty band. Difficulty is the human action baseline divided by the level count.
Population Scaling
Population extension cuts total actions to two‑thirds while improving most levels.
We evaluate the population extension by fixing the backbone, reasoning effort, and harness at their single‑model settings and varying only the number of retained pairs ($N = 2$ versus $N = 1$). The experiment runs on WA30, the game with the highest human‑action baseline among the public suite.
Population extension matches or improves per‑level action efficiency on eight of nine levels.
Table 4 shows per‑level ratios ≥0.97× for seven levels and 1.06× for L8, with total actions dropping from 899 to 597.
This table compares action counts across nine levels (L1-L9) for a Human baseline, an N=1 condition, and an N=2 population condition, including a ratio comparison between N=2 and N=1.
Atari Pong Case Study
Memento 3 keeps a natural‑language rulebook that is updated by reflection and compiled for planning.
The Atari Pong experiment replaces the level‑solving planner with a feedback policy that repeatedly queries the compiled world model to predict action consequences. The agent learns the dynamics by revising its rulebook as new ball‑bounce observations arrive, using full‑frame replay to expose prediction errors.
Our controller attains the maximum episode score of 21 in all three openings, while baselines fall short.
Table 5 reports a score of 21 for the learned controller and scores ≤ 21 for all model‑free and model‑based baselines.
Model‑free baselines (GDI, LASER, IMPALA, Rainbow) each require roughly 2.0 × 10⁸ frames, and model‑based EfficientZero variants need up to 4.0 × 10⁵ frames—orders of magnitude more interaction than our rulebook‑driven approach.
This figure illustrates a sequence of gameplay in an Atari Pong environment across three time steps: step 280, step 297, and step 304. The panels show the ball moving toward the agent's paddle, making contact, and then traveling back toward the opponent.
**Figure 8.** Pong episode scores. The three green curves show the learned controller under three different openings. Each reaches +21 without conceding a point. The same LLM without a world model finishes with a score of -19, and the random-policy episode shown here finishes at -20. The horizontal axis counts actions taken within each episode.
**Figure 9.** Three rulebook versions from one Pong learning session show how observed ball returns drive two revisions of the bounce rule shared by both paddles. (1) The initial four-band hypothesis excludes horizontal returns. (2) A horizontal return from a centre hit refutes this hypothesis and prompts a centred-deflection formula that permits $v_y = 0$. (3) Further contacts reveal that the formula overestimates vertical return speeds, leading to an asymmetric band map with $|v_y| \le 2$. Contact offset is the ball’s top row minus the paddle’s top row; $v_y$ denotes vertical velocity in pixels per emulator frame. Red marks claims revised later, and green marks new evidence or revised rules. Dashed white trails show recorded ball positions before contact, while solid white trails show positions after contact; step labels identify the supporting observations.
Related Work
We survey prior approaches to world modeling, language‑guided program induction, and reflective memory that inform our design.
World models support decision making by predicting action consequences, and many works combine real interaction with model‑generated experience for planning. Dyna interleaves real and simulated data, while neural world models enable imagined rollouts and tree search. Programs offer an alternative by exposing transitions as executable functions that planners can query directly.
Language‑guided program induction builds models from natural‑language specifications and tests them against input‑output examples. Parsel decomposes textual descriptions into executable components, and Hypothesis Search translates hypotheses into programs verified on examples. WorldCoder learns Python dynamics models and repairs them using counterexamples, while GIF‑MCTS searches candidate models guided by descriptions and trajectories.
Several ARC‑AGI‑3 agents tightly couple model construction, verification, and action selection. Rodionov’s baseline maintains a persistent Python model and verifies it before planning, though verification does not always improve performance. OPINE‑World synthesises object‑centric executables from natural‑language hypotheses, and Tycho studies when agents should construct, repair, or bypass executable models.
Textual memory lets frozen LLM agents incorporate interaction feedback as editable rules. Reflexion stores linguistic reflections on task feedback, AutoManual organises environment rules into reusable manuals, and WALL‑E updates symbolic rules by comparing predicted and observed trajectories. These approaches treat language as a mutable medium for both experience and rule knowledge.
The Memento series progresses from episodic reflective memory to stateful learning that links memory writing and reading to policy improvement. Code as Model extends this line by pairing a persistent natural‑language rulebook with its compiled executable, enabling joint revision of rules and code. Reflection can repair an environment rule or its implementation while retaining the rule, and replay checks the executable against recorded transitions.
Version spaces capture the ambiguity left by finite evidence, and minimum description length prefers simpler consistent hypotheses. Model‑based reinforcement learning represents uncertainty with probabilistic dynamics or ensembles, and posterior sampling retains a sampled hypothesis across an episode for coherent exploration. Our population extension maintains multiple rulebook‑executable pairs in parallel, approximating the version space and reducing reliance on a single early model.
Continual adaptation treats external software and workspaces as editable learning state. DreamTeam’s Workspace Optimization builds models, probes hypotheses, and plans; Continual Harness enables reset‑free prompt and skill revision; NOOA combines a Python object‑oriented interface with persistent modules and prediction‑error‑driven refinement. Belief‑Calibrated Optimization uses a language world model to guide prompt edits, while Code as Model predicts environment transitions for planning.
ARC‑AGI‑3 provides a shared benchmark for studying agents under unfamiliar rules and a human‑derived action budget. It also supports graph‑based exploration that records observed transitions and prioritises untested actions without a generative world model. Component studies report near‑saturation of the public set, indicating that improvements must consider model choice, reasoning budget, and action efficiency.