InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation
Zhuo Lin, Sirui Xu, Liuyu Bian, Yu-Xiong Wang, Liang-Yan Gui
InterEvolve adapts fixed humanoid controllers to novel loco-manipulation tasks by evolving reward programs at test time.
How can a humanoid robot learn to perform novel manipulation tasks at test-time without retraining its underlying motor policy?
Humanoid controllers trained for specific tasks often fail when faced with novel objects or complex, multi-stage interactions, as they lack a flexible interface to repurpose their existing motor skills. InterEvolve addresses this by representing tasks as "reward programs"—staged objectives with tunable constants—that an object-aware behavioral foundation model executes without retraining. An LLM agent iteratively revises these programs based on execution feedback, while a numerical optimizer calibrates their parameters. This approach allows a single fixed controller to solve diverse, long-horizon tasks, doubling the success rate of the best calibrated fixed programs on complex benchmarks.
Paper Primer
The core mechanism hinges on decoupling task specification from motor control: the system uses a forward-backward (FB) behavioral foundation model that maps any reward program into a latent prompt, which then steers a frozen, object-aware policy. The evolution process acts like a test-time search, where an LLM agent proposes structural changes to the reward stages, and a numerical optimizer (CMA-ES) fine-tunes the weights to ensure the controller's existing motor competence is effectively triggered.
Evolution-guided reward programs significantly outperform static, human-designed rewards in loco-manipulation tasks.
On a benchmark of eight task families, InterEvolve achieved an 86.5% success rate, more than doubling the 34.6% success rate of the best calibrated fixed program.
The system successfully enables autonomous long-horizon composition and real-world deployment.
Evolved skills were deployed on a physical Unitree G1 robot, and the system successfully chained multiple stages (e.g., carry, place, kick) by reusing programs from a skill library. Successful execution of multi-stage, contact-rich interactions on physical hardware.
Why is a reward program more effective than simply providing a goal state or motion reference?
Reward programs allow for the specification of contact-rich, multi-stage interactions that are difficult to capture with static goal states. By decomposing a task into phases with completion conditions, the system can adapt its strategy dynamically based on execution feedback.
Does this approach require retraining the humanoid controller for every new task?
No. The controller and its behavioral foundation model remain frozen. All adaptation happens at test time by evolving the reward program, which changes the latent prompt sent to the fixed policy.
Introduction and Motivation
Test‑time evolution lets a frozen humanoid controller discover and reuse its own skills for new tasks.
Humanoid controllers now exhibit broad whole‑body motion and contact‑rich loco‑manipulation, yet they encounter tasks in open‑ended environments that were never seen during training. When asked to tip a box onto another face, a controller that can push, lift, and carry boxes lacks a demonstration for this novel behavior, even though the required motions lie within its existing skill set. Test‑time evolution lets the robot discover and repurpose its own skills, improve from its own attempts within a trial budget, and retain what it learns so later composite tasks can build on earlier experience.
The robot already knows many motor primitives, but we need a way to discover and reuse them at test time without costly retraining.
InterEvolve realizes the needed interface with two components. First, an object‑aware Forward‑Backward (FB) behavioral foundation model maps rewards to whole‑body actions while reading object features via trainable residuals. Second, reward programs—staged rewards with completion conditions and tunable constants—are iteratively revised by a large language model (LLM) agent that uses execution feedback and a skill library, while a CMA‑ES optimizer tunes the constants; each candidate is verified across parallel simulation scenarios.
**Figure 1.** InterEvolve adapts a frozen whole-body controller with novel use. The controller trained on human-object interaction (HOI), such as lifting and pushing a box, is asked to rotate a box around in place, a behavior absent from its training data (Sec. D.7 and Figure 11). A human-designed reward tries to hack this behavior but fails. The reward program that InterEvolve evolves compensate with novel body used to succeed. **Bottom:** InterEvolve can achieve versatile evolved skills in simulation and on a real robot. More demos are on the project page.
The key shift is moving task‑specific compute from costly training to fast, test‑time reward evolution.
Related Work and Skill Library
We situate InterEvolve among prior humanoid control, foundation models, and test‑time search approaches.
InterEvolve builds on four strands of prior work: reusable humanoid loco‑manipulation policies, behavioral foundation models, language‑driven reward design, and test‑time search over task specifications.
Methods that learn whole‑body policies capable of both locomotion and object interaction, often via tele‑operation or motion‑capture data.
Foundation models that encode a broad repertoire of behaviors and can be steered by latent prompts derived from rewards or language.
Techniques where language models generate or refine reward functions, which are then evaluated by executing policies or simulators.
Approaches that allocate additional inference compute at test time to refine decisions, often via sampling, step‑level search, or program synthesis.
A persistent collection of verified reward programs that the agent can reuse when tackling new tasks.
The Evolution Mechanism
Execute reward programs without retraining and evolve them via a nested LLM‑driven loop.
Training a fresh policy for every candidate reward program would make test‑time adaptation infeasible, so the authors introduce a frozen behavioral foundation model that can execute arbitrary programs directly.
The FB model factorizes a policy’s discounted future‑state occupancy into a forward map that predicts how actions affect states and a backward map that scores states, allowing a single latent code $z$ to index many behaviours.
Compute the dot products: $F^{\top}B(s_1)=0.5$, $F^{\top}B(s_2)=0.5$.
Form the occupancy approximation $M_{\pi_z}(ds'\mid s,a)\approx[0.5,0.5]$ over the two states.
Given a reward vector $r=[2, -1]$, evaluate $Q_{\pi_z}(r)=0.5\cdot2 + 0.5\cdot(-1)=0.5$.
The FB model turns a full rollout into a cheap linear algebra operation; changing $r$ only changes the weighted sum, not the forward/backward caches.
A Reward Program is a staged specification of scalar rewards that the FB model consumes to produce a latent prompt $z_{j,t}$ at each control step.
How does this Reward Program differ from classic reward‑shaping where a scalar reward is added to the environment?
Here the program is staged and each stage produces its own latent prompt that directly modulates the frozen policy’s action distribution. Classic shaping merely adds a scalar term to the return; it does not change the policy’s internal representation or enable object‑conditioned residuals.
Initialize the current best program by prompting the LLM with the task, scene, verifier, and skill library.
For each outer‑loop round $k=1\ldots K$:
Prompt the LLM with the current best program and its rollout feedback; obtain a set of candidate programs.
Discard any candidate that violates code rules (e.g., reads an undeclared feature).
For each remaining candidate, run the inner loop: freeze its structure, sample constants $\theta$ with CMA‑ES, evaluate on parallel evolution scenarios, and update the sampling distribution toward higher scores.
Compare the tuned candidates to the current best on the same scenarios; select the top‑scoring candidate.
Re‑evaluate the top candidate and the current best on confirmation scenarios three times each; replace the current best if the improvement exceeds evaluation noise.
If the current best satisfies all verifier criteria, terminate early; otherwise continue to the next round.
After termination, add the final program and its verifier results to the skill library.
**Figure 2.** **InterEvolve overview.** (a) Given a task and scene, an LLM agent writes a staged reward program, drawing on verified programs in the skill library. (b) The forward-backward model scores the active stage's reward over a fixed state bank and projects it into a latent prompt, which a frozen body prior and a pretrained object residual execute without retraining. (c) A fixed verifier scores parallel rollouts, and criterion-level feedback drives the next structural revision (outer loop), while CMA-ES tunes the program constants (inner loop).
**Algorithm 1: Reward-program evolution at test time.** **Input:** task $\ell$, scene $c$, verifier $\mathcal{C}$, motor model $\pi$ with bank $\mathcal{B}$, skill library $\mathcal{H}$, round budget $K$. 1. The agent writes an initial program from $\ell$, $c$, $\mathcal{C}$, and $\mathcal{H}$; evaluate it and make it the current best. 2. **for** $k = 1, \dots, K$ **do** 3. Prompt the agent with $\ell$, $c$, $\mathcal{C}$, $\mathcal{H}$, and the current best program with its rollout feedback. 4. Parse and validate the proposed programs; discard invalid candidates. 5. *Inner loop*: tune the constants $\theta$ of each valid program with CMA-ES on the search scenarios. 6. Compare the tuned candidates with the current best on the same search scenarios. 7. Re-evaluate the top candidate and the current best on the confirmation scenarios; the candidate becomes the current best if its gain exceeds evaluation noise. 8. Summarize criterion- and stage-level feedback. 9. **if** the current best satisfies every criterion in $\mathcal{C}$ **then** break 10. **end for** 11. Add the current best program and its verifier results to $\mathcal{H}$. **Output:** the current best program.
Experimental Setup
InterEvolve’s test‑time adaptation dramatically boosts success while staying efficient.
InterEvolve attains an 86.5 % success rate on goal‑conditioned tasks, more than double the best fixed controller.
ULTRA achieves 15.68 % SR and BFM‑Zero 21.45 % SR on the same benchmark; InterEvolve reaches 86.5 % SR.
We pretrain the FB models on the OMOMO and GRAB datasets, then retarget them to the Unitree G1 robot with both rubber and Inspire hands using OmniRetarget. For each of the four box‑like objects we hold out 50 clips for evaluation and train on the remaining 3,866 clips, using the held‑out large‑box clips as the tracking benchmark. All controllers run in Isaac Lab, and DeepSeek‑V4‑Flash generates the reward programs.
**Table 2.** Comparison of different controllers and programs.
Performance and Ablations
Baselines. For tracking, we adapt the tracking policy of ULTRA (He et al., 2026), a generalist trained with motion imitation to follow loco‑manipulation references, which we train on the same data as our motor model, and with BFM‑Zero (Li et al., 2026), a pretrained body‑only FB model. For general‑purpose tasks, no public method applies directly to our setting under complex scene context, so we compare InterEvolve with several variations on the same controller: a human‑written program and the agent’s initial program, each executed directly or after CMA‑ES calibration.
4.2 REUSABLE CONTROL FOR REFERENCE TRACKING
Object‑aware FB can match a dedicated tracker. Table 1 tests whether one controller can mimic body motion and object interaction. ULTRA, an RL tracker trained for tracking alone, has the lowest errors. Our motor model, which also serves reward programs, approaches its success rate, whereas the body‑only BFM‑Zero loses the object on most clips, so object‑aware pretraining supplies the interaction competence that reward programs repurpose. Without the link‑to‑surface vectors or the object state, the controller still follows the body but rarely succeeds (Table 6), and the same network trained from scratch almost never works, whereas residuals on the frozen body prior improve with width (Table 7), so the prior carries body skills and the residual learns interaction.
Evolving rewards unlock the potential for motion tracking. Reward design matters even in tracking. Our actor observes the body and object in a local frame, so world‑frame drift from the reference is invisible to it. The agent finds a reward that supplies this signal: an object‑anchored drift correction blended with the reference prompt (Sec. B.2). It reduces both errors, lifts success above ULTRA, and nearly halves the final drift of root and object. The same procedure also tracks dexterous whole‑body manipulation of small objects with Inspire hands.
4.3 REWARD‑PROGRAM EVOLUTION FOR GENERAL‑PURPOSE TASKS
We next remove the reference motion and specify only task outcomes and constraints. The benchmark spans eight task families (Sec. D.3): pushing to a mark, carrying at several heights, tipping or reorienting an object, feet‑only interaction, lifting onto a support, and pushing through gates or around obstacles. Success depends on contact mode, obstacle layout, and when to switch phases. Each run starts from the task text and the observed scene, and only the reward program changes.
Formulation, not hyperparameters, limits a written reward. Table 2 separates three effects. A reward written once is not enough: even the agent’s initial program, which outperforms the human‑designed one, fails in most episodes. CMA‑ES calibration of the weights alone brings limited gains, since better weights cannot repair an objective with the wrong structure. InterEvolve more than doubles the success of the best calibrated program, and because both fixed programs are calibrated, this gap comes from structural changes: what is rewarded and how objectives are staged.
Table 3: Ablation of the search design. Stages and numerical calibration are most important designs. Each row disables one component (✗) of the full system; definitions are in Sec. D.5.
Search components | Results --- | --- CMA‑ES tuning | ✓ ✗ ✓ ✓ ✓ ✓ Targeted edits | ✓ ✓ ✗ ✓ ✓ ✓ Multi‑scenario | ✓ ✓ ✓ ✗ ✓ ✓ Scene context | ✓ ✓ ✓ ✓ ✗ ✓ Multi‑stage | ✓ ✓ ✓ ✓ ✓ ✗ SR (%)↑ | 86.5 51.6 68.9 68.4 78.7 44.7 Earned (%)↑ | 95.6 89.9 92.3 89.9 93.2 86.9 GPU‑h ↓ | 2.1 1.7 2.7 1.0 2.0 2.1 Tokens (M) ↓ | 0.23 0.27 0.20 0.17 0.21 0.19
Evolution helps most where fixed programs fail. Calibrated fixed programs solve pushing to a mark but succeed in at most a quarter of the episodes when carrying at chest height, tipping onto a new face, or pushing through a gate or around an obstacle (Table 13). Evolution lifts each of these families above 70% success. Kicking to a mark remains the hardest, since kicks are rare in the training data and need strong whole‑body coordination from the reward.
Staging and calibration matter most. Every evolution component contributes (Table 3; Sec. D.5 defines each variant). Forcing a single stage causes the largest drop, so contact‑rich interaction needs a task decomposed into phases with their own objectives. Removing numerical tuning is nearly as harmful, since a program can only be judged once its constants are calibrated. Dropping targeted edits, so that the agent rewrites each program from scratch, or multi‑scenario evaluation, so that each candidate runs from a single start, costs a similar amount: repairing a working program is more reliable than regenerating it, and varied starts tell a robust edit from a lucky one. Scene context, which describes obstacles and supports, has the smallest effect but still helps.
Success grows with test‑time scaling, though not monotonically. Table 4 tracks the current best program over rounds, with the controller weights fixed. The first round brings most of the gain by removing the initial program’s coarse failures, and later rounds resolve the remaining criteria. Earned tiers rise monotonically, whereas full success dips from round 2 to round 3, since it requires all criteria in the same rollout and fixing one criterion can briefly break another. Tokens grow linearly with rounds, but simulation dominates the cost, with minutes per round in the language model against tens of minutes of rollouts (Sec. C.4), so a faster simulator such as mjlab (Zakka et al., 2026) in place of Isaac Lab (Mittal et al., 2025) could shorten evolution substantially. Figure 7 shows that revised rewards change the contact strategy itself.
Qualitative results and real deployment from onboard perception. Figures 8, 9, and 10 show goal‑conditioned skills evolved in simulation, in which acquisition, transport, placement, and handling arise from reward programs run by the shared controller. Figure 4 deploys programs evolved for two tasks on a physical Unitree G1. The robot detects the box with its egocentric camera, and FoundationPose (Wen et al., 2024) estimates its 6‑D pose, which supplies the controller’s object features. Each program is evolved and verified in simulation, then run on the robot.
Table 4: Ablation on search rounds. Most of the gain comes in round 1; earned tiers rise monotonically. The final row repeats Table 2.
Round | SR↑ | Earned↑ | Tokens --- | --- | --- | --- Initial | 32.2 | 73.3 | 0.03 1 | 75.0 | 91.7 | 0.08 2 | 76.8 | 92.6 | 0.13 3 | 73.6 | 93.1 | 0.18 4 (final) | 86.5 | 95.6 | 0.23
4.4 REUSE AND COMPOSITION
Table 5: Reuse on composite tasks unseen by every library, under one target‑task budget. The full library solves the most episodes. Carry‑only keeps the three carrying programs. Tokens (millions, mean over the tasks) exclude library construction. Every condition searches for three rounds.
Library | Relocate | Stack | Carry‑place‑kick | Tokens --- | --- | --- | --- | --- Full | 8/10 | 4/10 | 5/10 | 0.16 None | 0/10 | 0/10 | 1/10 | 0.13 Carry‑only | 6/10 | 1/10 | 1/10 | 0.15
Composite tasks need a library that covers their contact modes. Evolved experience is valuable only if later tasks can reuse it. Table 5 evaluates three composite tasks that each combine previously evolved objectives in one episode: relocating an object between supports, stacking one box on another, and carrying, placing, then kicking an object. We compare the full skill library with no library and a carry‑only library. With the full library, the agent solves most relocation episodes and about half of the stacking and carry‑place‑kick episodes, but almost none without a library. The carry‑only library recovers most of relocation but helps little on stacking or kicking. Reuse therefore depends on covering the contact modes a task needs, not on having any stored experience.
Evolving adapts stored programs. A finer ablation on carry, place, and kick shows that stored programs run without evolution almost never solve the task (Table 16), so the gain comes from evolution integrating skill library rather than from direct retrieval. Broader coverage still helps.
Evolution for long‑horizon tasks. The evolution chains searched programs to arrange six scattered boxes into a ring. The same agent for evolution orders the boxes, and each leg runs a walking program to the box followed by the evolved pushing program. One controller executes all twelve phases from a single reset, so errors accumulate across legs instead of being reset away. In the take shown, all six boxes end within 11 cm of their cells and no placed box is disturbed by later legs. More are on the project page.
5 DISCUSSION AND CONCLUSION
InterEvolve suggests a different division of labor for humanoid intelligence. A behavioral foundation model learns how to move once, and task knowledge lives outside its weights, in reward programs that a language model can read, edit, and test by execution. Reasoning and control thus improve on separate timescales: the controller with more interaction data, the agent with more test‑time search and a growing program library, and neither requires retraining the other. Physical execution grounds the agent’s reasoning, since it revises what it asks of the controller from observed outcomes rather than assumptions. We see this as a step toward humanoids that, like language models at inference time, improve by testing more, and accumulate skills as inspectable, reusable programs.
Acknowledgments. We thank Derek Zhang for help with the real‑world deployment.
Motor Model Details
Details the pretrained motor model, its interaction encoding, residual architecture, and reward inference components.
The motor model is frozen after a single pretraining phase and then reused unchanged throughout all reward‑program searches.
The motor model converts a combined body‑pose and object‑state observation into joint‑level commands, acting as a deterministic controller while the LLM evolves reward programs.
**Table 13.** Success per task family (%). CMA denotes CMA-ES. The macro-average weights families equally.
**Table 7.** Residual capacity. MLP width $\times$ depth of the residual branch; the full-width residual mirrors the prior's module architectures and is the main model of Table 1 (without replanning). Parameter counts include all trainable modules. Errors and SR follow Table 1.
**Table 8.** Drift-correction ablation. Tracking protocol of Table 1; $E_h$ and $E_o$ in cm. The anchor is the quantity steered toward the reference, $k_p$ is the correction gain, and the shaded row is the configuration used in Table 1.
**Table.** Performance comparison across different task families.
**Figure 11.** **Evolved skills versus pretraining references.** Each point is one motion: an OMOMO largebox training clip (pink) or one rollout of an evolved skill (colored, 16 per skill; circles mark skill centers). A motion is described by its body posture relative to the pelvis and heading, summarized over time, and embedded with UMAP.
These diagnostics and tables together establish the motor model as a stable, object‑aware controller that can be leveraged unchanged while reward programs are iteratively evolved.
Reward Program Semantics
Details of reward program structure, agent prompt, selection process, and evolution cost accounting.
This appendix formalizes the reward‑program language used by the LLM and shows how programs are evaluated.
Reward code reads pre‑computed bank features (F) while live context (C) supplies per‑frame measurements; keeping them separate ensures the same reward definition applies to every rollout.
A concrete two‑stage program for the “`lift_rel`” task uses seven constants and defines separate reward and subgoal functions for each stage.
The agent receives a plain‑text prompt each round; the prompt first explains the role of a staged reward and then lists the allowed feature vocabulary.
Programs may only reference a fixed set of bank features (F) and live scalars (C) that the LLM must embed in its reward code.
Code rules enforced by a validator require reward(F,C,W) to return an (N,) tensor built from F, never from C, and transition(C,W) must return a Python bool without reading F.
Search proceeds in three phases: Explore varies stage decomposition, Refine tweaks reward terms, and Tune makes small weight adjustments with CMA‑ES.
**Table 12.** Task families for program search. The verifier scores outcomes and constraints independently of the agent-written reward.
**Figure 5.** **World-frame drift over time.** Horizontal distance between the simulated and reference root (left) and object (right) for native tracking and the object-anchored drift correction, on the 50 largebox clips of Table 1. Lines show the median and bands the 25–75% range over all 50 motions; each motion is counted to the end of its clip, and after its clip ends its final value is held. Without correction the median root drift grows to 0.29 m at 4 s and 0.32 m by the end, against 0.16 m and 0.18 m with the correction, which levels off after about 4 s. The two curves coincide for the first second because the correction has a 0.10 m deadband.
**Figure 6.** Reward tilt, bank size, and cost on the feet-only task. (a) SR against the reward tilt $\beta$ with 50k states. (b) SR against bank size at $\beta = 10$. (c) Peak memory (solid) and projection latency (dashed) against bank size.
Evolution cost counts every candidate rollout: with $\lambda_{E}$S=4, G=3, S=16 the cost is 192 rollouts per program, and each confirmed candidate adds another 192 rollouts, leading to 6.7–14.2 M simulation steps per five‑round search.
Experimental Protocols
Experimental details, additional results, and protocol limitations for the InterEvolve system.
Figure 10 demonstrates that the push‑to‑mark program, unchanged from the main experiments, generalizes to unseen goal bearings and box start positions, with success only dropping at the widest angles.
**Figure 10.** **Push to mark under new goals and starts**, with the selected program unchanged. (a) From one start, targets at bearings from $-45^\circ$ to $+45^\circ$, 2.5 m away; rings mark the targets. (b) The box starts 0.55, 1.2, or 1.8 m from the robot at bearings of $-40^\circ$, $0^\circ$, and $+40^\circ$, and is pushed to one target (red ring). Lines trace the box; each rollout shown is one of 16 per setting.