Look Before You Leap: Distilling Tree Search into Action Evaluation for Frozen VLA Models

Xinyi Xie, Zican Hu, Zhanyu Liu, Yicheng Dong, Wenhao Wu, Zhenhong Sun, Haoran Li, Chunlin Chen, Zhi Wang, Pichao Wang

SVA improves frozen VLA performance by distilling Monte-Carlo tree search into a lightweight consequence evaluator.

How can we improve the success rate of frozen Vision-Language-Action (VLA) models without retraining them, by using a lightweight value model to rerank action candidates?

Vision-Language-Action (VLA) models often fail on tasks outside their training distribution, not because they lack the ability to perform the task, but because they cannot distinguish successful actions from mediocre ones during single-shot generation. The authors propose Search, Value, and Act (SVA), a framework that uses Monte-Carlo tree search (MCTS) in simulation to discover high-return trajectories, which are then distilled into a lightweight Q-value model. At deployment, this evaluator ranks candidate actions proposed by the frozen VLA, enabling test-time scaling without updating the backbone. SVA consistently outperforms base policies across benchmarks, with a 9B model achieving higher success rates than a 27B model at 27% lower latency.

Paper Primer

The core mechanism hinges on decoupling action proposal from consequence evaluation. The VLA backbone acts as a proposal distribution, while a small, separately trained Q-model acts as a "look-before-you-leap" verifier: it scores candidate actions based on predicted long-term outcomes, allowing the system to select the most promising path without requiring simulator access at runtime.

SVA enables test-time scaling that is more cost-effective than increasing model size.

A 9B VLA with SVA outperforms a 27B VLA by 7 percentage points on manipulation tasks. 27% lower inference latency compared to the 27B baseline.

Frozen VLAs already contain competent behaviors in their output distribution.

A diagnostic pass@k study shows success rates rise from 33% at pass@1 to 92% at pass@32. 58-point absolute gain in success rate through multiple sampling.

Why use a learned Q-model instead of just running MCTS at deployment?

MCTS is computationally expensive and requires a simulator, which is impractical for real-time robotics. The Q-model distills the search knowledge into a fast, deployable function that generalizes across states.

Does this method require fine-tuning the VLA backbone?

No. The VLA backbone remains entirely frozen, preserving its generalist capabilities while the lightweight Q-model (using LoRA adapters) handles action evaluation.

Researchers should treat action evaluation as a first-class lever for VLA improvement, as it allows for performance gains that are orthogonal to, and often more efficient than, costly backbone fine-tuning.

Introduction: The VLA Evaluation Gap

We expose VLA evaluation limits and propose SVA to rerank frozen proposals at test time.

Vision‑Language‑Action (VLA) models can generate diverse action proposals, yet they lack the ability to assess which proposals will succeed long‑term. A diagnostic pass@k study shows that frozen VLAs already contain many successful actions, but single‑shot generation fails to pick them.

The VLA can suggest many actions, but without a way to evaluate their downstream outcomes it often selects poor ones.

SVA equips a frozen VLA with a lightweight evaluator that predicts the long‑term consequence of each candidate, allowing test‑time reranking without retraining the backbone.

**Figure 1.** Post-training vs. test-time scaling.

VLA models are good at proposing actions but bad at knowing which ones lead to success.

The Pass@k Diagnostic

Pass@k analysis reveals latent competence in frozen VLAs but highlights evaluation bottlenecks.

Increasing the number of sampled rollouts from 1 to 32 raises average success from 33 % to 92 %, a 58‑point absolute gain.

Measured across OpenVLA on SIMPLER, LIBERO, and $\pi$0.5 on ROBOTWIN (N = 50 rollouts per task).

Pass@k measures the chance that at least one of k independent attempts succeeds, exposing how much useful behavior already lives in a model’s output distribution.

**Figure 2.** Pass@k results across embodied benchmarks.

Pass@k performance scales with candidates, proving the model has the right action in its distribution but lacks the ability to pick it.

The SVA Method

We consider an embodied task as a language‑conditioned MDP ⟨S, A, T, R, $\gamma$, $\chi$⟩, where S and A are the state and action spaces, T(s′|s,a) is the state transition function, R(s,a) is the reward function, $\gamma$ is the discount factor, and $\chi$ is the language space. At each step t, the agent receives a state $s_t$ ∈ S that may comprise an egocentric image, robot proprioception, or both, and selects an action $a_t$ ∈ A according to a policy $\pi$(·|$s_t$; l) conditioned on the instruction l ∈ $\chi$ that describes the task (e.g., “open the door”).

We adopt an inference strategy to maximize the expected return of a frozen VLA policy $\pi$_$\theta$: at each step t we draw a set of N candidate actions {$a^{(1)}$,…,$a^{(N)}$} from $\pi$_$\theta$(·|$s_t$; l), and select the action that maximizes a learned Q‑function that predicts the expected consequence of each candidate, $\hat Q_\phi(s_t, a^{(i)}; l)$. This formulation treats $\pi$_$\theta$ as a fixed proposal distribution and redirects selected actions toward higher‑return regions via $\hat Q_\phi$, enabling policy improvement by scaling test‑time compute. Our setting is agnostic to the action granularity: a chunk, a high‑level skill (e.g., “find the apple”), or a low‑level control primitive (e.g., a continuous end‑effector displacement). Below we sometimes omit the notation l for ease of reading.

### 4.1 Search: Mining Evaluation Signals via MCTS

Recent work on RL for foundation models reinforces the view of the “Bitter Lesson”: the true value of RL lies not in parameter updates themselves, but in the search process – rolling out policies, comparing outcomes, and assigning credit. We employ Monte‑Carlo Tree Search (MCTS) to fully explore the VLA’s policy distribution, performing structured look‑ahead search to discover highly informative trajectories.

Each edge (s, a) of the search tree stores an action‑value Q(s,a), visit count n(s,a), and prior probability P(s,a). The tree is traversed starting from the root state with the following procedure:

**Selection.** Action $a_t$ is selected using the PUCT (Predictor Upper Confidence bound for Trees) rule

$$ a_t = \arg\max_a \Bigl[ Q(s_t, a) + c_{\text{puct}} \frac{P(s_t, a)}{1 + n(s_t, a)} \Bigr], $$

which maximizes the action value plus a bonus proportional to the prior probability but decaying with repeated visits to encourage exploration; $c_{\text{puct}}$ controls the exploration degree.

**Expansion.** When the traversal reaches a leaf node $s_L$ at step L, the leaf node may be expanded. The policy $\pi$_$\theta$ processes $s_L$ once to sample N candidate actions {$a^{(1)}$,…,$a^{(N)}$}; each candidate is added to the tree as a new edge, with the successor state as the corresponding new leaf node. The output probabilities are stored as prior probabilities P for each action, $P(s_L, a) = \pi_\theta(a|s_L)$.

**Evaluation.** From the newly expanded leaf node $s_L$ we estimate its value by performing a simulation rollout. Specifically, we clone the simulator state at $s_L$ and execute the policy $\pi$_$\theta$ forward for up to D steps (or until task termination), accumulating the discounted return

$$ G(s_L) = \sum_{i=0}^{D-1} \gamma^{i} r_{L+i}. $$

This rollout provides a Monte‑Carlo estimate of the long‑term consequence of reaching $s_L$ under $\pi$_$\theta$.

**Backup.** After obtaining the rollout return G($s_L$), we propagate it back along the traversal path from the leaf to the root. For every edge (s, a) visited during selection, the statistics are updated

$$ n(s,a) \leftarrow n(s,a) + 1, \qquad Q(s,a) \leftarrow Q(s,a) + \frac{G(s_L) - Q(s,a)}{n(s,a)}. $$

Thus the visit count is incremented and the action value is updated as a running mean of all rollout returns that have passed through that edge. These updated statistics refine the PUCT scores in subsequent iterations, progressively biasing the search toward higher‑return branches of the tree.

After several iterations of the selection‑expansion‑evaluation‑backup loop under a finite budget, for each task we collect a few trajectories from diverse search episodes, including both successful and failed rollouts, providing valuable contrastive signals for learning relative action quality (Sec. 4.2).

### 4.2 Value: Learning a Deployable Consequence Evaluator

Having used MCTS to search for informative trajectories, we now turn to the complementary pillar of the Bitter Lesson: learning. Rather than relying on expensive tree search at every deployment step, we distill the knowledge discovered by MCTS into a lightweight model $\hat Q_\phi$ that predicts the expected consequence of candidate actions. This amortization serves two purposes: (i) it compresses the rich contrastive signals produced by search into a compact, fast‑to‑evaluate function, enabling real‑time action selection without simulation; and (ii) because neural networks generalize across states, the learned $\hat Q_\phi$ can transfer the credit‑assignment insights obtained from searched trajectories to unseen situations/tasks, effectively extending the reach of search beyond its original computational budget.

The consequence evaluator $\hat Q_\phi(s,a; l)$ is built on a lightweight VLM backbone (e.g., Qwen‑3.5‑0.8B), augmented with LoRA adapters and an ensemble of small MLP value heads. We append a special <VALUE> token to a prompt template containing the textual instruction l, visual/proprioceptive inputs s, and the candidate action a. The hidden state positioned at this token after self‑attention is fed into the ensemble of value heads. The ensemble mean is used as the predicted Q‑value, while the standard deviation provides an uncertainty estimate. The model is optimized using a Smooth‑L1 loss that provides stable gradients on large errors (like L1) and smooth optimization on small errors (like L2):

$$ L(\phi) = \begin{cases} \frac{1}{2}\,\epsilon^{2} & \text{if } |\epsilon| < 1,\\[4pt] |\epsilon| - \frac{1}{2} & \text{otherwise}, \end{cases} \qquad \epsilon = \frac{1}{B}\sum_{i=1}^{B}\bigl(\hat Q^{(i)}_\phi(s,a; l) - Q(s,a; l)\bigr), $$

where B is the number of value heads, and target values $Q(s,a; l)$ are provided by the collected data in Sec. 4.1 and normalized by dataset statistics. Only the value heads and LoRA adapters are fine‑tuned.

### 4.3 Act: Evaluation‑Guided Action Selection at Test Time

At test time we use the VLA policy $\pi$_$\theta$ as a proposal distribution and the learned Q‑model $\hat Q_\phi$ as a verifier. The frozen $\pi$_$\theta$(·|$s_t$; l) generates N candidate actions {$a^{(1)}$,…,$a^{(N)}$}, where the empirical prior of each candidate is estimated by its sampling frequency

$$ p(a \mid s_t; l) = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{a^{(i)} = a\}. $$

Finally, we select the candidate with the highest uncertainty‑regularized Q‑value:

$$ a_t = \arg\max_{i\in\{1,\dots,N\}} \Bigl[ \mu_\phi(s_t, a^{(i)}; l) - \lambda_1 \sigma_\phi(s_t, a^{(i)}; l) + \lambda_2 \log p\bigl(a^{(i)} \mid s_t; l\bigr) \Bigr], $$

where $\mu_\phi(\cdot)$ and $\sigma_\phi(\cdot)$ are the mean and standard deviation of the Q‑ensemble, and $(\lambda_1, \lambda_2)$ are regularization coefficients. Action $a_t$ is executed until completion, task termination, or invalid feedback.

Experimental Setup

We detail datasets, baselines, latency, and scaling settings used to evaluate SVA.

Inference latency follows a best‑of‑$N$ scheme: $N$ forward passes of the VLA generate candidates and $N$ passes of the lightweight Q‑model score them. Because the Q‑model (≈0.8 B parameters) is orders of magnitude smaller than the VLA backbone (≈7 B), scoring adds only minimal overhead. Consequently, total generation cost stays far below the naïve $N$‑times estimate for modern VLAs.

The best‑of‑$N$ approach provides a test‑time compute knob: raising $N$ raises the chance of picking a high‑quality action. This adaptive scaling lets harder tasks receive more compute while easy tasks stay cheap. Empirically, scaling the evaluation budget yields better cost‑effectiveness than enlarging the model.

We evaluate SVA across five VLA backbones: GPT‑4o, Qwen3.5‑4B/9B/27B, and Gemma‑4‑E4B‑it. Comparisons include OpenVLA, the $\pi$0/$\pi$0.5 VLAs trained on large‑scale robot data, and $\pi$0/$\pi$0.5 + RoboMonkey, which reranks candidates via a synthetic preference model. This suite tests SVA’s model‑agnostic claim and its advantage over single‑step preference reranking.

**Figure 6.** Scaling candidates $N$ improves success rate while Q-model scoring adds negligible overhead versus proposal latency (Qwen3.5-9B).

Table 1 reports success rates on the EmbodiedBench suite for each backbone and the corresponding SVA‑augmented variants. The results illustrate the gains (green) and drops (red) relative to the base policies, providing a detailed view of SVA’s impact across tasks.

Results and Qualitative Analysis

Key quantitative gains of SVA across benchmarks and a qualitative case study.

SVA builds on frozen VLA backbones by reranking their action proposals with a lightweight Q‑model, turning test‑time evaluation into a performance lever. The following results quantify how this reranking improves success across diverse embodied tasks.

SVA raises EB‑Habitat success by an average of +15.4 points over five backbones.

Table 1 reports the per‑backbone gains and the aggregated average.

**Figure 4.** Success rates on SimplerEnv and RoboTwin. Full results in Tables 7 and 8.

On SIMPLERENV, $\pi$₀+SVA attains a 50.7 % average success rate.

Table 1 lists the per‑task results; the average is computed over all manipulation tasks.

On RoboTwin, $\pi$₀.5+SVA raises average success to 43.5 %, a +7.5‑point gain.

Table 1 aggregates the four RoboTwin tasks.

SVA outperforms the competitive RoboMonkey baseline by +5.2 points on RoboTwin.

Direct comparison in Table 1.

Full SVA achieves 56.11 % success on EB‑Navigation.

Ablation study results (Section 5.2).

SVA’s inference latency stays ≤ 1.33 s even with 32 candidates.

Latency measurements in Section 5.3.

Case Study: the instruction “On the sofa there’s an apple, but instead find a plate and move it to the brown table” trips the base policy into a distractor‑driven plan, while SVA selects a plate‑centric plan and finishes in four steps.

**Figure 7.** **Case 1 – distractor-aware instruction following.** The base policy follows the salient but irrelevant apple-on-sofa clause and fails after repeated invalid recovery actions. SVA selects a plate-centric plan and completes the task in 4 steps.

**Figure 8.** **Case 2 – spatial-relation grounding.** For the instruction *Find a wrench and move it to the right of the sink*, the base policy searches an incorrect location and issues repeated invalid pickup actions, while SVA grounds the spatial relation and completes the rearrangement.

Table 2 illustrates the underlying mechanism: the distractor‑driven candidate has a high prior (0.4375) but a negative Q‑value (‑0.1282), while the goal‑directed candidate has a low prior (0.0625) yet a positive Q‑value (0.2948), leading SVA to select the latter.

Limitations and Failure Modes

We outline SVA’s current constraints and concrete avenues for extending it.

SVA reranks VLA‑generated action candidates using a lightweight Q‑model at test time, improving success without retraining the backbone.

The first limitation is that tree search and value learning are staged separately; the search operates blind to the evaluator being learned, and the policy only gains from test‑time re‑ranking, leaving most of RL’s potential untapped.

A natural next step is to close the loop: let an up‑to‑date Q‑model steer MCTS while on‑policy rollouts continuously refine that Q‑model, thereby harvesting more of the RL signal without heavily perturbing the frozen backbone.

The second limitation is the reliance on a resettable simulator that supplies a binary task‑success signal; such simulators are standard for benchmarks like EmbodiedBench but are unavailable in many real‑world domains.

Future work could replace or supplement the simulator with learned world models or sparse human/auto‑labeled outcomes, making the Search‑Value‑Act pipeline applicable where high‑fidelity simulation is impractical.

The third limitation is that all experiments are confined to simulation; the calibration of the Q‑model on physical robots remains untested.

Validating SVA on real hardware—via sim‑to‑real co‑training or lightweight online residual calibration—constitutes the most urgent next step.

**Figure 5.** Ablation on EB-Navigation (Qwen3.5-9B). Full results in Table 9.

Benchmark Details

Benchmarks used to evaluate SVA and their relevance to real‑robot deployment.

We evaluate SVA on the EmbodiedBench suite and two VLA manipulation benchmarks to test generalization across discrete‑skill and continuous‑control domains.

EmbodiedBench aggregates several simulated embodied‑AI tasks that mimic real‑world robot perception and instruction following.

ManiSkill2‑based simulation of WidowX manipulation tasks, designed as a quantitative proxy for real‑world robot experiments.

SAPIEN‑based bimanual manipulation suite supporting five robotic arms and structured domain randomization for sim‑to‑real transfer.

All experiments run in simulation, yet the benchmarks are constructed to be predictive of real‑robot performance, and the backbones ($\pi$₀ and $\pi$₀.5) are themselves trained on large‑scale real‑robot demonstrations.

SVA remains simulator‑free at inference: it only consumes the frozen VLA’s RGB + proprioception + language inputs and the lightweight Q‑model, avoiding any online tree search or privileged simulator features.

Deploying SVA on a physical robot requires no algorithmic changes; the Q‑model can be used zero‑shot or fine‑tuned on a small set of real‑robot rollouts with the same MCTS‑style targets.

**Table 3.** Benchmark settings used in our experiments.

Implementation Details

Detailed configurations for the Q‑model, data collection, training, and evaluation across all benchmarks.

The Q‑model builds on the pretrained Qwen3.5‑0.8B backbone, extending it with a special <|VALUE|> token that serves as the state‑action representation for each candidate.

``` Dropout(0.1) -> Linear(1024, 512) -> GELU -> Linear(512, 1) ```

Implementation of a single Q‑head.

During search‑stage data collection, MCTS runs offline on the training split, recording instruction, observation, history, candidate actions, MCTS‑backed returns, and visit counts for each tree edge.

**Table 4.** Search-stage data collection settings.

Value‑stage training regresses the Q‑model outputs to the MCTS returns, normalizing each benchmark’s targets by its own mean and standard deviation.

At act time, the base policy samples multiple candidate action sequences (or continuous chunks), the Q‑model scores them in batch, and the highest‑scoring candidate is executed; the process repeats until success or budget exhaustion.

**Table 5.** Act-stage evaluation settings. EmbodiedBench uses sampling tempe

Baseline reranking methods follow the same candidate selection pipeline but replace the Q‑model scorer with either a RoboMonkey‑style verifier or a pairwise preference model trained on MCTS supervision.

Detailed Pass@k Results

Key quantitative outcomes of SVA on manipulation benchmarks.

SVA raises the average success rate on RoboTwin 2.0 from 38.3 % to 43.5 %.

Table 8 reports the average across ten tasks, with the “$\pi$0.5+SVA” column achieving 43.5 % versus 38.3 % for the RoboMonkey baseline.

**Table 6.** Task-level Pass@k results across manipulation benchmarks using popular VLA models.

**Table 8.** Results on RoboTwin 2.0. All values are success rates (%).

Additional Analyses

56.1% average success shows SVA’s full design outperforms all ablations.

Tables 7 and 8 report task‑level success rates for SimplerEnv and RoboTwin 2.0, where SVA consistently beats the $\pi$₀ baselines on temporally extended manipulation tasks.

SVA attains a 56.1% average success on EB‑Navigation, surpassing every ablated variant.

Table 9 shows the full SVA method achieving the highest average (56.11) versus 50.83, 43.33, and 39.17 for the three ablations.

**Table 9.** Detailed ablation results on EB-Navigation.

Additional Qualitative Cases

Qualitative examples show SVA fixes distractor and spatial errors of the base policy.

SVA completes both distractor‑aware and spatial‑relation tasks significantly faster than the base policy.

In Case 1 the base policy fails after 25 steps while SVA succeeds at step 4; in Case 2 the base fails after 20 steps versus SVA success at step 4.

SVA ranks candidate action sequences by their predicted long‑horizon return rather than by proposal likelihood, allowing it to discard distractor‑driven or spatially incorrect plans.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers