ACTIONOPD: Distilling Decisions, Not Tokens

Huining Yuan, Xiangmin Yi, Zelai Xu, Zixu Wang, Xinlei Chen, Xiao-Ping Zhang, Yu Wang, Chao Yu

ActionOPD distills agent models by aligning semantic actions rather than token sequences, bypassing the need for teacher probabilities.

How can we distill language-model agents by aligning their high-level decision-making actions rather than just imitating their token sequences?

Distilling capable language-model agents into smaller models is difficult because standard methods require matching the teacher's exact token sequences, which forces the student to mimic specific reasoning styles and requires access to teacher token probabilities. ActionOPD instead treats semantic actions—such as tool calls or communicative intent—as the primary unit of supervision. It samples actions from a black-box teacher and rewards the student for generating responses that semantically match those decisions, using a policy-gradient objective that ignores the specific token-level path taken to reach them. This approach improves performance across diverse out-of-distribution benchmarks by 3.3–6.9 percentage points, enabling effective distillation without requiring shared model families or tokenizer alignment.

Paper Primer

ActionOPD functions as a semantic-matching filter: it samples multiple candidate responses from the student, compares them against a multiset of teacher-generated actions, and uses the resulting agreement as a reward signal for reinforcement learning.

ActionOPD consistently improves agent performance on unseen tasks.

Training Qwen3-4B and Qwen3-8B students with two different teacher families across curated tool-use tasks. Average gains of 3.3–6.9 percentage points across four out-of-distribution benchmarks.

The method scales effectively with both student size and teacher capability.

Larger students (8B vs 4B) and stronger teachers (DeepSeek-V4-Flash vs Qwen3.7-Flash) consistently yielded higher average performance. Consistent performance improvements across all four tested student-teacher configurations.

Why is this method better than simply training on teacher-generated sequences?

Offline imitation of teacher sequences often suffers from state-distribution mismatch, where the student encounters states the teacher never visited. ActionOPD uses on-policy rollouts to supervise the student at states it actually encounters, reducing this mismatch.

What is the primary advantage of using "semantic actions" over token-level supervision?

It decouples the decision from the realization. By rewarding the semantic outcome (e.g., calling the correct tool) rather than the specific token sequence, the student can learn the teacher's decision-making logic without being forced to replicate the teacher's specific reasoning or writing style.

ActionOPD provides a flexible, black-box-compatible distillation pipeline that prioritizes decision quality over token-level imitation, making it a robust choice for cross-family agent distillation.

Motivation and Problem Framing

Frames why token‑level imitation is brittle and motivates action‑level alignment.

Distilling large language‑model agents into smaller models raises two intertwined questions: which states should the teacher supervise, and at what granularity should student and teacher behavior be aligned? Prior approaches either provide dense token‑level supervision that mismatches the student’s state distribution, or they require token‑probability access and compatible tokenizers, limiting applicability. Recognizing that a tool call or a natural‑language response encapsulates a decision, we argue that aligning semantic actions—rather than exact token sequences—offers a more robust supervision signal.

When a student model is forced to copy the teacher’s token stream, it inherits the teacher’s decisions only superficially; the underlying intent of each action is lost, making the distilled agent brittle.

**Figure 2.** Supervision for agent learning. Offline distillation (KD) imitates teacher tokens at teacher-induced states. Token-level OPD compares token distributions along student-generated interactions. Outcome-based RL uses terminal feedback. ActionOPD instead rewards semantic agreement with teacher actions at student-encountered states. Here, we sample only one response per state for demonstration. For implementation, we expand the sampled teacher multiset and student state group.

Shifting from token‑level imitation to decision‑level alignment via ActionOPD resolves the brittleness of prior distillation.

The ActionOPD Mechanism

ActionOPD replaces brittle token‑level imitation with action‑level alignment.

Token‑level distillation forces the student to copy exact strings, which breaks when many surface forms express the same underlying action. ActionOPD sidesteps this brittleness by rewarding semantic agreement instead of token identity.

At each state the student samples a small set of candidate responses, scores them against a shared teacher multiset, and uses the highest‑reward action to advance while training on all candidates.

Count matches: $c(s_g, A)=2$ (two teacher $A$), $c(s_g, B)=1$, $c(s_g, C)=0$.

Compute rewards with $\lambda=0.5$: $r_\lambda(A)=1+\lambda\frac{2-1}{2}=1.25$, $r_\lambda(B)=1$, $r_\lambda(C)=0$ (invalid actions would be $-1$).

Select the highest‑reward action $A$ (index $i^*$) for execution; all four responses are kept for the PPO update.

The frequency bonus lets the learner prefer actions that appear repeatedly in teacher samples, while still giving a non‑zero signal to a single match.

How does ActionOPD differ from traditional token‑level distillation?

Token‑level distillation aligns the student’s token probabilities with the teacher’s exact token sequence, which penalizes semantically equivalent paraphrases. ActionOPD instead aggregates teacher samples into a multiset, scores student responses by semantic agreement, and rewards any response that realizes the same underlying action, regardless of surface wording.

Instead of matching token strings, the loss aligns the probability mass the student places on each semantic action with the teacher’s action distribution.

Why can the REINFORCE gradient be used with a binary semantic reward?

The REINFORCE identity only requires a scalar reward that is a function of the sampled trajectory. Here the reward $r_T(s,\phi_s(y))$ is binary but still provides a non‑zero signal whenever the student’s response semantically matches the teacher, allowing the gradient $\nabla_\theta\log\pi_\theta(y\mid s)$ to be weighted appropriately.

**Figure 3:** ActionOPD at one state. Student candidates are scored against a shared teacher multiset. One highest-reward action is committed. All candidates in retained groups contribute to training.

**Algorithm 1** ActionOPD **Require:** Student $\pi_{\theta}$, teacher $\pi_{\text{T}}$, matcher $m_s$, tasks, $K = 3, N = 4, \lambda = 0.5$ 1: **for** each training iteration **do** 2: $\quad$ Freeze $\theta_{\text{old}} \leftarrow \theta$; initialize state-group buffer $\mathcal{B} \leftarrow \emptyset$ 3: $\quad$ **for** each sampled task **do** 4: $\quad\quad$ Reset the environment and obtain public state $s$ 5: $\quad\quad$ **while** task is active and interaction budget remains **do** 6: $\quad\quad\quad$ Set group index $g \leftarrow |\mathcal{B}| + 1$ and state $s_g \leftarrow s$ 7: $\quad\quad\quad$ Sample teacher responses $\mathcal{M}_{\text{T}}(s_g) = [y_1^{\text{T}}, \dots, y_K^{\text{T}}]$ using $\pi_{\text{T}}$ 8: $\quad\quad\quadSampley_{g,i} \sim \pi_{\theta_{\text{old}}}(\cdot \mid s_g)$ and retain parsed actions, for $i = 1, \dots, N$ 9: $\quad\quad\quad$ Compute $c(s_g, y_{g,i})$ and $r_{g,i} = r_{\lambda}(s_g, y_{g,i})$ (Equation 5; Equation 6) 10: $\quad\quad\quad$ Add the complete state group $(s_g, \{y_{g,i}, r_{g,i}\}_{i=1}^N)$ to $\mathcal{B}$ 11: $\quad\quad\quad$ Select $i^* \in \arg \max_i r_{g,i}$; execute the parsed action in $y_{g,i^*}$ and update $s$ 12: $\quad\quad$ **end while** 13: $\quad$ **end for** 14: $\quad$ Exclude groups without supervision or with equal rewards 15: $\quad$ Center all $N$ rewards per retained group and whiten across the batch (Equation 7) 16: $\quad$ Broadcast advantages to generated tokens and update $\theta$ with the clipped objective (Equation 9) 17: **end for**

Related Distillation Methods

We situate ActionOPD among prior distillation and RL approaches, highlighting their objectives and limitations.

Prior work on language‑model distillation spans token‑level imitation, outcome‑based reinforcement, and various hybrid schemes that combine supervision with on‑policy feedback.

Students are trained to reproduce the exact token probabilities of a teacher at each position, treating every token as an independent supervision target.

Instead of matching tokens, the student is rewarded for actions that lead to high‑reward outcomes, using a reinforcement‑learning objective.

Transfers a teacher’s predictive distribution to a student via a softened cross‑entropy loss, without considering sequential generation dynamics.

Imitates the teacher’s generated token sequences, aligning student outputs with teacher rollouts on the same inputs.

Guided Knowledge Distillation supervises student‑generated sequences with token‑level divergences, requiring teacher logits for each token.

Distills a large language model into a smaller one by aligning token‑level outputs on sampled teacher sequences.

Transfers probabilistic supervision across different tokenizers by explicitly aligning teacher and student token spaces.

Minimizes token‑level divergence on student rollouts using a shared‑weight teacher conditioned on reference solutions.

Distills feedback‑conditioned predictions, supervising the student with teacher outputs that incorporate external feedback signals.

Studies shared‑parameter training where the teacher receives privileged information, enabling on‑policy reverse KL supervision.

Jointly trains a discriminator to reward teacher‑like responses, turning the distillation problem into a reward‑learning task.

Evaluates responses against rubrics derived from teacher‑student contrasts, focusing on structured evaluation criteria.

Uses verbal teacher scores to evaluate reasoning quality at both step and trajectory levels during on‑policy interaction.

Compares selected student reasoning chunks with teacher continuations sampled at the same student prefixes.

Rewards trajectory‑wide tool‑set agreement, final correctness, and execution validity, integrating multiple supervision signals.

Derives a semantic‑agreement policy gradient and uses a world model to compare UI transitions induced by generated and reference actions.

Queries expert demonstrations at learner‑visited states, aggregating them to mitigate covariate shift.

Provides local teacher corrections before supervised and short‑horizon RL refinement, blending imitation with reinforcement.

Gradually withdraws teacher turns through a rollout curriculum, easing the student into autonomous behavior.

Combines OPD and RL with annealed coefficients and turn‑level weighting, balancing imitation and reinforcement signals.

Retrospectively groups repeated states across trajectories and derives relative advantages from returns, improving sample efficiency.

Empirical Evaluation

Key performance gains of ActionOPD across benchmarks, scaling, and ablations.

ActionOPD aligns student action distributions with teacher‑preferred outcomes, avoiding brittle token‑by‑token imitation.

ActionOPD raises the four‑benchmark average by up to $6.9$ pp over base students.

Table 1 shows improvements ranging from $3.3$ pp to $6.9$ pp across BFCL, ToolSandbox, AppWorld, and Tau.

**Figure 1.** Student–teacher scaling of ActionOPD. Larger students attain higher final averages. DeepSeek-V4-Flash yields higher student averages than Qwen3.7-Flash.

Across both teachers, the larger Qwen3‑8B student consistently outperforms Qwen3‑4B, yielding an average gain of about $2.7$ pp under identical budgets.

ActionOPD attains $0.4$ pp higher pass1 than KD while using on‑policy supervision.

Table 3 shows ActionOPD’s pass1 exceeds KD’s by $0.4$ pp on the held‑out test set.

Ablations confirm that removing the semantic reward or frequency bonus drops average pass1 by up to $0.4$ pp, validating the combined design.

Table 3 compares ActionOPD against the three variants and shows the largest performance drop for the Validity Reward variant.

Self‑ActionOPD improves test average pass1 by $4.8$ pp over the Base student.

Table 5 reports a $4.8$ pp gain in pass1 (and $2.5$ pp in pass4) when the privileged teacher guides the student.

Ablation Studies

Ablation analysis quantifies how each ActionOPD component affects gradient estimation and reward shaping.

This section isolates the contribution of each ActionOPD ingredient by removing it and observing the resulting change in the gradient or reward signal. The derivations reveal which components are essential for stable learning.

Omitting the action‑independent baseline $b(s)$ does not alter the gradient.

Equation 4 shows $\mathbb{E}_y[\nabla_\theta \log \pi_\theta(y\mid s)]=0$, so adding or removing $b(s)$ leaves the expectation unchanged.

Subtracting the sample‑group mean scales the expected gradient by $(N\!-\!1)/N$.

Text following Equation 4 states “subtracting that mean scales the expected gradient by $(N-1)/N$”.

The frequency‑bonus weight $\alpha = \lambda/(K\!-\!1)$ controls how much the match count $c(y)$ influences the reward.

Equation 6 defines $r_\lambda(s,y)=\alpha\,c(y)+(1-\alpha)\mathbf{1}\{c(y)>0\}-\mathbf{1}\{\text{invalid}\}$ with $\alpha=\lambda/(K-1)$.

Dual clipping introduces a lower bound $c_{\text{dual}}=3$ for negative advantages, preventing extreme policy updates.

Definition of $h_{\text{dual}}$ in Section A.4 uses $c_{\text{dual}}=3$ for $A<0$.

Implementation and Training Details

Details of data, training setups, evaluation protocols, and ablation configurations.

Training data are drawn from two pools, AWM and EnvScaler. AWM context selection retains tasks whose native prompt (including tool schemas) fits within $16{,}000$ tokens under the Qwen3‑4B tokenizer; this cutoff differs from the training context limit. A deterministic audit then checks scenario, database, and verifier integrity before static feasibility screening evaluates whether the environment can satisfy the requested goal.

**Table 6.** Task-selection funnel for the pools used in this study. Task counts denote unique tasks remaining after each stage, not rollout attempts. The final row reports environment coverage. Context selection is an AWM-specific stage. The dash denotes that no separate corresponding stage was used for EnvScaler.

Training configurations follow the settings summarized in Tables 7 and 8. All experiments use AdamW optimizer with a constant learning rate of $10^{-6}$ and no warm‑up. PPO updates run for one epoch per collection step with minibatch size 32; clip values are $0.2$ and dual‑clip $3$. Advantage normalization applies group centering and batch whitening. Loss aggregation uses an active‑token mean, and no KL or entropy bonuses are applied.

**Table 7.** Standard ActionOPD and ablation training configurations. Each configuration corresponds to one training run. The number of samples refers to responses at one state, not independent complete trajectories.

**Table 8.** Model roles and generation settings.

Evaluation follows four protocols. BFCL and ToolSandbox use the same task subsets and per‑turn limits (20 steps for BFCL, 30 messages for ToolSandbox). AppWorld evaluates 168 Normal and 417 Challenge tasks with a $100{,}000$‑token budget per task; TGC measures the fraction of tasks passing all goal tests, while SGC measures completion of all scenario variants. Tau evaluation aggregates success over four trials per task, computing per‑task success rates $p_{1j}=c_j/n$ and $p_{4j}=(c_j/4)/(n/4)$, then averaging across domains.

Baseline and ablation variants include KD (knowledge distillation), OPD (outcome‑based policy distillation), standard ActionOPD, and Self‑ActionOPD. KD filters teacher trajectories for success before SFT; OPD samples student trajectories and applies a reverse‑KL loss using teacher log‑probabilities; ActionOPD samples four candidates per state and executes one, while Self‑ActionOPD fixes teacher parameters during a collection step and updates only the shared student‑teacher parameters thereafter.

Collection budgets differ across methods. KD and OPD each collect four trajectories per task start. ActionOPD samples four candidates per state, but only the executed candidate contributes to the loss; however, all candidates in retained groups affect training. Self‑ActionOPD also samples four candidates, but teacher samples remain fixed during optimization, and teacher‑only tokens are excluded from the student loss.

Self‑ActionOPD uses a current‑weight teacher (Qwen3‑4B) and a fixed update target. Teacher inputs include privileged task summaries (private goals, constraints, and tool arguments) that are not exposed to the student during inference. These privileged inputs guide the teacher’s action selection but do not replace the public confirmation required for evaluation.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers