DAPD: Dual-Anchored Policy Distillation

Jianyu Wu, Yizhou Wang, Encheng Su, Chen Tang, Shixiang Tang

DAPD aligns reference and rollout behavior under matched information conditions to eliminate privilege illusion in self-distillation.

How can we prevent language models from relying on "privileged" information (like ground-truth answers) during training, which causes them to fail when that information is absent at inference time?

On-policy self-distillation (OPSD) trains models using privileged information, but this creates an information asymmetry: the teacher sees the reference while the student must predict without it, causing the student to hallucinate unsupported claims. DAPD resolves this by introducing "anchors" that force the teacher and student to operate under matched information availability, using both reference and rollout completions as guidance sources. This approach significantly reduces privilege illusion, outperforming standard OPSD by +2.00 points on average and maintaining stable gains across model scales up to 32B.

Paper Primer

The core issue is "Entangled Distillation," where the student learns to mimic a teacher that relies on privileged information it cannot access at inference. DAPD fixes this by constructing a self-conditioned "bridge" distribution that matches the information available to both the teacher and the student, ensuring the student only learns reproducible reasoning.

DAPD uses two mechanisms: Dual-Path Anchoring (DPA) aligns behavior across unconditioned and privileged paths to ensure consistency, while Dual-Source Anchoring (DSA) balances reliable reference guidance with student-reachable rollout guidance. Think of it like a tutor who provides both a textbook answer (reference) and a peer's scratchpad (rollout) to ensure the student learns the logic, not just the final result.

DAPD consistently outperforms OPSD across model scales.

Performance gains on Qwen3 models persist as scale increases, unlike OPSD where gains diminish.

DAPD significantly reduces inference-time hallucinations.

Qualitative analysis of AIME24 reasoning tasks shows a reduction in late-stage wrong claims.

Why does this approach work better than simply filtering the teacher's signal?

Previous methods modify the teacher's output but leave the underlying information asymmetry intact. DAPD changes the supervision structure itself, ensuring the student and teacher are always evaluated under matched information conditions.

Does this method require curated reference solutions for every task?

No; while references provide reliable guidance, the authors demonstrate a "dual-rollout" variant where the model uses two independent self-generated completions to provide complementary guidance without needing a curated reference.

By matching information availability during distillation, DAPD allows models to scale their reasoning capabilities without falling into the trap of privilege illusion, making it a robust framework for post-training.

Introduction: The Privilege Illusion

We expose how on‑policy distillation creates a privilege illusion that harms inference performance.

On‑policy (self) distillation (OPSD) has become a popular post‑training technique for language models because it provides dense token‑level supervision from a teacher that sees privileged information such as a reference completion.

This privileged teacher‑student mismatch creates a “privilege illusion”: the student learns to depend on the reference it cannot access at inference, leading to unsupported claims and degraded task performance.

OPSD trains a student by distilling a teacher that is conditioned on privileged data (e.g., a reference completion) at each token, while the student itself never sees that data.

The core problem is that students learn to rely on the teacher's privileged reference rather than internalizing the reasoning process.

Analyzing the Privilege Illusion

Identifying how privileged references cause students to over‑rely on unavailable information.

On‑policy distillation (OPSD) trains a student by letting a teacher see the ground‑truth reference $y^{*}$ while the student must predict without it. This privileged supervision creates a “privilege illusion”: the student behaves as if the unavailable reference were present, leading to a sharp performance collapse when the reference is removed at inference.

The student internalizes patterns that rely on the reference $y^{*}$, so at test time it “hallucinates” the missing information and makes confident but unsupported predictions.

To quantify the illusion, the authors track “wrong claims” (unsupported assertions) and the Avg@12 reasoning score across training steps. Across five Qwen‑3 scales, wrong claims climb from 13.0 to 37.0 per 10 000 generations while Avg@12 falls from 59.56 to 53.24, evidencing that the illusion intensifies as training proceeds.

The root cause is the information asymmetry between the teacher’s Cross distribution and the student’s None distribution. The teacher can exploit $y^{*}$, whereas the student cannot, mixing useful reference guidance with signals that are impossible to reproduce at inference – a phenomenon the authors label Entangled Distillation.

Introducing a Self distribution $p_{\text{Self},\theta,t}=p_{\theta}(\cdot\mid x, y_{<t}, y)$ removes the asymmetry: Self receives the same full completion $y$ that the student will eventually generate, matching the information available to Cross while remaining trainable. This bridging anchor aligns the teacher and student information spaces, mitigating privilege illusion.

The DAPD Framework

Dual‑Anchored Policy Distillation aligns reference and rollout guidance through bidirectional anchors.

The privilege illusion in OPSD leaves the student over‑reliant on privileged references, causing collapse when those references disappear at inference.

DAPD introduces two complementary anchoring mechanisms so that the student can learn both from privileged references and from its own on‑policy rollouts, eliminating the information gap.

Compute the weighted reference contribution: 0.6 × 0.5 = 0.30.

Compute the weighted rollout contribution: 0.4 × 0.8 = 0.32.

Sum the two contributions: 0.30 + 0.32 = 0.62.

The resulting DAPD loss of 0.62 guides the student to balance the two signals.

Even with a modest reference weight, the rollout term can dominate if its loss is larger, preventing the student from collapsing to the privileged reference alone.

DPA composes three directed objectives into two complementary paths—one that never sees privileged data (unconditioned) and one that does (privileged)—so that both sides of the distillation are aligned under identical information conditions.

Entangled Distillation loss (ent) = 0.2 (difference between teacher and student on None).

Inference Anchor loss (infer) = 0.1 (student aligning to None on the opposite side).

Privileged Anchor loss (priv) = 0.05 (teacher‑student alignment when privileged info is present).

Combine: 0.2 + 0.1 + 0.05 = 0.35 as the DPA loss for this direction.

The privileged term is small but crucial; without it the student would drift away from the high‑quality reference.

DSA runs DPA in both directions—reference‑to‑rollout and rollout‑to‑reference—and mixes their losses, letting the model benefit from both reliable reference signals and reachable rollout signals.

**Figure 3.** Overview of DAPD. *Left:* Dual-Source Anchoring uses both reference-to-rollout and rollout-to-reference guidance to balance correctness and student reachability. *Right:* Dual-Path Anchoring details the rollout-to-reference guidance direction by introducing *Self* as a trainable bridge. The unconditioned path pairs Entangled Distillation with the Inference Anchor to align the two *None* distributions through *Self*, while the Privileged Anchor aligns *Self* and *Cross* when privileged information is available. The opposite source direction applies the same construction with $y$ and $y^*$ exchanged.

**Table 1.** Performance comparison of our proposed DAPD with baselines on Qwen3-4B across reasoning, coding, and instruct benchmarks. Avg. is the mean over all six benchmarks. $\Delta$ reports DAPD’s improvement over OPSD. Bold and underlined values mark the best and second-best results, respectively.

By jointly training with DAPD, the student learns to reason without relying on privileged cues, achieving robust performance when privileged information is unavailable at inference.

Empirical Results

Key performance numbers demonstrate DAPD’s advantage over OPSD.

DAPD achieves a six‑task average score of 57.34, outperforming OPSD by +2.00 points.

Table 1 reports DAPD leading across all six benchmark tasks.

OpenThoughts is a curated dataset of reasoning, coding, and instruction examples that excludes any evaluation instances, ensuring the model never sees test data during training.

How does OpenThoughts differ from typical instruction‑tuning data?

OpenThoughts removes the benchmark evaluation examples and splits the data by task family, so the student never encounters the exact test questions during training, unlike standard instruction datasets that often mix training and test distributions.

**Figure 1.** DAPD consistently improves over OPSD across (a) six tasks on Qwen3-4B and (b) five Qwen3 model scales evaluated by Avg@12 on AIME24, AIME25, and HMMT25.

**Figure 2.** Training dynamics for OPSD, Privileged Anchor, and DAPD, averaged across five scales: (a) wrong claims and (b) Avg@12 over AIME24, AIME25, and HMMT25.

DAPD attains the best out‑of‑distribution average of 49.64, gaining +4.82 points on LCB v5 and +1.37 points on average over OPSD.

Table 2 reports the OOD results across coding and instruction benchmarks.

DAPD consistently outperforms OPSD across all model scales, particularly in reasoning tasks.

Related Work

We survey prior post‑training and on‑policy distillation approaches for reasoning models.

Reasoning language models are often post‑trained using curated reference completions, self‑generated rollouts, or reward feedback, each offering distinct benefits and drawbacks.

References provide reliable, correctness‑oriented guidance but can be off‑policy, whereas rollouts reflect the model’s current behavior yet may contain errors.

Dual‑Anchored Policy Distillation (DAPD) exploits this reference–rollout complementarity as a design principle for dense token‑level post‑training.

On‑policy (self) distillation, also called OPD, supplies dense teacher supervision on student‑sampled rollouts, mitigating the rollout mismatch inherent in off‑policy distillation.

Subsequent work refines OPD by altering teacher contexts, objectives, or token‑weighting schemes.

On‑Policy Distillation (OPSD) removes the need for a separate teacher by conditioning the same model on privileged information, and related self‑distillation methods employ reference solutions, environmental feedback, or successful rollouts as auxiliary guidance.

Recent extensions further calibrate, filter, route, relax, or localize this supervision, while analyses examine scale effects, rollout quality, and optimization dynamics.

DAPD retains on‑policy sampling and privileged‑teacher construction but reorganizes supervision around matched information availability and both completion sources.

Theoretical Analysis

Analyzes how the unconditioned path aligns rollout and reference distributions.

Recall that OPSD creates a privilege illusion: the student leans on privileged references, which collapses when those references disappear. This section analyzes the unconditioned path that bridges rollout and reference distributions.

B.1 Conditional Alignment of the None Distributions establishes a bound on how the rollout‑side and reference‑side None distributions can be brought together.

Proposition 1 shows that the expected TV distance between the two None distributions is bounded by the square‑root of the inference KL term, the bridge constant $\varepsilon$, and the square‑root of the entanglement KL term.

B.2 Sequence‑Level Transfer from Entangled Distillation lifts the token‑level alignment to a bound on full‑sequence distributions.

B.3 Shared‑Parameter Transfer from Inference Anchor explains how a gradient step on the inference loss propagates to the Cross logits via shared parameters.

Empirical compatibility check: across four LoRA initializations and 32 math examples, the mean gradient dot product is 256.62 (95 % CI [147.55, 393.53]) and the mean cosine similarity is 0.101 (95 % CI [0.058, 0.144]), confirming the positivity required by the compatibility condition.

Implication for DAPD: Proposition 1 bounds the None‑to‑None alignment via the inference and entanglement terms; Proposition 2 shows that Entangled Distillation transfers this alignment to full sequences; and Eq. (21) demonstrates that a compatible Inference Anchor step moves the Cross distribution toward the reference‑side anchor, jointly justifying the effectiveness of the DAPD framework.

Extended Results

Key scale‑wise gains and reduced privilege‑illusion artifacts for DAPD.

This section reports the absolute scale‑wise scores, the privilege‑illusion measurements, and implementation sensitivity.

DAPD consistently outperforms the scale‑matched OPSD policy, yielding an average gain of 2.37 points.

Table 6 shows per‑benchmark improvements of 1.94, 2.69, 2.41, 2.04, and 2.78 points from 1.7 B to 32 B.

**Table 4.** User-message templates for the *None* view without privileged information and the reference-conditioned *Cross* view. Braced fields are replaced per example.

Qualitative Analysis and Limitations

Qualitative inference results and forward‑looking research avenues.

Recall that on‑policy distillation supplies a privileged reference during training, which creates a privilege illusion: the student leans on the reference and fails when it is absent at inference.

This figure displays three stacked boxes comparing different methods for solving a geometry problem. The top box presents the problem statement and the correct answer (127). The middle box, labeled "OPSD: unsupported answer recalled," shows an incorrect reasoning process that leads to the answer 43. The bottom box, labeled "DAPD: answer derived from the prompt," shows a correct step-by-step derivation leading to the answer 127.

Table 9 extends the qualitative evidence by reporting inference‑time correctness over $12$ generated samples for each method; neither OPSD nor DAPD receives a reference at test time, and both use the same prompt and decoding.

Across the five model scales, Table 6 shows that DAPD attains the highest Avg@12 score in every case except the $14\,B$ scale, where it ties with OPSD, confirming the qualitative trend seen in Figure 4.

Table 7 reveals that the privileged anchor replaces the trainable “None” distribution with a self‑reference while keeping the cross‑teacher, and DAPD merges both anchoring paths, explaining its improved performance.

Table 8 reports that removing forward KL or reverse KL components degrades Avg@12, indicating that both divergence terms contribute to the DAPD objective.

The authors outline three promising research avenues to further reduce reliance on privileged references.

First, a reference‑free DAPD could replace the fixed reference with a consensus rollout that is updated as the policy improves, preserving matched information while eliminating curated solutions.

Second, adaptive source trust would estimate the reliability of rollout versus reference online and weight them continuously, avoiding a brittle global coefficient.

Third, extending the framework to other privileged sources such as retrieved documents, tool traces, or verifier feedback would test how anchor design must adapt to different information modalities.

Experimental Setup and Configurations

Details of datasets, training protocols, privileged information construction, and baseline configurations.

We evaluate on three math‑reasoning contests (AIME 2024, AIME 2025, HMMT Feb 2025) by sampling twelve solutions per problem with temperature 1.0, top‑p 0.95, and up to 38,912 new tokens; Avg@12 reports the average correctness over the twelve samples, and the final score is the unweighted mean of the three contest scores.

Broader evaluation uses LiveCodeBench v5 (Pass@1), BFCL v3 (multi‑turn function‑call accuracy), and IFBench (verifier‑based score), each with its official prompts and parsing rules; every benchmark follows the capability‑specific data protocol described later.

All task‑specific models are trained on OpenThoughts data, split into three protocols: Reasoning (math domain), Coding (code domain), and Instruct (mixed math, code, science); each baseline shares the same data split and prompt format, and benchmark examples are excluded from training.

Privileged information follows the OPSD reference‑injection format: for each problem x we insert the reference completion y* between explicit delimiters in the teacher’s message, while the student rollout prompt contains only x; Table 4 illustrates the resulting user messages.

**Table 1.** (a) **Path components.** Both the unconditioned and privileged paths are necessary for each source. (b) **Guidance sources.** Reference and rollout guidance are complementary, and using both is best.

During training we generate two auxiliary completions for each source completion s ∈ {y, y*}: the Cross message inserts the opposite completion \bar{s} into the privileged prompt, and the Self message repeats s itself; both are evaluated on the prefixes and target tokens of s, but never appear in the inference prompt.

A concrete example asks to order p = 23009, q = 32006, r = 51003; the reference solution rewrites them as 81003, 91003, 51003 and concludes r < p < q, which is then inserted into the Cross and Self messages as explicit worked completions.

The user‑message template first presents the problem, then optionally includes a reference solution delimited by “=== Reference Solution Begin ===” and “=== Reference Solution End ===”, and finally asks the model to reason step‑by‑step without copying the reference.

DAPD training proceeds per minibatch with five steps: (1) sample an on‑policy rollout y from the LoRA‑on student; (2) for each source completion s ∈ {y, y*} construct Self, Cross, and direct token distributions; (3) evaluate Entangled Distillation, Inference Anchor, and Privileged Anchor losses in both directions while detaching teacher distributions; (4) combine the three loss terms using coefficients from Equation (9) and back‑propagate; (5) periodically copy the current LoRA parameters into the shared snapshot used by the two Entangled‑Distillation teachers.

The LoRA‑on student generates all rollouts and trainable distributions; the Entangled‑Distillation teachers share a snapshot of this LoRA‑on model, while the Inference‑Anchor and Privileged‑Anchor teachers use a detached LoRA‑off base policy.

Baseline comparisons keep the same student backbone and on‑policy data budget but vary the supervision construction: Base uses the original Qwen3 policy; OPSD distills the Cross distribution into the None prompt; SDFT adds an EMA self‑teacher conditioned on expert demonstrations; SDPO conditions an EMA self‑teacher on successful rollouts; Purified OPSD removes privileged supervision predictable from the reference via a pointwise‑mutual‑information correction; DOPD dynamically routes token‑level supervision between privileged and student teachers based on advantage gaps.

Optimization adapts all attention and MLP projections with LoRA rank 64 and scale 128, uses a learning rate of 5 × 10⁻⁶ with a linear 500‑step schedule (no warmup), gradient‑norm clipping at 0.1, bfloat16 precision, gradient checkpointing, and an effective batch size of 32.

All training runs fix random seed 42 before model and LoRA initialization, data shuffling, and rollout sampling; matched‑seed protocols ensure identical initialization, data order, and rollout streams across methods.

Training runs on eight NVIDIA A100‑80GB GPUs using PyTorch 2.8.0, Transformers 4.57.1, DeepSpeed 0.18.2, and vLLM 0.11.0; data parallelism adjusts per‑device micro‑batch size while preserving the effective batch size.

During rollout generation the student samples one rollout per problem with temperature 1.1, top‑p 0.95, top‑k 20, up to 1,024 new tokens, and a training context limited to 20,000 tokens.

Divergence is implemented as a component‑clipped forward‑KL surrogate: ℓ_c(q, p) = ∑_{v∈V} min{ $q_v$ (log $q_v$ − log $p_v$), c } with component cap c = 0.05, evaluating teacher and student logits divided by temperature T = 1.1 at every non‑padding token.

Main DAPD configurations apply all three directed losses in both guidance directions; coefficients are parameterized by $\kappa$ (total loss weight), $\lambda$ (fraction to reference‑guided direction), and $\beta$ values that redistribute weight among Entangled, Inference, and Privileged anchors. The Privileged Anchor ablation fixes $\beta_{r}$ef^infer = 1, while the Inference Anchor ablation fixes $\beta_{r}$ef^priv and $\beta_{r}$oll^priv = $\beta_{r}$oll $\beta_{r}$ef^infer.

**Table c.** Anchor weights. Larger scales prefer stronger Privileged Anchor weights.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers