Thinking Cross-Tokenizer on-Policy Distillation: From Alignment Coverage to Supervision Reliability

Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang

Cross-tokenizer distillation succeeds by focusing on strict alignment rather than maximizing coverage.

Why does adding "span supervision" (matching teacher distributions over multi-token spans) often hurt performance in cross-tokenizer On-Policy Distillation, despite the intuition that it should provide more reliable training signals?

Cross-tokenizer On-Policy Distillation (OPD) struggles to align teacher and student models because they use different vocabularies and token boundaries, making it unclear which parts of a response are safe to compare. The authors show that strict 1:1 token alignment already covers most generated tokens and captures nearly all predictive probability mass, allowing for effective distillation using only a small, student-selected subset of the shared vocabulary. This compact approach outperforms methods that attempt to force complete supervision coverage, which the authors find introduces weakly aligned, conflicting gradients that degrade downstream accuracy.

Paper Primer

The paper introduces a "strict" distillation objective that restricts training to 1:1 token alignments where both models agree on the span. By selecting only the top-k shared vocabulary entries at these positions, the method achieves high performance without the noise of broader, less reliable alignment strategies.

Compact supervision on strict positions retains nearly all distillation benefits.

Training on a student-selected top-16 subset of the shared vocabulary preserves at least 96% of the accuracy gains achieved by full shared-vocabulary distillation. Outperforms four cross-tokenizer baselines by 0.51–1.05 percentage points in full-average accuracy.

Expanding coverage to mismatch groups via span supervision is counterproductive.

Adding mean squared error (MSE) loss on span log-probabilities to achieve complete coverage consistently lowered accuracy across all 18 tested weight settings. Accuracy dropped by 0.27–1.20 percentage points compared to the strict-only baseline.

Why does adding more supervision coverage hurt performance?

Gradient diagnostics reveal that span-based supervision signals have weak or negative directional agreement with strict signals and grow in relative magnitude during training, effectively introducing conflicting training noise.

What is the core assumption about vocabulary overlap?

The authors assume that static vocabulary overlap (which can be low) is a poor proxy for learning value; instead, they find that predictive probability mass is highly concentrated on a small set of shared tokens, making broad coverage unnecessary.

Researchers should prioritize supervision reliability over alignment coverage; compact, high-confidence signals at strictly aligned positions are more effective than exhaustive structural matching.

Introduction to Cross-Tokenizer Distillation

We expose the tension between strict token alignment and span‑based supervision in cross‑tokenizer OPD.

On‑Policy Distillation (OPD) trains a student model on its own generations while receiving feedback from a teacher. When the teacher and student use different tokenizers, their predictions must be aligned both at the sequence level and across vocabularies, a problem we call cross‑tokenizer alignment.

OPD lets a student learn from the teacher by feeding the teacher’s probability distribution back into the student on the exact prefixes the student currently generates.

Across three heterogeneous teacher–student pairs on math reasoning and code generation, strict 1:1 token groups already cover the majority of student‑generated tokens, even though static vocabulary overlap (Jaccard similarity) can be as low as 39 %.

**Figure 1.** Three diagnostics of Cross-Tokenizer OPD for Qwen2.5-7B-Instruct→Llama-3.2-3B-Instruct. **Left:** The logged student-side alignment ratio remains high during Strict full training despite substantially lower static vocabulary overlap. **Middle:** Adding supervision on mismatch groups lowers mathematics and code performance. **Right:** At strictly aligned positions in student-generated responses before distillation, the shared vocabulary retains nearly all teacher and student probability mass on average, and a student-selected top-16 subset retains most of it on both sides.

When we augment the strict loss with mean‑squared‑error supervision on span log‑probabilities, every positive weight yields full structural coverage but consistently lowers downstream accuracy. Gradient diagnostics show that span gradients have weak or negative directional agreement with strict gradients and grow in magnitude relative to them, suggesting interference.

The core tension is that maximizing alignment coverage via span supervision can introduce unreliable gradients, whereas compact supervision at strictly aligned positions often yields better downstream performance.

Notation and Strict Alignment

Defines notation and the strict‑alignment lens for cross‑tokenizer OPD.

We first fix the symbols that will appear throughout the paper, then recall the standard OPD loss before extending it to the cross‑tokenizer setting.

When the student and teacher use different tokenizers, we must align their predictions not only across time steps but also across vocabularies.

Strict alignment isolates those positions where a single student token and a single teacher token cover exactly the same text span – a “1‑to‑1” match that lets us compare distributions without any token‑group mismatch.

How does Strict Alignment differ from the general token‑group alignment used elsewhere in the paper?

General token‑group alignment simply pairs intervals of text, regardless of how many tokens each side contributes. Strict Alignment restricts attention to intervals where *both* sides contribute exactly one token, enabling a direct KL comparison without any aggregation or redistribution of probabilities.

Interval $r=1$ (the whole sentence) yields $S_{\theta}^{1}=\{1,2,3\}$ and $S_{T}^{1}=\{1\}$.

Since $|S_{\theta}^{1}|=3\neq1$, this interval belongs to $A_{\text{mis}}(y)$ – a mismatch group.

If both tokenizers had produced a single token for the whole sentence, we would have $S_{\theta}^{1}=\{1\}$, $S_{T}^{1}=\{1\}$, placing $r=1$ in $A_{1:1}(y)$ as a strict 1:1 group.

Strict 1:1 groups exist only when the two tokenizers agree on token boundaries; any disagreement pushes the interval into the mismatch set, which the strict‑alignment loss deliberately skips.

The Failure of Span Supervision

We quantify how supervision coverage impacts downstream accuracy across three teacher‑student pairs.

We evaluate how different supervision signals affect the quality of distilled student models for three teacher–student pairs.

Span supervision adds a loss over mismatched token spans, encouraging the student to match the teacher’s probability distribution across whole text segments rather than only at strict token‑to‑token positions.

How does span supervision differ from the standard strict‑alignment loss?

Strict alignment penalises only exact 1:1 token pairs, while span supervision also penalises the entire probability path for mismatched groups, matching the teacher’s log‑probability over the whole span.

Strict token coverage stays above 85 % even when static vocabulary Jaccard overlap is as low as 40 %.

Table 1 reports Jaccard overlap 39.49–64.87 % versus strict student‑token coverage 85.57–96.98 % across the three pairs.

Adding span supervision ($\lambda$ > 0) consistently reduces full‑average accuracy.

Across $\lambda$ = 0.25–1.5 the full‑average drops 0.27–1.20 percentage points relative to the $\lambda$ = 0 baseline for all three pairs.

Adding span supervision is like installing extra safety checks on a car that end up adding weight and slowing it down — the extra coverage sounds beneficial but the performance suffers.

Shared vocabulary entries capture virtually all predictive probability at strict positions.

Across the three pairs the shared vocabulary retains 99.69–99.90 % of teacher mass and 98.99–99.81 % of student mass on average.

**Figure 2.** Full-average accuracy (%) across mismatch weights. Dashed lines mark strict supervision ($\lambda = 0$), and annotations give the decrease at $\lambda = 1.5$ in percentage points.

Span supervision consistently degrades performance despite higher coverage.

Learning with Student-Selected Subsets

Evaluates how much distillation benefit remains when training on student‑selected top‑k vocab subsets.

We evaluate whether limiting training to the student‑selected top‑k vocabulary preserves the gains of full‑vocab distillation. Experiments vary $k$ and compare against strict full supervision and several baselines.

Instead of using the entire shared vocabulary, the student model trains only on its own top‑$k$ predicted tokens, forcing alignment on a compact, high‑confidence subset.

How does training on a student‑selected top‑$k$ subset differ from using the full shared vocabulary?

With the full vocabulary the KL loss compares distributions over all tokens, many of which have negligible probability. The top‑$k$ variant discards low‑probability tokens, renormalizes the remaining mass, and thus focuses the loss on the most likely predictions, yielding similar alignment with far fewer tokens.

**Figure 3.** Probability mass retained by student-selected top-k subsets and the full shared vocabulary at strict positions before distillation.

**Table 1.** Performance comparison of different vocabulary alignment methods across three model pairs (Qwen→Llama, Granite→Phi, Granite→Qwen) evaluated on Math, Code, and Full datasets.

Gradient Interference Analysis

OPD aligns student and teacher; span supervision can introduce gradient interference that harms accuracy.

Recall that OPD forces the student to match the teacher’s token‑level predictions, while span supervision adds an auxiliary loss that can clash with the strict alignment objective.

When two loss terms push model parameters in different directions, their gradients can cancel or dilute each other, reducing the effectiveness of both objectives.

How does Gradient Interference differ from a generic “gradient conflict” that appears in multitask learning?

Gradient Interference here is specific to the strict‑vs‑span supervision pair in OPD: the strict loss aligns token‑level distributions, while the span loss aligns longer‑range token groups. Their interference is measured on the same sampled responses, whereas generic multitask conflict often involves unrelated tasks and separate data.

Compute gradient cosine and norm ratio for a given checkpoint.

**Figure 4.** Mismatch–strict gradient cosine $c_{\text{mis}}$ (solid) and split-half reference $c_{\text{ctrl}}$ (dashed) at checkpoints from training with only the strict loss.

**Figure 5.** Mismatch-to-strict gradient norm ratio $\rho$, at checkpoints from training with only the strict loss, with growth factors from step 0 to 100 computed from unrounded checkpoint medians.

**Table 6.** Gradient diagnostics at checkpoints from training with only the strict loss. $c_{mis}$ and $c_{ctrl}$ denote the mismatch-strict and split-half strict gradient cosines, respectively. $\rho$ is the unweighted span-to-strict norm ratio. Values are rounded to three decimals; figure growth factors use the unrounded medians.

Compute cosine: $\langle g_{1:1}, g_{\text{mis}}\rangle = (2)(-1) + (-1)(2) = -4$.

Norms: $\|g_{1:1}\|_{2} = \sqrt{2^{2}+(-1)^{2}} = \sqrt{5}$, $\|g_{\text{mis}}\|_{2} = \sqrt{(-1)^{2}+2^{2}} = \sqrt{5}$.

Cosine similarity $c_{\text{mis}} = -4 / ( \sqrt{5}\,\sqrt{5}) = -0.8$ (strong negative alignment).

Norm ratio $\rho = \sqrt{5} / \sqrt{5} = 1.0$ (equal magnitude).

Even with equal magnitudes, a negative cosine means the two updates cancel each other, illustrating gradient interference.

Downstream Results

Downstream benchmarks reveal how supervision and decoding choices affect distilled model accuracy.

We now present downstream benchmark performance of the distilled models under different supervision and decoding settings. The results expose systematic patterns across tasks and model pairs.

Table 7 shows that frozen teachers consistently outperform their undistilled student counterparts on every benchmark, with the largest gaps observed on the MATH500 and AIME series. This pattern confirms that strict token‑level alignment alone yields a strong baseline.

When evaluating a larger teacher (Qwen3‑235B‑A22B‑Instruct‑2507→Granite‑4.1‑8B), Table 14 again shows the teacher dominating the student across all benchmarks, especially on the AIME and HMMT math suites where gaps exceed 20 points.

On the ALFWorld interactive benchmark, Table 16 reports that the Strict top‑16 variant attains the highest success rate (SR) while also requiring the fewest interaction turns among all distilled models, confirming its efficiency.

Related Work and Conclusion

We review related distillation work, summarize findings, and discuss ethical considerations.

On‑Policy Distillation (OPD) trains a student on its own rollouts with teacher supervision; MiniLLM uses reverse KL to penalize unlikely student regions, while GKD supports multiple divergences and mixes student‑generated and fixed data.

Yang et al. cast OPD as dense KL‑constrained RL with per‑token rewards from teacher‑to‑reference log‑probability ratios, and subsequent work investigates which teacher signals are effective at student‑visited prefixes.

Fu et al. analyze unreliable guidance and propose distribution matching on teacher‑selected top‑k supports; Entropy‑aware OPD augments reverse KL with forward KL at high‑entropy positions, while TIP selects trajectories using student entropy and teacher–student disagreement.

Li et al. relate distillation success to teacher–student compatibility, showing that the intersection of their top‑k predictions contains most probability mass and can retain the benefits of student top‑k supervision.

Cross‑Tokenizer Distillation faces token‑boundary and vocabulary mismatches; methods such as ULD compare sorted output probabilities, DSKD and CDM build representation or vocabulary mappings, and DWA‑KD adds entropy‑based weighting.

Other approaches match likelihoods across tokenizations, aggregate tokens into span representations, or convert teacher predictions to byte probabilities; GOLD4 merges text‑equivalent token groups for on‑policy training, and SimCT adds minimal aligned multi‑token units.

Byte‑Prefix Marginalization maps teacher probability mass onto student tokens via byte‑prefix relations and masks harmful supervision at whitespace positions to prevent training collapse.

Our experiments show that strict 1:1 groups already cover most student‑generated tokens despite substantial static vocabulary mismatch, and the shared vocabulary retains nearly all teacher and student probability mass at strict positions.

A compact student‑selected top‑k subset of the shared vocabulary retains most of this mass on both sides, and restricting reverse KL to a top‑16 subset at each strict position preserves at least 96 % of the full‑average improvement of OPD.

Adding span log‑probability MSE achieves complete supervision coverage but consistently reduces downstream accuracy; span gradients grow in magnitude while their directional agreement with strict gradients remains weak or negative, explaining the observed accuracy drop.

We used generative AI tools only for language polishing; they were not involved in experimental design, implementation, execution, or analysis.

Human annotations were employed solely for early methodological research, with informed consent, privacy protection, and fair compensation.

All datasets are sourced from official releases and have been anonymized, though we cannot guarantee the absence of socially harmful or toxic language.

Experimental Setup

Details of data, training, baselines, and evaluation protocols for the OPD experiments.

We build a source pool of 20 k prompts, split evenly between mathematics (sampled from DAPO‑Math‑17K) and code (sampled from CodeForces5). Each student tokenizes its prompts and we keep only those whose tokenized length does not exceed 2,048 tokens, so the retained count varies with the tokenizer.

The probability‑mass probe evaluates the full, unfiltered 20 k prompts as described in Appendix B. All prompts are formatted using each model’s official chat template before being fed to the rollout engine.

For each sampled prompt the student generates a single response. Loss is computed by re‑tokenizing the decoded response together with the prompt on both student and frozen teacher sides, then masking the next‑token positions that belong to the response. An EOS token is appended when the response is not length‑truncated.

Training proceeds for 100 on‑policy iterations, with checkpoints saved every 10 iterations. Optimization runs with FSDP2 on a single node equipped with eight NVIDIA H20 GPUs, multiplexing components via sleep mode.

We reduce memory pressure by using packed samples, bfloat16 arithmetic, gradient checkpointing, and token‑chunked loss evaluation with a chunk size of 2,048 tokens.

SGLang serves as the rollout engine, and the global optimizer batch aggregates 512 responses across student workers.

This table lists the hyperparameters and settings used for training, optimization, distillation, and rollout.

All four baselines adopt the KDFlow implementation; they use unit KD weight, temperature 1.0, and omit cross‑entropy mixing. Variants ULD, Extended ULD, and GOLD further exclude the terminal EOS from their distillation losses.

Evaluation uses publicly available benchmarks. For mathematics we report mean@32 accuracy over seven datasets; for code we report mean@8 accuracy over three datasets. The overall “Full” score averages the Math and Code means equally.

Math scoring extracts the final boxed answer; missing or malformed answers count as incorrect, and answers longer than 300 characters are truncated before verification with Math‑Verify⁶.

Code scoring runs HumanEval and MBPP through EvalPlus, preserving original test suites, while LiveCodeBench uses its v6 runner with a 10‑second per‑candidate timeout and a 120‑second total evaluation cap.

**Table 4.** Static vocabulary overlap across the three teacher–student pairs. Vocabulary counts follow the shared-vocabulary convention in Section 2. Overlap ratios are reported as percentages.

Diagnostic Protocols

We detail the metrics and probes used to evaluate tokenizers and gradient behavior.

To assess how well student and teacher tokenizers align, we first compute a static vocabulary overlap and then probe retained probability mass on strict positions. Both metrics expose mismatches that can affect downstream performance.

For the probability‑mass probe, each pre‑distillation student generates one response per prompt from the full 20 k‑prompt set, using the rollout and length settings of Table 3. We discard empty responses, byte‑mismatched teacher–student pairs, responses lacking scorable strict positions, and any sample whose encoded input exceeds a model’s context limit or the analysis sequence‑length cap.

The table displays gradient diagnostics for three model pairs (Qwen→Llama, Granite→Phi, and Granite→Qwen) across four training steps (0, 20, 60, 100). For each pair, it reports values for $c_{mis}$, $c_{ctrl}$, and $\rho$.

Gradient diagnostics are reported in Table 6, tracking three statistics at checkpoints 0, 20, 60, and 100: $c_{\text{mis}}$ (mismatch‑strict cosine), $c_{\text{ctrl}}$ (split‑half strict cosine), and $\rho$ (span‑to‑strict norm ratio). Over training, $c_{\text{ctrl}}$ declines while $\rho$ generally rises, reflecting increasing alignment between span‑based and strict losses.

Benchmark Results

Benchmark tables and span‑MSE weight sweeps for the appendix.

This appendix collects the full result tables referenced in the main text. The tables compare three baseline distillation pipelines (ULD, Extended ULD, GOLD, SimCT) against three variants of the Strict alignment strategy, and they also explore the effect of the span‑MSE weighting $\lambda$ on downstream performance.

The table presents performance metrics across various benchmarks (MATH500, GSM8K, AIME, AMC23, Minerva-Math, HumanEval, MBPP, LiveCodeBench) comparing different methods: ULD, Extended ULD, GOLD, SimCT, and three variants of Strict (full, top-16, top-128). Aggregated results for Math, Code, and Full categories are provided at the bottom.

**Table 10.** Benchmark-level comparison for Granite-4.1-8B→Qwen2.5-7B-Base.

**Table.** Performance across various benchmarks with different Span-MSE weight $\lambda$ values.

**Table 12.** Benchmark-level span-MSE weight sweep for Granite-4.1-8B→Phi-4-mini-instruct.

Extended Benchmark Results

This appendix presents detailed benchmark tables for span‑MSE weighting, teacher‑student scores, distillation methods, and ALFWorld evaluation.

We first show how varying the span‑MSE weight $\lambda$ influences performance across a suite of math and code benchmarks.

**Table 13.** Benchmark-level span-MSE weight sweep for Granite-4.1-8B→Qwen2.5-7B-Base

Table 14 compares the undistilled student model against its frozen teacher, revealing that the teacher substantially outperforms the student on both math (80.81 → 91.51) and code (95.88 → 94.24) aggregates.

**Table 15.** Benchmark-level distillation results for Qwen3-235B-A22B-Instruct-2507→Granite-4.1-8B. Bold marks the best Math, Code, and Full averages.

Table 16 reports ALFWorld evaluation where the “Strict top‑16” distilled model attains the highest success rate (SR = 46.94) and the lowest turn count (Turns = 41.15), confirming its advantage in interactive environments.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers