Self-Supervised Visual On-Policy Distillation

Yijiang Li, Yijun Liang, Yunjie Tian, Bingyang Wang, Ke Zhang, Zhenfei Yin, Di Fu, Philip Torr, Nuno Vasconcelos

S2VOPD creates informative teacher-student asymmetry by degrading the student's visual input, enabling self-supervised distillation without privileged data.

How can we perform on-policy distillation for vision-language models without needing privileged ground-truth data or a separate, stronger teacher model?

On-policy distillation typically requires a stronger teacher or privileged information like ground-truth regions to provide a useful learning signal. When such external guidance is unavailable, models lack the necessary asymmetry to improve their own performance. Self-Supervised Visual On-Policy Distillation (S2VOPD) inverts this requirement: it keeps the teacher on the original image while forcing the student to learn from a degraded, augmented view. This input discrepancy creates a predictive gap that serves as a free, perception-aligned training signal. This approach improves a 4B model's average accuracy on perception benchmarks from 70.7% to 77.4%, outperforming much larger models and methods that rely on expensive privileged annotations.

Paper Primer

S2VOPD functions as a self-distillation loop where the teacher model observes the clean image and the student model observes a transformed version. The core move is to treat the predictive discrepancy between these two views as the training objective: the student is forced to match the teacher's distribution, effectively learning to recover information lost to the augmentation.

S2VOPD achieves state-of-the-art performance for its size without requiring privileged supervision.

Across six fine-grained perception benchmarks, S2VOPD improves Qwen3.5-4B accuracy by 6.7%, surpassing models up to 235B parameters and matching the 397B scale.

The effectiveness of the distillation signal depends on the semantic consistency of the augmentation.

Information-reduction augmentations (downscaling, noise) consistently outperform geometric ones like cropping, which can remove the evidence required to answer the question entirely. Moderate downscaling (0.3–0.6x) peaks at 75.65% accuracy, while aggressive cropping degrades performance below the unaugmented baseline.

Why does this method use augmentation to create a gap instead of just using a larger teacher model?

Conventional distillation requires a stronger teacher or privileged information, which is costly or unavailable. S2VOPD creates the necessary asymmetry "for free" by withholding information from the student, allowing the model to generate its own perception-aligned supervision.

Does this method trade off reasoning capabilities for perception gains?

No. Unlike privileged-supervision methods that often weaken math reasoning, S2VOPD improves both fine-grained perception and mathematical reasoning, achieving near-best results in both domains.

Motivation and Problem Framing

We expose the core problem: reliance on privileged data for teacher–student asymmetry.

Visual on‑policy distillation hinges on an informative teacher–student asymmetry, typically supplied by a larger teacher or privileged supervision. When no privileged data is available, the question becomes: how can we obtain a useful asymmetry? The paper answers by inverting the source of asymmetry, withholding information from the student rather than adding it to the teacher.

Effective distillation requires the teacher to see something the student cannot, but existing approaches depend on external annotations or stronger models, which are costly or unavailable.

The reliance on privileged data in existing distillation pipelines is the primary bottleneck this work eliminates.

The S$^2$VOPD Mechanism

We describe S²VOPD, the self‑supervised on‑policy distillation mechanism that creates a teacher‑student view asymmetry.

Standard on‑policy distillation assumes a stronger, privileged teacher, which is unavailable in many vision‑language settings. Without such a teacher the student receives no informative signal about the correct policy. S²VOPD solves this by deliberately corrupting the student’s visual input while keeping the teacher’s view pristine, thereby generating a useful supervision gap for free.

The student policy processes a stochastically augmented image $\tilde{x}=T(x)$, while an EMA teacher processes the original clean image $x$, so the teacher’s predictions are more stable and can guide the student’s corrupted rollouts.

Compute the EMA update: $\phi_1 = 0.9\,\phi_0 + 0.1\,\theta_0$ – the teacher moves only 10 % toward the current student.

Student policy $\pi_{\theta}$ samples a token sequence $y=(\text{"cat"},\text{"sits"})$ conditioned on $\tilde{x}$ and question $q$.

Teacher evaluates the same prefix on clean $x$, yielding a distribution $p^{\tau}_2$ that places 0.70 probability on “sits”.

Student’s distribution $p^{s}_2$ (on $\tilde{x}$) assigns 0.45 to “sits”.

The divergence $D^{0.5}_{\text{JS}}(p^{\tau}_2\parallel p^{s}_2)$ is computed on the top‑$k=3$ tokens, producing a loss of roughly $0.12$.

The noise‑induced gap (0.70 vs 0.45) provides a learning signal; the EMA teacher’s slow drift ensures this signal remains stable across updates.

How does S²VOPD differ from conventional knowledge distillation where a larger teacher is pre‑trained?

In standard distillation the teacher is a separate, stronger model trained on external data, so the student benefits from higher capacity and privileged supervision. S²VOPD instead creates the asymmetry internally: the teacher shares the same parameters as the student (via EMA) and only differs in the visual input it receives. The supervision therefore comes from the input corruption, not from a more powerful network.

Training loop for S²VOPD.

Creating Teacher-Student Asymmetry

Build a teacher‑student gap by augmenting only the student view.

Without a teacher‑student gap the student simply copies the teacher’s predictions, defeating the purpose of on‑policy distillation. The trick is to corrupt only the student’s observation, forcing it to rely on the teacher’s cleaner signal.

Apply a stochastic visual transformation $T$ to the student’s image while leaving the teacher’s view untouched, thereby creating a controllable asymmetry that the student must bridge.

Sample the global flag $z\sim\text{Bernoulli}(p)$ with $p=1$, so $z=1$ (augmentation is active).

Sample the downscale strength $s=0.45$, yielding a $29\times29$ image after downsampling (no resizing back to $64\times64$).

Sample the noise flag $b_{\text{noise}}\sim\text{Bernoulli}(0.5)$; assume $b_{\text{noise}}=1$, so noise is added.

Apply $A_{\text{down}}$: the image now has $29\times29$ pixels, reducing the number of visual tokens fed to the student.

Apply $A_{\text{noise}}$: each pixel receives i.i.d. Gaussian noise with $\sigma\approx0.11$, slightly perturbing the low‑resolution image.

The final student view $T(x)$ is a $29\times29$ noisy image, while the teacher still sees the original $64\times64$ clean image.

This combination shrinks the student’s token count and injects stochastic corruption, forcing the student to rely on the teacher’s high‑fidelity predictions to recover the original semantics.

How does this differ from ordinary data augmentation used to improve generalization?

Standard augmentation is applied symmetrically to all training inputs, so both teacher and student would see the same distorted view. Here the augmentation is deliberately asymmetric: only the student’s observation is altered while the teacher receives the pristine image, creating a purposeful prediction gap that the student must learn to close.

**Table 1.** Overview of augmentation operators on the student policy.

Performance and Benchmarks

S^2VOPD delivers a 77.44% average accuracy, surpassing larger models without privileged data.

S^2VOPD raises average accuracy on the FineVision benchmark to 77.44%, a +6.76% gain over the base Qwen3.5‑4B model.

Table 2 shows the base model at 70.68% and S^2VOPD at 77.44% average.

**Table 2.** Performance of S$^2$VOPD trained on a 12k subset of FineVision [Wiedmann et al., 2026], compared with open-source, proprietary, and methods that use privileged information on fine-grained perception benchmarks (accuracy, %; AVG is the average over the 6 benchmarks).

**Table 3.** Comparison with prior methods trained on the Vision-OPD training data, across six fine-grained perception and three mathematical-reasoning benchmarks at different model scales. **Bold** and <u>underline</u> mark the best and the second-best entry in each column within each base model; § indicates that best instead of last checkpoint performance is reported.

Benchmark Performance Details

S$^2$VOPD leads the benchmark, improving overall average by up to 2.0%.

S$^2$VOPD improves the overall average by +2.0% over the strongest self‑rewarding baseline at the 4B scale.

Table 3 reports the best performance across 13 checkpoints; S$^2$VOPD’s average is 2.0 % higher than the top baseline.

At the 9B scale, S$^2$VOPD gains +0.7% over the strongest baseline.

Table 3 shows a 0.7 % lift in the overall average for the larger model.

S$^2$VOPD also outperforms every supervised baseline—including ZwZ, Vision‑OPD, and OPSD—at the 4B scale, and it ties Vision‑OPD for overall rank at 9B while requiring no privileged annotations.

Methods that ingest privileged visual information boost perception scores but degrade math reasoning: OPSD cuts MathVision accuracy by 9.3 % at 4B and 7.9 % at 9B, while ZwZ lowers MathVerse accuracy by 27.1 % at 4B.

Self‑rewarding RL approaches (TTRL, Intuitor, RENT) tend to improve math reasoning, likely because their longer, deliberative responses encourage deeper computation, yet they achieve only modest perception gains compared with Vision‑OPD.

Analyzing Augmentation Utility

Which augmentations create a useful teacher‑student predictive gap?

Recall that S$^2$VOPD creates a teacher‑student predictive gap by augmenting only the student, letting it learn from the teacher’s clean predictions. This asymmetry is the core signal for effective distillation.

**Figure 2.** (a) Performance of each augmentation family applied alone; (b) Accuracy vs. the teacher-student predictive gap induced by the augmentation. We measure the teacher-student gap as the token-level JS Divergence between the teacher and the student distributions, averaged over the first ten steps of training.

Information‑reduction augmentations raise accuracy to 75.65 %.

75.65 % vs. 70.58 % baseline (≈ +5.07 %).

Photometric transformations raise accuracy to 74.40 %.

74.40 % vs. 70.58 % baseline (≈ +3.82 %).

Geometric transformations raise accuracy to 74.30 %.

74.30 % vs. 70.58 % baseline (≈ +3.72 %).

Occlusion augmentations raise accuracy to 72.44 %.

72.44 % vs. 70.58 % baseline (≈ +1.86 %).

Ablations and Sensitivity Analysis

Ablations reveal that both augmentation and EMA are essential for S²VOPD’s gains.

Crop‑based augmentations strip away task‑relevant evidence, and their strength directly harms downstream scores.

Strong cropping reduces average accuracy to 67.44 %, a 4.09 % drop from the mild setting.

Performance falls from 71.53 % (mild) → 68.76 % (moderate) → 67.44 % (strong) across the three crop strengths.

We next isolate the two core components of S²VOPD—student‑view augmentation and the EMA teacher—via the ablations in Table 4.

Removing the student‑view augmentation collapses the gain to 70.52 %, only 0.16 % above the base model.

Table 4 shows “w/o Aug.” achieving 70.52 % versus the base model’s 70.68 % on the six perception benchmarks.

Freezing the teacher (no EMA updates) costs only 0.40 % of performance, reaching 75.95 %.

Table 4 reports “w/o EMA (fixed teacher)” at 75.95 % average, just 0.40 % below the full S²VOPD result.

Varying the EMA decay from 0.95 to 0.999 changes average accuracy by at most 0.8 %.

Table 5 lists averages 72.87 % (decay 0.95), 71.29 % (0.99), and 76.35 % (0.999); the span is 0.8 % after rounding to one decimal.

Using Jensen‑Shannon Divergence improves average accuracy by 2.8 % over forward KL and by 5.0 % over reverse KL.

Table 6 reports averages 71.92 % (forward KL), 74.74 % (JSD), and 69.75 % (reverse KL).

Both the EMA teacher and the student‑view augmentation are required for stable, high‑performing distillation.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers