Visual Contrastive Self-Distillation
Yijun Liang, Yunjie Tian, Yijiang Li, Yuqi Jia, Furong Huang, Tianyi Zhou, Di Fu
VCSD uses visual-conditioning contrast to generate self-distillation targets without external teachers or labels.
How can we perform on-policy self-distillation in vision-language models without an external teacher or privileged answer hints?
Vision-language models often struggle to prioritize fine-grained image evidence over generic language priors, yet existing self-distillation methods require external teachers or manual annotations to force this focus. Visual Contrastive Self-Distillation (VCSD) constructs a supervision signal by comparing the model's predictions on the original image against a content-erased control, isolating the specific influence of visual content. This contrastive signal sharpens the teacher's distribution toward visually-supported tokens, consistently outperforming base models and standard on-policy distillation across multiple benchmarks and scales.
Paper Primer
VCSD is a self-distillation method that derives its learning target from the model's own internal behavior: it evaluates the teacher on the same prefix twice—once with the original image and once with a blank control—to measure how much the visual input shifts the probability of each next token. The core move is to use this log-probability difference as a "visual-evidence reward" to sharpen the teacher's original-image distribution, creating a refined target that is then distilled into the student.
VCSD consistently improves performance across diverse vision-language benchmarks without requiring external supervision.
Aggregate scores on seven benchmarks for Qwen3-VL models. Improvements of +4.77% at 2B, +1.86% at 4B, and +3.75% at 8B parameters.
The method is robust across different model families and scales.
Consistent gains of 2.9% to 4.3% over base models for the Qwen3.5 family. Consistent performance boost across all tested scales from 2B to 9B.
Why is a content-erased control necessary for this distillation process?
Standard self-distillation often fails to provide a stronger learning signal than the student's current prediction. The content-erased control acts as a reference point, allowing the model to isolate which tokens are specifically driven by the image rather than by language priors.
Does this method introduce additional computational overhead during inference?
No. The contrastive evaluation is performed only during training to generate the distillation target; the final student model retains its original architecture and requires no extra inference-time branches or passes.
VCSD demonstrates that model-internal predictive differences can replace external teachers, enabling effective visual grounding through purely self-contained, input-conditioned contrast.
Motivation and Problem Framing
We expose why on‑policy self‑distillation needs asymmetric targets and introduce VCSD to create that asymmetry from raw inputs.
On‑policy self‑distillation (OPSD) removes the need for an external teacher but still requires an asymmetric teacher‑student relationship; without it the self‑teacher offers little more than the student’s current prediction.
OPSD lets a model teach itself by using a slowly‑updated EMA copy as a teacher, but the teacher must see something the student does not, otherwise the target distribution adds no new learning signal.
Target asymmetry means the teacher’s target distribution is deliberately different—typically sharper—than the student’s, ensuring the student receives a stronger learning signal.
**Figure 1.** Comparison of different sources of target asymmetry. Existing methods sharpen the teacher distribution using privileged answers or visual evidence. Our method instead contrasts teacher predictions under the original image and a content-erased control, producing a visually informed target using only the image and question, without auxiliary supervision.
VCSD creates asymmetry by contrasting the teacher’s token probabilities on the original image with those on a content‑erased version, using the difference to sharpen the original distribution.
How does VCSD differ from prior methods that use privileged answers or visual evidence?
Prior methods give the teacher extra inputs (answers, cropped regions) that the student never sees, whereas VCSD keeps both teacher and student conditioned on the same prompt and only varies the image content, using the erased version as a reference to derive asymmetry without any external supervision.
Asymmetry is essential for self‑distillation; VCSD achieves it solely through input conditioning, eliminating the need for privileged answers or visual evidence.
The VCSD Mechanism
Build an asymmetric teacher target by contrasting visual conditions during on‑policy self‑distillation.
The student’s on‑policy responses provide the prefixes at which supervision is applied, but without an asymmetric teacher the target collapses to the student’s own distribution. VCSD injects asymmetry by contrasting the teacher’s predictions under the original image versus a content‑erased control.
**Figure 2.** **Overview of VCSD.** At each student-generated prefix, the EMA teacher contrasts next-token distributions under the original image and a content-erased control. The contrast sharpens the plausible original-image distribution to form the target, which is distilled into the student via forward KL. The response prefix is omitted for clarity.
VCSD builds a teacher target by asking “how does the presence of the actual image change token probabilities compared to a blank image?” and then sharpening the original distribution toward tokens that gain support from the visual content.
Compute contrasts: $\Delta_t(a)=\log0.6-\log0.4\approx0.41$, $\Delta_t(b)=\log0.3-\log0.4\approx-0.29$, $\Delta_t(c)=\log0.1-\log0.2\approx-0.69$.
Set $\beta=0.5$ of the max (max is 0.6), so $S_t(\beta)=\{a\}$ because only $a$ exceeds $0.5\times0.6=0.3$.
Apply weighting with $\alpha=1$: numerator for $a$ is $0.6\cdot e^{0.41}\approx0.6\cdot1.51=0.906$; denominator is the same value, so $q^{\ast}_t(a)=1$.
All other tokens receive zero probability because they are outside $S_t(\beta)$.
The example shows that VCSD can concentrate all probability on a single token when the contrast strongly favors it and the plausibility filter removes the rest.
How does VCSD differ from conventional contrastive learning that pulls together embeddings of matching pairs?
Conventional contrastive learning operates on vector embeddings and requires explicit positive/negative pairs. VCSD works at the token‑distribution level: it contrasts two *probability* distributions produced by the same model under different visual conditions, without any external pairing or embedding loss.
The control image is simply a black canvas of the same size as the original, preserving every preprocessing step while removing all instance‑specific visual information.
Pass $J$ through the visual encoder → yields token embeddings $e_1,\dots,e_4$.
Pass $J_{\text{ctrl}}$ through the same encoder → yields four identical zero embeddings $z_1,\dots,z_4$.
Because the embeddings differ only in content, any change in the teacher’s token probabilities can be traced back to the visual information removed in $J_{\text{ctrl}}$.
The control isolates the effect of visual content while keeping every other processing step identical, making the contrast a clean measure of visual contribution.
Why not use a blurred version of the image instead of a black one as the control?
A blurred image still retains low‑frequency structure that can influence the teacher’s predictions, contaminating the baseline. A black image removes all content, guaranteeing that any difference in logits is due solely to the presence of visual information.
Student $\pi_\theta$ samples a prefix $y_{<t}$ on the original image $J$.
EMA teacher $\pi_\phi$ evaluates $y_{<t}$ on $J$, producing $p^{\phi}_{J,t}$.
Teacher evaluates the same prefix on the content‑erased control $J_{\text{ctrl}}$, producing $p^{\phi}_{0,t}$.
Compute token‑wise log‑probability contrast $\Delta_t(v)=\log p^{\phi}_{J,t}(v)-\log p^{\phi}_{0,t}(v)$.
Form the plausibility set $S_t(\beta)=\{v\mid p^{\phi}_{J,t}(v)\ge\beta\max_u p^{\phi}_{J,t}(u)\}$.
Construct the contrast‑shaped target $q^{\ast}_t$ by reweighting $p^{\phi}_{J,t}$ with $\exp(\alpha\Delta_t)$ and renormalizing over $S_t(\beta)$.
Compute forward KL $\mathrm{KL}(q^{\ast}_t\|p_{\theta,t})$ and back‑propagate only through the student.
Update the EMA teacher parameters: $\phi \leftarrow \mu\phi + (1-\mu)\theta$.
Empirical Performance and Dynamics
VCSD consistently raises accuracy across benchmarks and model scales.
The experiments evaluate VCSD against the base model and OPSD across three Qwen families and three scales each. The results demonstrate that VCSD consistently outperforms the alternatives.
VCSD raises average accuracy by +1.86 % over the base model across all model scales.
Table 1 reports average‑accuracy gains ranging from +1.86 % to +4.77 % for VCSD versus the base model.
**Figure 4.** Training dynamics of VCSD and OPSD. Seven-benchmark aggregate accuracy (left) and per-benchmark accuracy on MMStar (middle) and MathVista (right) over the same training run.
**Figure 5.** Qualitative comparison on MathVista (Qwen3.5-2B). The decisive visual quantity is the number of thousand-cubes. Red marks the miscounts made by the base and OPSD models; green marks the correct count used by VCSD.
Ablations and Theoretical Analysis
We examine how each VCSD component influences performance and prove the contrast‑shaped target formulation.
The central premise is that VCSD creates the asymmetry needed for on‑policy self‑distillation by contrasting the teacher’s output on the original image with a content‑erased control.
Distillation Divergence measures how far a student distribution deviates from a teacher distribution that has been reshaped by visual evidence.
How does Distillation Divergence differ from a plain KL divergence?
Plain KL only penalises deviation from the teacher’s raw output. Distillation Divergence adds a visual‑evidence term $\alpha$ rᵛⁱˢₜ, which reshapes the target distribution toward tokens that the teacher assigns high contrast scores, thus creating the asymmetry required for on‑policy distillation.
Removing the forward‑KL objective (using reverse KL) drops average accuracy by 0.18 percentage points, confirming that a mode‑covering loss is essential. Replacing the content‑erased control with plain Gaussian noise reduces accuracy by 0.90 points, showing that the contrast between original and erased images drives the improvement. Omitting the VCSD anchor lowers performance across most benchmarks, underscoring the anchor’s role in stabilising the contrast‑shaped target.
Questions & answers
What is the main contribution of Visual Contrastive Self-Distillation (VCSD)?
VCSD introduces a self-distillation method that derives its learning target entirely from the model's own internal behavior by comparing predictions on the original image against a content-erased (black) control, isolating the influence of visual content without any external teacher or manual annotations.
What problem does VCSD address?
VCSD addresses the tendency of vision-language models to prioritize generic language priors over fine-grained image evidence, a problem that existing self-distillation methods fail to solve without external teachers or manual annotations.
Why is standard on-policy self-distillation (OPSD) insufficient on its own?
Standard on-policy self-distillation requires an asymmetric teacher-student relationship to provide a useful learning signal; without it, the self-teacher offers little more than the student's current prediction, causing the target distribution to collapse to the student's own output.
How does VCSD create the asymmetry needed for effective self-distillation?
VCSD achieves asymmetry solely through input conditioning: it evaluates the teacher twice on the same prefix—once with the original image and once with a content-erased (black) control—and uses the log-probability difference as a visual-evidence reward to sharpen the teacher's distribution.
Why is a content-erased (black) image used as the control rather than a blurred image?
A blurred image retains low-frequency structure that can still influence the teacher's predictions, contaminating the baseline; a black image removes all visual content, ensuring that any difference in logits is attributable solely to the presence of visual information.
How does VCSD differ from conventional contrastive learning methods?
Conventional contrastive learning operates on vector embeddings and requires explicit positive/negative pairs, whereas VCSD works at the token-distribution level by contrasting two probability distributions produced by the same model under different visual conditions, without any external pairing or embedding loss.
How does VCSD differ from prior self-distillation methods that use privileged inputs?
Prior methods give the teacher extra inputs such as answers or cropped regions that the student never sees, whereas VCSD keeps both teacher and student conditioned on the same prompt and only varies the image content, using the erased version as a reference without any external supervision.
What is Distillation Divergence and how does it differ from plain KL divergence?
Distillation Divergence extends plain KL divergence by adding a visual-evidence term α·r_t^vis, which reshapes the target distribution toward tokens that the teacher assigns high contrast scores, creating the asymmetry required for effective on-policy distillation; plain KL only penalizes deviation from the teacher's raw output.
Does VCSD add computational overhead at inference time?
No. The contrastive evaluation against the content-erased control is performed only during training to generate the distillation target; the final student model retains its original architecture and requires no extra inference-time branches or passes.
What models and scales were used to evaluate VCSD?
The experiments evaluate VCSD against the base model and OPSD across three Qwen model families at three scales each; the paper does not specify the exact scale sizes beyond this description.
What are the key empirical results of VCSD?
VCSD consistently outperforms both the base model and standard on-policy self-distillation (OPSD) across multiple benchmarks and all evaluated model families and scales; the paper does not report a single aggregate accuracy number but demonstrates consistent improvement.
What do the ablation studies reveal about VCSD's design choices?
Removing the forward-KL objective (using reverse KL instead) drops average accuracy by 0.18 percentage points; replacing the content-erased control with plain Gaussian noise reduces accuracy by 0.90 points; and omitting the VCSD anchor lowers performance across most benchmarks, confirming that each component contributes to the method's effectiveness.
What is the role of the student's on-policy responses in VCSD?
The student's on-policy responses provide the prefixes at which supervision is applied; VCSD then injects asymmetry at those prefixes by contrasting the teacher's predictions under the original image versus the content-erased control.
What are the limitations of VCSD as described in the paper?
The paper does not explicitly enumerate limitations; it acknowledges that standard self-distillation collapses without asymmetry but does not discuss failure cases, out-of-distribution generalization, or computational cost of the dual forward pass during training.
What venue, authors, and date are associated with this paper?
The paper is available on arXiv at arxiv.org/abs/2607.21556; the paper does not specify author names or a publication venue in the provided text.
Key terms
- VCSD (Visual Contrastive Self-Distillation)
- A self-distillation training method for vision-language models that constructs a supervision signal by comparing the model's token-level predictions on the original image against predictions on a content-erased control image.
- on-policy self-distillation (OPSD)
- A distillation approach in which the model acts as its own teacher using responses sampled from its own current distribution, without any external teacher model.
- content-erased control
- A blank (black) image used as a baseline input to the teacher model so that any difference in predictions relative to the original image can be attributed purely to visual content.
- visual-evidence reward (r_t^vis)
- A per-token score computed as the log-probability difference between the teacher's predictions on the original image and on the content-erased control, quantifying how much each token is driven by visual input.
- Distillation Divergence
- A training objective that extends KL divergence by incorporating the visual-evidence reward to reshape the teacher's target distribution toward tokens that are specifically supported by the image.
- language prior
- The tendency of a language model to predict tokens based on statistical patterns in text alone, independent of any visual input.
- teacher-student asymmetry
- A condition in self-distillation where the teacher has access to information or a perspective that the student does not, enabling the teacher to provide a learning signal stronger than the student's current predictions.
- forward KL divergence
- A mode-covering divergence measure that penalizes the student for assigning low probability to any token the teacher assigns high probability, used as the base loss in VCSD.
- vision-language model (VLM)
- A neural network model that jointly processes visual inputs (images) and textual inputs to perform tasks such as visual question answering or image captioning.
- token-distribution contrast
- The operation in VCSD of comparing two probability distributions over the next token produced by the same model under different visual conditions, rather than comparing vector embeddings.
- Qwen
- A family of language and vision-language models used in the paper's experiments to evaluate VCSD across multiple architectures and scales.