Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidence
Xi Xiao, Tianchen Zhao, Youngeun Kim, Zhuowei Li, Linghan Xu, Jiaye Wu, Zheng Zhang, Xiang Xu, Xuanbai Chen, Farhan Tejani, Jakub Zablocki, Julia Xu, Yifan Xing
ReaLVR bridges the latent evidence-credit gap by using answer-contrastive supervision to ground latent reasoning in visual evidence.
How can we ensure that latent visual reasoning tokens actually ground their computation in relevant visual evidence rather than just predicting the correct answer?
Multimodal Large Language Models (MLLMs) use continuous latent tokens for intermediate reasoning, but these tokens often fail to preserve the specific visual evidence required for a correct answer. This "latent evidence-credit gap" occurs because standard reinforcement learning only optimizes the final output, leaving the internal reasoning trajectory ungrounded. ReaLVR introduces a contrastive supervision mechanism that forces latent tokens to align with relevant visual features while ignoring mismatched ones. It uses the model's own wrong-answer predictions to identify which latent positions are most critical, then applies visual supervision specifically to those positions during training. This approach consistently outperforms existing latent reasoning baselines across five benchmarks, achieving a 63.7% average accuracy on Qwen2.5-VL-7B and demonstrating robust scaling up to 235B parameters.
Paper Primer
The core mechanism of ReaLVR is an on-policy distillation process that treats latent generation as a differentiable trajectory. It uses two contrastive signals: answer-contrastive readout to weight the importance of specific latent positions, and visual-contrastive loss to define the content those positions must preserve.
Think of the latent span as a workspace: ReaLVR acts like a supervisor who watches the model's internal scratchpad, identifies which "notes" (latent tokens) the model actually uses to reach the correct answer versus a wrong one, and then forces those specific notes to match the relevant visual evidence in the image.
ReaLVR improves latent visual reasoning performance across diverse model scales.
Five-task average accuracy on Qwen2.5-VL-7B reached 63.7%, outperforming all evaluated latent-reasoning baselines. Consistent gains observed from 7B up to 235B parameter models.
Why does standard reinforcement learning fail to ground latent reasoning?
Standard methods like Group Relative Policy Optimization (GRPO) only provide feedback on the final text output, which provides no direct signal to the model about which internal latent tokens are responsible for the answer or which visual features they should have prioritized.
Does ReaLVR require changing the model's architecture or inference procedure?
No. ReaLVR is a training-time intervention; the supervision branches are used only during training, allowing the model to maintain its original architecture and inference procedure at test time.
ReaLVR demonstrates that continuous latent reasoning can be effectively supervised at frontier scales, shifting the focus from final-answer accuracy to the internal dynamics of how models process visual evidence.
Introduction: The Latent Grounding Problem
We expose why latent visual reasoning often ignores visual evidence and introduce ReaLVR to fix it.
Latent visual reasoning models generate internal latent tokens that guide answers, yet these tokens are rarely tied to the visual evidence that determines the correct answer, creating a latent evidence‑credit gap.
LVR lets a multimodal model perform intermediate computation in a continuous latent space, using hidden tokens to process visual information without explicit textual steps.
**Figure 1.** **ReaLVR connects visual evidence to latent reasoning.** (a) ReaLVR identifies the stroller missed by LVR-7B in a complex scene; dashed boxes mark image details. (b) Visual and answer contrast determine *what to preserve* and *where to supervise*, during training only. (c) Replacing the top-8 tokens ranked by answer-to-token attention, with other states fixed, decreases correct-answer probability by 4 and 11 percentage points. The larger ReaLVR drop indicates greater local answer dependence.
The disconnect between correct answers and visual evidence indicates that latent tokens must be explicitly anchored to visual cues.
Preliminaries: Latent Reasoning Setup
Formal definitions of latent visual reasoning components and their two‑stage training procedure.
We introduce the concrete objects that underlie latent visual reasoning and describe how they are trained in two distinct phases.
During generation the model inserts a sequence of continuous latent states $z_t$ between the image input and the final answer, allowing the decoder to carry intermediate visual information without emitting a vocabulary token.
The model first learns to reconstruct visual targets from ground‑truth regions, then refines its latent dynamics using outcome‑level feedback while keeping the visual encoder fixed.
The ReaLVR Method
ReaLVR grounds latent tokens by contrastive visual and answer supervision during training.
Latent visual‑reasoning models often generate correct answers while their internal latent tokens remain unanchored to the visual evidence. This lack of grounding hampers interpretability and can waste capacity, motivating a training signal that forces latent states to preserve the image information that actually matters for the answer.
Credit is assigned to each latent position according to how much more the correct answer reads it than the average wrong answer does, turning answer‑contrast into a per‑token supervision signal.
Compute the differences: $0.2-0.4=-0.2$, $0.6-0.3=0.3$, $0.2-0.3=-0.1$.
Apply the positive part: $\gamma_1=0$, $\gamma_2=0.3$, $\gamma_3=0$.
The resulting credit vector is $(0,\;0.3,\;0)$, highlighting token 2 as the privileged position.
Only positions that the correct answer reads *more* than any wrong answer receive supervision, automatically filtering out noisy or irrelevant tokens.
How does this credit differ from simply using the attention weights of the correct answer?
Using raw correct‑answer attention would also reward positions that all answers attend to equally, which provides no discriminative signal. By subtracting the average wrong‑answer attention, the credit isolates *answer‑specific* preference, ensuring supervision focuses on evidence unique to the correct answer.
Positions are weighted by the contrast between correct‑answer and wrong‑answer attention, turning the contrast into a scalar weight $w_t$ that modulates the visual loss.
Compute each weight: $w_1 = 0.05 + 0.8\times0 = 0.05$.
$w_2 = 0.05 + 0.8\times0.4 = 0.05 + 0.32 = 0.37$.
$w_3 = 0.05 + 0.8\times0 = 0.05$.
$w_4 = 0.05 + 0.8\times0.1 = 0.05 + 0.08 = 0.13$.
The weight vector $(0.05,\;0.37,\;0.05,\;0.13)$ concentrates supervision on token 2 while still giving a minimal signal to the others.
The uniform term prevents the model from ignoring positions that receive zero contrast, keeping the training signal dense.
Why not renormalize the weights after adding the contrastive term?
Renormalization would erase the magnitude information encoded in $\gamma_t$, which the visual loss uses to scale the margin. Keeping the raw $w_t$ (detached) lets the loss reflect both *where* and *how strongly* supervision should be applied.
Latent tokens are encouraged to stay close to a visual prototype of the relevant region and far from prototypes of mismatched regions, using a cosine‑margin objective.
Compute cosine similarity of $z_1$ with $p^+$ and $p^-$: $\operatorname{sim}(z_1,p^+) = 0.9$, $\operatorname{sim}(z_1,p^-) = 0.2$.
Margin $g_1 = 0.9 - 0.2 = 0.7$.
Assuming $m_{ev}=0.5$, the hinge $[m_{ev} - g_1]_+ = 0$, so token 1 incurs no loss.
For token 2, similarities are $0.3$ (positive) and $0.8$ (negative), giving $g_2 = -0.5$ and a hinge loss $[0.5 - (-0.5)]_+ = 1.0$.
The example shows how a token that aligns more with a mismatched prototype is penalized, while a token that already respects the relevant prototype incurs no loss.
Why use the maximum similarity over negatives instead of an average?
The max captures the *hardest* negative—the one most likely to be confused with the positive—so the model learns to push away the worst distractor. Averaging would dilute this signal, allowing some negatives to dominate the loss without directly addressing the most confusable case.
Generate a full latent trajectory $z_{1:K}$ by recursively feeding the model its own hidden states.
Compute the positive visual prototype $p^+$ from the annotated mask and negative prototypes $\{p_j^-\}$ from mismatched examples.
For each latent token, evaluate the cosine margin $g_t$ against $p^+$ and the hardest negative.
Run the model forward to produce the correct answer $y^*$ and a set of wrong answers $Y^-_x$, extracting raw attention masses $r_t(y)$.
Derive the selective credit $\gamma_t = [r_t^+ - r_t^-]_+$ and blend with the uniform baseline to obtain weights $w_t$.
Form the visual loss $\ell_{ev} = \sum_t \operatorname{sg}(w_t)[m_{ev} - g_t]_+$ and add it to the original LVR objective with coefficient $\lambda_{ev}$.
Back‑propagate through the entire pipeline, updating the language‑model parameters while keeping the visual encoder frozen.
**Figure 2.** RealLVR learns visual grounding on its own latent trajectory. The model first generates continuous latent states from the image and question. Relevant and mismatched visual prototypes specify what these states should preserve. Correct and wrong answers are separately teacher-forced after the shared latent span; their attention contrast determines the supervision weights. The weights are detached, and the visual loss trains the latent-generation process. Both supervision branches are used only during training; inference follows the original LVR procedure.
Experimental Results
ReaLVR sets new state‑of‑the‑art accuracy on five visual reasoning benchmarks.
ReaLVR achieves the highest average accuracy across the five benchmarks, beating the best latent‑reasoning baseline by roughly 2 percentage points.
Table 1 reports a mean ± std of 63.7 ± 0.2 for ReaLVR versus 61.5 ± 0.3 for the next‑best latent method.
**Table 1.** Comparison with SOTA methods. Five-benchmark accuracy on Qwen2.5-VL-7B. We report mean ± standard deviation over three random seeds. Purple method cells mark non-latent baselines, green method cells mark latent-reasoning baselines.
**Figure 5.** Image and token views of the same three visual questions. ReaLVR and LVR on the same fence, bus, and icy-ground questions: image saliency overlays (top) and token-pair maps (bottom). Each pair is labeled with its question and answer.
**Figure 6.** A real visual answer and its latent-state text readout. RealVR (Gemma-3-12B) answers “27B” from an HR-Bench image (a). The original vocabulary head reads each of its eight latent states as <, as it does all 376 states from 47 questions (b), yet predicts 7 after 2, B after 7, and </ after B during answer generation (c). The latent readouts do not form an intermediate explanation.
ReaLVR consistently outperforms LVR baselines across diverse visual reasoning tasks.
The Evidence-Credit Gap
Details on training setup, benchmarks, and the evidence‑credit gap addressed by ReaLVR.
The paper’s core claim is that latent visual reasoning models often answer correctly without tying their internal tokens to the visual evidence, and ReaLVR remedies this by enforcing outcome‑contrastive supervision.
We call the mismatch between what a token attends to, what visual evidence it actually encodes, and whether changing it alters the answer the “Evidence‑Credit Gap”. It comprises three distinct questions: readout, grounding, and utility.
Three orthogonal checks expose whether a latent token truly contributes visual evidence to the final answer.
Benchmarks span low‑level visual discrimination (MMVP), perception tasks (BLINK), high‑resolution scenes (HRBench‑4K/8K), and real‑world complexity (MME‑RealWorld). Five backbones—Qwen2.5‑VL‑7B, Qwen3‑VL‑8B, Qwen3‑VL‑30B, InternVL3‑8B, and Gemma‑3‑12B—receive full results on all five benchmarks; the larger Qwen3‑VL‑235B is evaluated on three.
**Table 2.** Applicability across model sizes and families. The 235B evaluation covers MMVP, BLINK, and MME-RealWorld; a five-task average is reported only when all five scores are available.
Evaluation follows a matched protocol per family: we compute task accuracy, then average across benchmarks only when all five scores are present. Reported gains are the differences between the displayed one‑decimal means.
If a region‑of‑interest (ROI) annotation is missing or its mask empty, we set $a_n = 1$ for every token, yielding a whole‑image target $p^{+}$. This stabilizes the pooling denominator and guarantees a non‑zero prototype for the cosine margin; negative prototypes $p^{-}_s$ are built from mismatched examples using the same pooling rule.
Wrong‑answer construction discards parses that fail, correct answers, and candidates lacking answer‑content positions. The retained set $x = \text{Unique}\{(cid:98) y_i : i \in \{1,\dots,G\}, (cid:98) y_i \neq \bot, \text{Correct}((cid:98) y_i, y^\star) = 0, J((cid:98) y_i) \neq \emptyset\}$ ensures only valid, distinct, incorrect candidates contribute to contrastive supervision.
Gradients flow through the autoregressive latent‑generation process, so a loss term at position $t$ can update earlier generation steps and shared parameters. A token’s weight $w_{b,t}$ specifies its contribution to the loss rather than restricting updates to that position alone.
Selective credit defines $\gamma_t = [r^{+}_{\text{span}} \leq 1, (cid:80) t]$, bounded by $0 \le \gamma_t \le 1$. The unassigned mass $\gamma_{\emptyset} = 1 - \sum_{t=1}^{K} \gamma_t$ captures the remainder not allocated to any token.
**Figure B.1:** From final reward to latent evidence credit. The output reward indicates whether the answer is correct. RealVR uses relevant and mismatched visual evidence to determine what the latent tokens should preserve, then contrasts ground-truth and wrong-answer readouts to determine where supervision should be stronger. Intervention separately evaluates whether the answer depends on the latent tokens and is not part of the training objective.
Gradient Decomposition
How the model splits routing weights to keep supervision stable while avoiding shortcut paths.
The evidence‑credit gap creates a shortcut: gradients can lower a token’s routing weight instead of improving its visual evidence margin. To block this shortcut we decompose each token’s routing weight into a uniform component and a selective component, then stop‑gradient the uniform part.
Each latent token receives a blend of a tiny uniform share (to keep supervision alive) and a data‑driven selective share (to focus on the most relevant evidence).
Uniform share: $\frac{\eta}{K} = \frac{0.5}{4}=0.125$ for every token.
Selective shares: $(1-\eta)\gamma = 0.5 \times [0.4,0.3,0.2,0.1] = [0.20,0.15,0.10,0.05]$.
Final weights: $w = [0.125+0.20,\,0.125+0.15,\,0.125+0.10,\,0.125+0.05] = [0.325,\,0.275,\,0.225,\,0.175]$.
The bookkeeping token gets $w_{\varnothing}= (1-\eta)\gamma_{\varnothing}=0.5\times0=0$, so no extra loss term appears.
All weights sum to \$0.325+0.275+0.225+0.175 = 1.0$, confirming mass preservation.
The uniform component prevents any token’s weight from dropping to zero, which would otherwise let the gradient shortcut through the router instead of improving visual evidence.
How does this routing weight decomposition differ from simply using a softmax over evidence scores?
A plain softmax would allocate all mass based on $\gamma_t$, allowing some tokens to receive zero weight and creating a gradient shortcut through the router. By adding the uniform term $\frac{\eta}{K}$ and detaching its gradient, we guarantee every token keeps a non‑zero weight while preventing the loss from directly shaping the router.
Detaching the uniform part (stop‑gradient) means the optimizer only back‑propagates through $h_t$ and the selective term $(1-\eta)\gamma_t$, eliminating the second gradient path that could otherwise lower $w_t$ without improving visual evidence.
Latent-Length Ablation
We test how inference‑time latent length K and each ReaLVR component affect performance.
We keep the Qwen2.5‑VL‑7B ReaLVR checkpoint fixed and vary the inference‑time latent token budget K, including K = 0 which skips the latent span entirely.
**Table.** Latent-token budget $K$ (difference from $K = 8$)
Using the latent span (K = 8) yields the highest mean accuracy across the four benchmarks.
Table F.1 shows the mean difference is 0.0 at K = 8 and –4.4 at K = 0.
Removing the latent span entirely (K = 0) reduces mean accuracy by 3.0 percentage points.
Table F.1 reports a –4.4 pts mean drop at K = 0; the average of the four benchmarks is a 3.0‑pt loss relative to K = 8.
**Table F.2.** Component ablation on Qwen2.5-VL-7B. Each row removes or replaces one ingredient of ReaLVR; all other settings follow Appendix A. “Uniform routing” sets $\eta = 1$ so every latent position receives weight $1/K$ and the answer contrast is unused. “Selective only” sets $\eta = 0$. “No negatives” replaces the margin with a plain cosine alignment to $p^+$. “Raw attention” routes with $r_t^+$ instead of $[r_t^+ - r_t^-]_+$. “Undetached” removes the stop-gradient on $w_t$. “Off-policy” applies the evidence loss to the saved rollout latents instead of a regenerated trajectory. LVR-RL is the $\lambda_{ev} = 0$ endpoint.
Uniform routing ($\eta$ = 1) degrades average performance by 1.3 pts, with a 2.5‑pt drop on MMVP.
Table F.2 shows the “Uniform routing” row loses 1.3 pts on average and 2.5 pts on MMVP compared to the full model.
Routing with raw correct‑answer attention alone reduces the average score to 62.9 pts, 0.8 pts below the full ReaLVR result.
Table F.2 lists “Raw attention” achieving 62.9 pts versus 63.7 pts for the full model.
Selective‑only routing ($\eta$ = 0) lowers the average score by 0.5 pts.
Table F.2 reports “Selective only” at 63.2 pts versus 63.7 pts for the full model.
Omitting the negative‑contrast term (“No negatives”) drops average performance by 1.6 pts.
Table F.2 shows the “No negatives” row at 62.1 pts, 1.6 pts below the full model.
Using saved rollout latents instead of regenerated trajectories (“Off‑policy”) loses 1.8 pts on average.
Table F.2 records the “Off‑policy” variant at 61.9 pts, a 1.8‑pt deficit.
Removing the stop‑gradient on routing weights (“Undetached”) reduces average score by 1.1 pts.
Table F.2 lists “Undetached” at 62.6 pts, 1.1 pts below the full model.
Diagnostic Results
Each ablation reveals how much the full ReaLVR model relies on its components.
The ablation study isolates each design choice of ReaLVR to quantify its contribution to overall performance.
Removing the negatives component lowers the average accuracy by 6.2 points.
Full ReaLVR achieves 66.6 % average; the negatives‑only variant reaches 60.4 %.
Replacing the routing with raw attention reduces the average accuracy by 11.3 points.
The raw‑attention variant scores 55.3 % average versus the 66.6 % baseline.
Leaving the stop‑gradient attached (undetached weights) hurts performance by 0.4 points.
The undetached variant records 66.2 % average, only 0.4 % below the baseline.
Applying the evidence loss off‑policy drops the average accuracy by 3.4 points.
The off‑policy variant attains 63.2 % average compared with 66.6 % for the full model.
Using the LVR‑RL endpoint ($\lambda_{e}$v = 0) decreases the average accuracy by 4.5 points.
The LVR‑RL variant reaches 62.1 % average, 4.5 % lower than the full ReaLVR.
Omitting both negatives and off‑policy mechanisms together reduces the average accuracy by 10.8 points.
The combined ablation records 55.8 % average, a 10.8 % gap from the full model.
When the top‑k answer‑read latent tokens are progressively replaced, ReaLVR’s correct‑answer probability falls from 0.70 to 0.59, a 0.11 drop.
Figure G.1 shows a monotonic decline across k = 0…8, exceeding the declines of Monet and both LVR baselines.
ReaLVR enriches target‑region attention up to a factor of ≈2 in middle layers, outperforming Monet (≈1.6) and LVR variants (≤1.3).
Figure G.2 plots the enrichment ratio $\rho$_ℓ, where ReaLVR peaks near 2 while others stay near 1.
Answer‑conditioned saliency maps reveal that ReaLVR’s attention aligns with visual tokens more strongly than LVR‑7B, despite similar overall patterns.
Figure G.2 (saliency panels) displays higher intensity in ReaLVR for the same answer positions.
Latent trajectories remain highly consistent (RCS ≈ 0.9999) while occupying far fewer principal directions than visual embeddings.
Table G.2 reports Rank90 values of 111 for latent tokens versus >300 for visual embeddings, and participation ratios around 2 versus >5.
Forcing more latent tokens to follow target vectors improves cosine similarity, reaching perfect alignment when all 16 tokens are forced.
Table G.4 shows cosine rising from 0.2145 (free‑running) to 1.0000 (force all k = 16) for both BLINK and VISCOT.
Task‑label decodability is highest for raw pre‑hidden representations (ARI ≈ 0.95) and drops sharply for LVR‑based representations (ARI ≈ 0.39).
Table G.5(a) lists ARI values across representations, highlighting the gap.
Model flips on synthetic image edits are far below the ground‑truth flip rate, indicating limited sensitivity to answer‑changing perturbations.
Table G.5(b) shows “Model flip” percentages between 4 % and 13 % versus “Should flip” around 85 %.
**Figure 3.** Visual evidence and latent-token readout. Attention of LVR (a) and ReaLVR (b) on Qwen2.5-VL-7B, averaged over all layers and heads and 200 HR-Bench-4K examples, with image, question, and answer tokens averaged into 8/4/4 bins and the $K = 8$ latent positions shown individually. The gray upper triangle is the causal mask: each query can attend only to its own and earlier positions (Vaswani et al., 2017). Each row sums to one over allowed keys. Green tick labels mark the same visual-evidence keys (4, 6, and 7), the image bins overlapping the annotated ROI. The narrow frames show latent queries reading these visual keys; the bottom frames show answer queries reading latent states. Panel (a) shows weak readout along both links; panel (b) shows stronger readout along both. Please see Appendix I for further attention analyses.
**Figure G.1.** Fixed-context latent-token dependence. Each cell gives the correct-answer probability after replacing the top-k answer-read latent tokens, with all remaining latent states held fixed. Columns increase k from 0 to 8. RealVR's probability falls from 0.70 to 0.59, the largest endpoint drop among the four methods.
**Figure G.2:** **Target-region attention enrichment.** Layer-wise answer attention on the annotated region relative to matched background windows. Values above 1 indicate target-region enrichment; green and red curves correspond to correct and incorrect generations, respectively. Their overlap shows that alignment alone does not determine answer correctness.
Qualitative Results
Qualitative visual evidence shows ReaLVR’s latent tokens align more closely with the required visual cues.
ReaLVR exhibits a markedly higher top‑token variation gap (0.07) than Monet (0.02) and all LVR variants (0.01), indicating stronger position‑wise latent specialization.
Measured across the latent token indices 1–16, the gap quantifies how much variation concentrates on a few positions; ReaLVR’s gap is roughly three to seven times larger than the baselines.
**Figure H.1:** Attention and saliency cases (01–03). Top to bottom: the cap on a table, the bus beside a person, and the chair beside a tennis player.
**Figure H.2.** Attention and saliency cases (04–06). Top to bottom: the object behind the blender, the furniture left of the heater, and a furniture-material comparison.
**Figure H.3.** Attention and saliency cases (07–09). Top to bottom: pans left of the pots, the teddy-bear answer comparison, and a bear in water. In the middle example (Case 08), RealVR answers “teddy bear” and LVR-7B answers “blanket” to “What is touching the bed?”
**Figure H.4.** Attention and saliency cases (10–12). Top to bottom: the object in front of an airplane’s front wheel, the chair-and-bird scene, and the vehicle containing the pilot.
**Figure H.5.** One question from each of the five benchmarks. Images, questions, and answers come from the official datasets (Tong et al., 2024; Fu et al., 2024; Wang et al., 2025; Zhang et al., 2025). Robot bubbles show annotated answers, not recorded RealVR predictions; the chart calculation is added for clarity. Insets enlarge source-image details, and the chart is cropped to the relevant panel. Point and value labels are enlarged for readability. Multiple-choice options are omitted, and the chart question is shortened.
**Figure H.6:** **Position-wise latent variation.** (a) Cross-example variation. (b) Variation across questions about the same image. Heatmap columns index latent positions 1–16. (c) Reported top-token variation gap: 0.07 for RealVR, 0.02 for Monet, and 0.01 for each LVR variant. RealVR concentrates more of the measured variation at particular latent positions.