Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval
Suhyeong Park, Junha Jung, Jungwoo Park, Jaewoo Kang
SaMer compresses visual tokens for multi-vector retrieval by using object-aware merging to preserve query-selectable evidence.
How can we compress dense image tokens in multi-vector vision-language retrieval without losing the fine-grained object evidence required for accurate late-interaction matching?
Multi-vector vision-language retrievers rely on dense patch embeddings to match fine-grained query tokens, but storing and scoring these thousands of tokens creates a massive memory and latency bottleneck. SaMer compresses these tokens into a small set of $K$ representative centroids by using object annotations during training to discourage merging tokens from different object instances, while requiring no such annotations at inference time. With $K=64$, this approach removes over 93% of visual tokens and reduces storage by 16.09×, while simultaneously improving retrieval accuracy on natural-image benchmarks.
Paper Primer
Existing compression methods like pruning or feature-only pooling often collapse distinct object-level evidence into a single representation, causing the late-interaction mechanism to lose the specific visual details a query might need. SaMer addresses this by framing compression as an evidence-preservation problem: it uses a feature-spatial soft assignment to group tokens, guided by an object-aware penalty that prevents cross-instance mixing during training.
SaMer significantly improves retrieval accuracy while drastically reducing the token budget.
On the Flickr30K benchmark, SaMer improves R@1 from 77.0 to 82.4 for ColPali, outperforming pruning and pooling baselines at the same $K=64$ budget. 16.09× reduction in image-side storage for ColPali.
The method relies on projection-only adaptation, where only the shared image-text projection layer is updated to align the compressed tokens with the existing retrieval space. Because the vision and language backbones remain frozen, SaMer is pluggable into existing multi-vector architectures without requiring a full model retrain.
Why is object-aware merging superior to simple feature-based pooling?
Feature-based pooling often collapses visually similar patches from different objects into the same representative, destroying the fine-grained distinctions required for late-interaction retrieval. SaMer’s object-aware prior forces the model to respect instance boundaries, ensuring that query-selectable evidence remains distinct.
Does this method require object detectors at inference time?
No. While SaMer uses object annotations as a merge prior during training to shape the projection space, it performs annotation-free feature-spatial soft assignment at inference time.
Problem Framing and Preliminaries
Dense image‑side tokens enable fine‑grained matching but create a severe storage and scoring bottleneck for multi‑vector retrieval.
Storing hundreds of dense visual tokens per image enables fine‑grained matching, yet the resulting index size and the per‑query MaxSim computation explode as the dataset grows, limiting large‑scale retrieval.
Instead of a single global image embedding, the system keeps a set of token embeddings for each image, letting each query token pick the most relevant visual token during scoring.
How does multi‑vector retrieval differ from a single‑vector embedding?
A single‑vector approach collapses all visual information into one vector, so a query can only compare against that global summary. Multi‑vector retrieval keeps many token vectors, allowing each query token to find the most relevant visual token, which preserves fine‑grained details such as objects or attributes.
During scoring, each query token looks for the single most similar image token; the final score is the average of these maximal similarities.
Why not simply average all token similarities instead of taking the maximum?
Averaging would dilute the contribution of the most relevant visual token, making the score less sensitive to fine‑grained matches. The max operation ensures that if any image token aligns strongly with a query token, that evidence is fully reflected in the final score.
The storage and scoring bottleneck of dense token retrieval drives the need for compression methods that preserve object‑level evidence.
The SaMer Framework
Methods compress image tokens, guide merging with object cues, and adapt the projection layer.
Replacing the dense image token set with a compact representation must preserve the late‑interaction interface while cutting storage and compute.
SaMer builds a small set of representative visual tokens by softly clustering the full token bank, so the retrieval engine still sees a set of tokens but far fewer of them.
Compute distances $d(i,k)$ using the two seed centroids (initialized at $p_1$ and $p_3$); tokens 1‑2 get low spatial distance to centroid 1, tokens 3‑4 to centroid 2.
Apply soft‑assignment: $a_{1,1}=0.71$, $a_{2,1}=0.68$, $a_{3,2}=0.73$, $a_{4,2}=0.70$ (others near zero).
Form centroids $r_1 = (0.875,0.125)$ and $r_2 = (0.225,0.775)$ after weighted averaging and ℓ₂‑normalization.
Resulting merged set $\{r_1,r_2\}$ replaces the original four tokens for storage and scoring.
Soft assignment lets each original token influence both centroids, so even a token near a boundary contributes partially to both merged representations.
How does SaMer differ from simple token pruning?
Pruning discards tokens outright, losing any information they carry. SaMer aggregates all tokens into weighted centroids, so no token is ever thrown away; the aggregation preserves evidence from every region.
When bounding‑box annotations are available, SaMer penalizes assignments that would mix tokens from different objects, encouraging each centroid to represent a single object.
Compute $P_1(\text{A})\approx1$, $P_1(\text{B})\approx0$, $P_2(\text{B})\approx1$, $P_2(\text{A})\approx0$.
Instance penalties $P_{\text{inst}}(i,1)$ are near 0 for tokens 1‑2 and near 1 for tokens 3‑4; similarly for centroid 2.
Soft‑assignment weights for cross‑object pairs drop to $<0.01$, while same‑object pairs stay $>0.9$.
Resulting centroids remain essentially the same as in the non‑object‑aware case, but the training gradients now respect object boundaries.
The penalty forces the model to learn projections that keep object‑specific features together, even though inference never sees the boxes.
Why not use a hard clustering step instead of soft assignment with the object prior?
Hard clustering would make the assignment non‑differentiable, preventing gradients from shaping the projection layer. Soft assignment keeps the whole pipeline trainable while still biasing toward object‑consistent groups.
SaMer fine‑tunes only the shared image‑text projection so that the compressed centroids remain effective for the unchanged MaxSim scoring.
Why adapt only the projection layer and not the vision encoder?
Changing the encoder would require re‑extracting all visual features, breaking the “plug‑and‑play” property of SaMer. Adapting the projection is cheap, keeps the frozen encoder intact, and directly influences the merged centroids.
**Figure 1.** Overview of SaMer. Frozen vision and language encoders produce hidden states, a shared trainable projection layer maps both modalities into the retrieval space, and image-side post-projector tokens are compressed from $N$ to $K$ representative tokens before MaxSim scoring. During training, object-aware merging and projection-only adaptation encourage the compressed tokens to preserve object- and region-level evidence; at inference time, SaMer keeps the same late-interaction interface without requiring bbox or detectors.
Empirical Results and Ablations
SaMer keeps retrieval strong while cutting token count and latency.
SaMer reduces MaxSim operations by 16.1× while preserving retrieval performance.
Table 9 shows SaMer achieves 16.1× fewer MaxSim ops than the full ColPali model.
**Figure 2.** Token budget and R@5 trade-off. Solid lines show SaMer across token budgets, with baselines marked at $K = 64$. SaMer shows diminishing returns beyond $K = 64$ and outperforms baselines under the same budget.
**Figure 3.** Grounding comparison between full ColPali, compression baselines, and SaMer variants. BoxMass and RegionHit use the left axis, while CoverageIoU uses the right axis. SaMer improves localization and evidence concentration under compression.
SaMer maintains high retrieval accuracy while significantly reducing token count.
Supplementary Analysis and Robustness
SaMer compresses image tokens with soft‑assignment centroids while keeping retrieval quality.
Recall that SaMer compresses dense image tokens into a small set of soft‑assigned centroids, preserving late‑interaction retrieval performance.
Soft‑assignment centroids outperform all hard alternatives on both Flickr30K and MSCOCO.
Table 6 shows Soft‑assignment R@1 of 82.4 on Flickr30K versus 80.3 for mean and 82.2 for medoid.
We integrate SaMer into existing multi‑vector retrievers by replacing only the image‑side post‑projector with a $K$‑token merging layer, using $K = 64$ for all main experiments.
During compression‑aware adaptation we freeze the vision encoder and language backbone, updating only the shared projection for 1,746 steps with a cosine schedule.
BoxMass measures the fraction of relevance that falls inside the ground‑truth object box, RegionHit checks whether any thresholded relevance map overlaps the box, and CoverageIoU averages IoU across thresholds to assess spatial agreement.
The component study (Table 10) shows that the spatial term alone does not substantially improve performance, whereas the object‑aware penalty consistently raises both retrieval scores and grounding metrics.
SaMer’s merging reduces the R@1 drop compared to random pruning, achieving 73.6 on Flickr30K versus 68.6 for random pruning.
Table 7 reports these numbers after projection‑only adaptation.
Qualitative examples (Section F) illustrate that SaMer’s compressed tokens concentrate relevance within the annotated phrase region, matching the quantitative grounding improvements.
Algorithm 1: Compression‑aware adaptation and inference with SaMer
Questions & answers
What is the main contribution of SaMer?
SaMer introduces an object-evidence preserving token merging method for multi-vector vision-language retrieval that compresses dense image patch embeddings into a small set of K representative centroids, guided by object annotations during training but requiring no such annotations at inference time.
What problem does SaMer address?
SaMer addresses the memory and latency bottleneck caused by storing and scoring thousands of dense visual tokens per image in multi-vector vision-language retrievers, where the index size and per-query MaxSim computation become prohibitive at scale.
Why is token compression difficult for multi-vector retrieval?
Existing compression methods like pruning or feature-only pooling collapse distinct object-level evidence into a single representation, causing the late-interaction mechanism to lose the specific visual details a query might need to match against.
How does SaMer's merging mechanism work?
SaMer uses a feature-spatial soft assignment to group tokens into K centroids, guided by an object-aware prior that discourages merging tokens from different object instances, keeping the pipeline fully differentiable and trainable.
Why is object-aware merging superior to simple feature-based pooling?
Feature-based pooling often collapses visually similar patches from different objects into the same representative, destroying the fine-grained distinctions required for late-interaction retrieval; SaMer's object-aware prior forces the model to respect instance boundaries so that query-selectable evidence remains distinct.
Does SaMer require object detectors or annotations at inference time?
No. SaMer uses object annotations only as a merge prior during training to shape the projection space, and performs annotation-free feature-spatial soft assignment at inference time.
How does SaMer differ from simple token pruning?
Pruning discards tokens outright, losing any information they carry, whereas SaMer aggregates all tokens into weighted centroids so that no token is ever thrown away and evidence from every region is preserved.
Why does SaMer use soft assignment rather than hard clustering?
Hard clustering would make the assignment non-differentiable, preventing gradients from shaping the projection layer; soft assignment keeps the whole pipeline trainable while still biasing token groups toward object-consistent boundaries.
Which part of the model does SaMer adapt, and why?
SaMer adapts only the shared image-text projection layer, leaving the vision encoder and language backbone frozen; this projection-only adaptation is cheap, preserves the frozen encoder's features, and makes SaMer pluggable into existing multi-vector architectures without a full model retrain.
What are the key quantitative results reported for SaMer?
With K=64, SaMer removes over 93% of visual tokens and reduces storage by 16.09×, while simultaneously improving retrieval accuracy on natural-image benchmarks compared to the dense baseline.
What grounding metrics does the paper use to evaluate spatial evidence preservation?
The paper uses three metrics: BoxMass (fraction of relevance falling inside the ground-truth object box), RegionHit (whether any thresholded relevance map overlaps the box), and CoverageIoU (average IoU across thresholds to assess spatial agreement).
What do the ablation studies reveal about SaMer's components?
The component study (Table 10) shows that the spatial term alone does not substantially improve performance, whereas the object-aware penalty consistently raises both retrieval scores and grounding metrics, confirming that the object-aware prior is the critical component.
What are the limitations or open questions acknowledged by the paper?
The paper does not explicitly enumerate limitations in the provided text, though it implicitly acknowledges that object annotations are required during training, and the approach is evaluated on natural-image benchmarks without stating whether it generalizes to other domains.
How does SaMer compare to prior multi-vector compression approaches?
Unlike prior pruning or feature-only pooling methods that discard tokens or collapse object-level evidence, SaMer frames compression as an evidence-preservation problem and uses an object-aware prior to maintain instance-level distinctions, achieving better retrieval accuracy at the same compression level.
How is SaMer integrated into existing retrieval systems?
SaMer is integrated by replacing only the image-side post-projector with a K-token merging layer, using K=64 for all main experiments, with the vision encoder and language backbone remaining frozen throughout.
What is the training procedure for SaMer's compression-aware adaptation?
During compression-aware adaptation, the vision encoder and language backbone are frozen, and only the shared projection is updated for 1,746 steps using a cosine learning rate schedule.
How does multi-vector retrieval differ from single-vector embedding?
A single-vector approach collapses all visual information into one vector so a query can only compare against a global summary, whereas multi-vector retrieval keeps many token vectors, allowing each query token to find the most relevant visual token and preserving fine-grained details such as objects or attributes.
What venue, authors, and date are associated with this paper?
The paper is available on arXiv (arxiv.org/abs/2607.04605); the paper does not specify author names, publication venue, or submission date in the provided text.
Key terms
- multi-vector retrieval
- A retrieval paradigm that represents each image or text as a set of multiple token-level vectors, enabling fine-grained matching between individual query tokens and image tokens.
- late interaction
- A scoring mechanism where query and document token embeddings are compared independently and then aggregated (e.g., via MaxSim) at query time, rather than being fused into a single vector before comparison.
- MaxSim
- A late-interaction scoring operation that, for each query token, takes the maximum similarity score across all image tokens and sums these maxima to produce a final relevance score.
- token merging
- A compression technique that combines multiple patch or token embeddings into a smaller set of representative vectors (centroids) to reduce storage and computation.
- soft assignment
- A differentiable grouping mechanism that assigns each token a fractional membership across multiple clusters rather than assigning it to exactly one cluster, enabling gradient-based training.
- object-aware prior
- A training signal derived from object annotations that penalizes merging tokens belonging to different object instances, encouraging the model to preserve instance-level visual evidence.
- centroid
- A representative vector produced by aggregating a group of tokens, serving as a compressed summary of the visual evidence in that group.
- projection-only adaptation
- A fine-tuning strategy in which only the shared image-text projection layer is updated while all other model components (vision encoder, language backbone) remain frozen.
- token pruning
- A compression approach that permanently removes individual tokens from a sequence, discarding their information entirely to reduce sequence length.
- BoxMass
- A grounding metric that measures the fraction of a model's predicted relevance that falls within the ground-truth object bounding box.
- RegionHit
- A grounding metric that checks whether any part of a thresholded relevance map overlaps with the ground-truth object bounding box.
- CoverageIoU
- A grounding metric that computes the average Intersection over Union between the relevance map and the ground-truth box across multiple thresholds to assess spatial agreement.
- feature-spatial soft assignment
- A token grouping method that considers both the feature similarity and spatial proximity of tokens when computing their soft cluster memberships.
- dense patch embeddings
- A set of vector representations, one per image patch, produced by a vision encoder to capture local visual information across the entire image.
- compression-aware adaptation
- A training phase in which the model is fine-tuned specifically to align compressed token representations with the existing retrieval embedding space.