CORE: Improving Compositional Reasoning in MLLM Embedding via Reranker Distillation

Tingyu Song, Mingxin Li, Yanzhao Zhang, Dingkun Long, Chu Liu, Pengjun Xie, Yilun Zhao, Shu Wu

CORE improves compositional reasoning in multimodal embeddings by distilling fine-grained reranker judgments.

How can we improve compositional reasoning in MLLM-based embedding models by distilling knowledge from rerankers using synthetic graded data?

Multimodal embedding models often fail to distinguish scenes with identical concepts but different attribute-object bindings, such as swapping the colors of two objects in a scene. CORE addresses this by synthesizing graded compositional candidate lists and using a listwise Rank-KL objective to distill the reranker’s fine-grained compositional judgments into the embedding model. This approach achieves state-of-the-art compositional reasoning performance while preserving general retrieval capabilities on standard benchmarks.

Paper Primer

The core move is a two-stage framework: first, a synthesis pipeline generates candidate lists organized into a five-level taxonomy of compositional similarity; second, a Rank-KL distillation loss forces the embedding model to reproduce the reranker's relative scoring of these candidates. This is like a teacher providing a student with a ranked list of "how wrong" each answer is, rather than just marking them as correct or incorrect.

CORE-RERANKER-8B achieves superior compositional reasoning compared to existing rerankers.

Performance on compositional benchmarks (COLA, SUGARCREPE++, NEGBENCH). 82.7% total average, outperforming Jina-Reranker by 10.7 points.

CORE-EMBED-8B improves compositional reasoning without sacrificing general retrieval.

Evaluation on MCMR, COCO, and Flickr30K benchmarks. Improves MCMR R@1 from 0.375 to 0.412 over the backbone.

Why is a reranker better at compositional reasoning than an embedding model?

Rerankers use cross-attention to compare query and image tokens directly, allowing them to capture fine-grained attribute-object bindings that are often lost when compressing images and queries into fixed-length embedding vectors.

Why use Rank-KL distillation instead of standard contrastive learning?

Standard contrastive learning treats all negatives as equally wrong, whereas Rank-KL preserves the teacher's graded ranking, allowing the student to learn the relative compositional similarity between different types of errors.

By treating compositional reasoning as a ranking distillation problem rather than a binary classification task, researchers can inject fine-grained structural awareness into existing embedding backbones without losing general retrieval performance.

The Compositional Reasoning Gap

Embedding models struggle with fine‑grained compositional scenes, prompting a new distillation framework.

Multimodal information retrieval now underpins e‑commerce and web search, and Multimodal Large Language Models (MLLMs) are frequently used as embedding backbones. Yet these MLLM‑based embeddings still stumble on compositional reasoning, failing to separate scenes that differ only in fine‑grained attribute‑object bindings.

Figure 2(a) illustrates a concrete failure: the embedding model cannot tell apart two images that differ by swapping “white” and “black” between a plate and a chair, while both the MLLM and a cross‑attentive reranker correctly rank the appropriate candidate higher.

It is the capability to recognize and separate scenes that share the same objects but differ in how attributes are attached to those objects.

**Figure 2.** Motivating observations and data synthesis pipeline. (a) A fine-grained compositional case: the embedding model cannot distinguish the two candidates, while the MLLM and the reranker both judge correctly. (b) MRL diagnostic experiment: the experiment suggests that some compositional signals become more accessible at larger embedding dimensions. (c) The detailed data synthesis pipeline.

The failure of current embeddings to distinguish fine‑grained compositional scenes drives the need for CORE’s rank‑distillation approach.

Framework Overview and Benchmarks

We situate CORE within prior work on compositional reasoning, multimodal embeddings, and retrieval objectives.

Related work clusters around three themes: compositional reasoning benchmarks, multimodal embedding models, and objective functions for fine‑grained retrieval. Understanding these baselines clarifies why the CORE framework is needed.

**Figure 1.** **Overview of this study.** (a) The CORE framework: our data synthesis pipeline turns a random image into a graded five-level candidate list (top), which supports three training objectives: contrastive learning, CoSENT, and Rank-KL distillation (bottom). (b) CORE-RERANKER and CORE-EMBED achieve the best total average scores among rerankers and embedding models, respectively.

Benchmarks such as COLA and SUGARCREPE++ evaluate a model’s ability to bind attributes, objects, and relations in novel combinations, exposing a “bag‑of‑concepts” failure mode in many vision‑language systems.

Large‑scale MLLM‑based embedding models (e.g., CLIP‑style backbones) excel at generic cross‑modal retrieval but lack sensitivity to fine‑grained compositional distinctions without additional training.

Contrastive learning (CLIP), pairwise ranking (CoSENT), and listwise ranking (Rank‑KL) each provide a different signal for aligning queries with candidates, but their relative effectiveness for compositional tasks is underexplored.

CORE (Compositional Reasoning via Reranker Distillation) synthesizes graded candidate lists and distills the reranker’s compositional judgments into the embedding model, enabling embeddings to respect fine‑grained attribute‑object bindings.

These benchmarks present a five‑level taxonomy of compositional similarity, from full match to complete mismatch, allowing precise evaluation of a model’s ability to capture attribute‑object bindings.

The CORE Distillation Pipeline

Inject compositional reasoning into an embedding model via graded rank‑KL distillation.

Embedding models struggle with fine‑grained compositional differences because they never see graded similarity signals. We solve this by distilling a teacher’s nuanced ranking into the student via a single rank‑KL loss, while keeping the original retrieval ability.

We generate a short list of images that span a spectrum of matching levels, giving the model a graded supervision signal instead of a binary positive/negative.

Level 1 caption: “a red ball next to a blue cube” → image shows exact colors and shapes.

Level 2 caption: “a green ball next to a blue cube” → image changes ball color only.

Level 3 caption: “a red ball next to a metal cube” → image changes cube material.

Level 4 caption: “a blue ball next to a red cube” → image swaps the colors of both objects.

Level 5 caption: “a yellow sphere beside a purple pyramid” → image bears no relation to the original scene.

The graded list forces the student to assign intermediate similarity scores (e.g., level 2 should be higher than level 5 but lower than level 1), which a binary label cannot convey.

Instead of forcing all non‑matches away equally, we ask the student to mimic the teacher’s soft ranking over the graded list, preserving the relative ordering of partial matches.

Compute teacher probabilities: $\exp(0.8/0.1)=\exp(8)=2980$, $\exp(0.15/0.1)=\exp(1.5)=4.48$, $\exp(0.05/0.1)=\exp(0.5)=1.65$; normalize to get $P_T$.

Compute student probabilities: $\exp(0.6)=1.82$, $\exp(0.3)=1.35$, $\exp(0.1)=1.11$; normalize to get $P_S$.

KL divergence: $0.94\log(0.94/0.55)+0.05\log(0.05/0.33)+0.01\log(0.01/0.12)\approx 0.31$.

The loss penalizes the student most where its ranking diverges from the teacher (here the second candidate), driving the embeddings to respect the graded ordering.

Sample a minibatch of queries and their graded candidate lists $C$ (size $K=5$).

Run the teacher reranker on each $(q,d_i)$ pair to obtain scores $s_T(q,d_i)$.

Encode queries and images with the student dual encoder to get $v_q$ and $v_{d_i}$, then compute cosine scores $s_S(q,d_i)$.

Convert both score sets to probability distributions $P_T$ and $P_S$ via softmax (teacher temperature $\tau_T$, student temperature $\tau_S$).

Evaluate the Rank‑KL loss and back‑propagate gradients to update the student encoder.

Repeat until convergence, keeping the optimizer, learning‑rate schedule, and batch size fixed across all model sizes.

With Rank‑KL distillation the embedding model gains compositional awareness without sacrificing its original retrieval performance, ready for the downstream benchmarks.

Main Empirical Results

CORE‑EMBED outperforms baselines on compositional benchmarks.

CORE‑RERANKER‑8B attains the highest total average of 0.827, beating the strongest existing reranker, Jina‑Reranker (0.720), by 10.7 points.

Table 1 reports the total‑average scores for all rerankers across compositional benchmarks.

Across the benchmarks, the Swap subsets of SUGARCREPE++ are uniformly harder than the Replace subsets, indicating that detecting attribute‑object transpositions is a tougher compositional challenge for all models.

CLIP‑based models fall far behind MLLM‑based embeddings, reinforcing that strong compositional reasoning requires the semantic capacity of large vision‑language backbones.

CORE‑EMBED consistently outperforms baselines on compositional benchmarks.

Data Source and Scaling Analysis

CORE‑EMBED‑2B sets the top score on compositional benchmarks, and we dissect data and model choices that drive it.

CORE‑EMBED‑2B attains the highest COLA score (0.826), surpassing all prior embedding models.

Table 2 shows CORE‑EMBED‑2B leading with 0.826 on COLA, while the next best (UMarvel‑Qwen2VL‑7B) scores 0.813.

**Figure 3.** **Full dataset scaling results across all benchmarks.** Per-benchmark performance of CORE-EMBED as the number of training examples increases. Results are shown for COLA, SUGARCREPE++ (Replace Attribute, Replace Object, Replace Relation, Swap Attribute, and Swap Object), and NEGBENCH (VOC MCQ, COCO MCQ, and COCO Negation Retrieval) subtasks.

Increasing the LoRA rank during reranker fine‑tuning proves essential: ranks of 512 and 1 024 consistently outperform lower‑rank settings on COLA, highlighting the need for higher adaptation capacity when bridging the gap between synthetic training distributions and the fine‑grained compositional demands of the benchmark.

Training Strategy and Teacher Choice

Assessing how training choices affect compositional reasoning and general retrieval.

Distilling from the off‑the‑shelf reranker yields higher compositional benchmark scores than distilling from its pointwise fine‑tuned counterpart.

Average score $0.589$ vs $0.521$ on the compositional benchmark.

CORE improves compositional reasoning performance when applied to a different backbone (GME‑2B), confirming algorithmic robustness.

Table 4 shows consistent gains for both VL‑EMB‑2B and GME‑2B backbones.

CORE‑EMBED models retain strong general‑purpose retrieval performance while boosting compositional reasoning.

On COCO and Flickr30K, CORE‑EMBED‑8B achieves the highest R@5 and R@10 in both text‑to‑image and image‑to‑text directions.

Compositional gains transfer to an external multi‑condition retrieval benchmark (MCMR), indicating the method’s broader applicability.

CORE‑EMBED‑8B improves R@1 from $0.375$ to $0.412$ and MRR@10 from $0.469$ to $0.506$.

**Table 5.** Retrieval results on the COCO and Flickr30K datasets (R@5 and R@10) and on the MCMR benchmark (R@1 and MRR@10). Bold = best; underlined = second best.

Comparison of Training Objectives

Ablation compares four training objectives, showing Rank‑KL consistently outperforms alternatives.

This ablation isolates the effect of each training objective by keeping data, backbone, and tuning budget constant.

Rank‑KL training objective yields the highest scores across all five benchmarks.

Table 6 shows Rank‑KL surpasses Backbone, CL, and CoSENT on every metric.

Graded Structure and Limitations

We evaluate graded compositional similarity and analyze training objectives and limitations.

The paper’s premise is that MLLM embeddings miss fine‑grained compositional differences because they lack graded similarity training; CORE synthesizes graded candidate lists and distills reranker knowledge to remedy this.

We create a graded evaluation by holding out disjoint candidate lists and scoring each query against 100 graded candidates retrieved from Qwen3‑VL‑Embedding‑8B, treating this as an in‑distribution diagnostic.

Three objectives are compared under identical conditions: CoSENT (pairwise ranking), CL (InfoNCE contrastive), and Rank‑KL (listwise distillation).

Rank‑KL improves the subtask‑macro average to 0.641, surpassing the backbone’s 0.604.

Subtask‑macro average: 0.641 (Rank‑KL) vs. 0.604 (backbone).

The trend follows the amount of level information each objective uses: CL ignores levels and performs worst, CoSENT uses pairwise level order, while Rank‑KL further exploits teacher scores, yielding the strongest overall result.

CORE‑RERANKER‑8B reaches an 82.7 % total average, beating the previous best reranker by 10.7 points.

Total average: 82.7 % (CORE‑RERANKER‑8B) vs. 72.0 % (previous best).

CORE‑EMBED‑8B also attains the best average among evaluated embedding models, and the gains transfer to MCMR without sacrificing general retrieval performance.

Limitations include modest embedding gains, the in‑distribution nature of the graded evaluation, minimal improvement on COLA, and the fact that our diagnostic uses a single backbone and two SUGARCREPE++ subsets, leaving the compression bottleneck claim unproven.

Extended Benchmark Results

Key result: Qwen3VL‑2B reaches 0.936 total average, beating all rerankers.

Qwen3VL‑2B attains a total average of 0.936 on compositional reasoning benchmarks, outperforming all other reranking models.

Table 8 shows Qwen3VL‑2B achieving 0.936 total average, the highest among the listed rerankers.

**Figure.** COLA score performance across 5 epochs for different LoRA rank configurations (r=128, 256, 512, 1024).

Implementation and Ethics

Core implementation and ethical details for the CORE framework.

Our synthesized data derives from LAION‑400M seed images and is generated with publicly released models (Qwen3‑VL and Z‑Image‑Turbo), used under their licenses. Because the seed corpus is web‑crawled and the generated captions inherit social biases, both seed images and synthesized captions may contain undesirable content. The human verification study in §3.3 only asks annotators to judge compositional matching, collects no personal data, and all benchmarks are publicly available for research use only.

The bibliography lists all datasets, models, and prior works referenced throughout the paper.

The synthesized compositional dataset comprises $92,211$ query‑candidate tuples spanning five matching levels defined in §3.3, covering the full spectrum from complete matches to hard negatives. We train the reranker directly on this data, while the embedding model uses the same tuples for Rank‑KL distillation, with Z‑Image‑Turbo employed for data synthesis.

We fine‑tune the VL‑EMB student embedding model via LoRA adaptation applied to the attention projections ($q_{\text{proj}}$, $k_{\text{proj}}$, $v_{\text{proj}}$) and MLP layers (up, down, gate projections), keeping the visual encoder frozen. The LoRA rank is set to $32$, training runs for one epoch with learning rate $3 \times 10^{-5}$, a cosine schedule, warm‑up ratio $0.1$, and weight decay $0.1$, using bf16 mixed precision and gradient checkpointing. Inputs are truncated to $1{,}500$ tokens, embeddings are obtained via last‑token pooling followed by $\ell_2$ normalization, with student and teacher temperatures fixed at $\tau_S = 0.05$ and $\tau_T = 0.05$, and training is performed on four A100 GPUs.

The off‑the‑shelf QWEN3VL‑RERANKER‑2B and QWEN3VL‑RERANKER‑8B models are fine‑tuned on the same compositional data, yielding CORE‑RERANKER‑2B and CORE‑RERANKER‑8B. LoRA adaptation with a higher rank of $512$ is applied to the same attention and MLP projections, using the same AdamW optimizer, cosine schedule, and bf16 precision. This higher rank follows the analysis in §5.2, which shows that compositional attribute‑object binding benefits from greater adaptation capacity.

Evaluation Settings

Lists the models evaluated and the token limits used for evaluation.

Table 7 enumerates the 13 models we evaluate, split into two families. The CLIP‑based family includes SigLIP2, NegCLIP, and Triplet‑CLIP; the MLLM‑based embedding family comprises VLM2Vec, VL‑EMB‑2B, VL‑EMB‑8B, UniME‑2B, UniME‑7B, GME‑2B, GME‑7B, UMarvel‑Qwen2VL‑7B, and UMarvel‑Qwen3VL‑4B. All evaluations use a cap of 1,000 image tokens and 200 text tokens unless a model specifies otherwise.

**Table 7.** Details of the evaluated embedding and reranking baselines used in the main paper.

Synthesis Pipeline Prompts

Appendix C lists the exact prompts and validation filters used for compositional data synthesis.

This appendix provides the concrete prompts that drive our compositional data synthesis pipeline and the filters that ensure generated images match the intended queries.

Information Extraction Prompt – extracts structured scene information from a seed image.

Query and Image Caption Generation Prompt – creates a retrieval query and five graded captions.

The Image Filter checks whether a generated image is consistent with both its caption and the retrieval query.

Image Filter logic – validates caption and query consistency.

Compliance and reproducibility: all datasets and models were used under their licenses, and any LLM assistance was limited to grammar checking, with authors retaining full responsibility for the content.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers