WeMM-Embedding: WeChat Multi-Modal Embedding

Junjie Zhou, Ke Mei, Lei Li, Tianyi Wang, Fengyun Rao, Jing Lyu

WeMM-Embedding is a family of universal multimodal models that achieves state-of-the-art retrieval performance via a two-stage training strategy.

How can we build a universal multimodal embedding model that effectively aligns heterogeneous content (text, image, video, document) for large-scale industrial applications?

Universal multimodal embedding models struggle to maintain high performance across diverse tasks like image retrieval, video understanding, and document analysis because they often lack fine-grained relevance supervision and suffer from skewed semantic distributions in large-scale training data. The authors introduce WeMM-Embedding, a model family that uses a two-stage training process: first, broad multimodal alignment on massive heterogeneous data, followed by refinement on a curated corpus using hard-negative enrichment, reranker-derived supervision, and knowledge distillation from a larger teacher model. The 9B variant achieves a state-of-the-art score of 80.6 on the MMEB-v2 benchmark, while the compact 2B variant outperforms previous 8B-scale open-source baselines.

Paper Primer

WeMM-Embedding leverages the Qwen3.5 backbone to encode arbitrary interleaved text, image, and video inputs into a unified dense space. The core mechanism is a two-stage training pipeline: stage one establishes broad alignment via contrastive learning, while stage two refines the model using curated data and "soft" similarity targets distilled from a larger teacher model.

WeMM-Embedding achieves state-of-the-art performance on the MMEB-v2 benchmark.

The 9B model variant reached an overall score of 80.6, outperforming all listed open-source and proprietary models on the official leaderboard.

The model family demonstrates high parameter efficiency.

The 2B variant surpasses the performance of previously leading 8B open-source baselines on the MMEB-v2 benchmark.

Why is a two-stage training strategy necessary instead of just training on the largest possible dataset?

Large-scale data provides broad coverage but often lacks the fine-grained relevance signals required for challenging retrieval tasks. The second stage uses curated data and teacher distillation to improve semantic balance and discrimination among semantically similar candidates.

What is the practical scope of this model beyond public benchmarks?

The model is deployed in production at WeChat, supporting recommendation and search across Official Accounts, Moments, and e-commerce, where it has shown consistent gains in 14 online A/B tests.

WeMM-Embedding demonstrates that combining large-scale alignment with targeted distillation and curated hard-negative mining allows smaller, more efficient models to outperform significantly larger baselines.

Introduction and Motivation

We expose the need for universal multimodal embeddings that handle arbitrary interleaved inputs at industrial scale.

Current multimodal embedding models rely on modality‑specific encoders that process each modality separately. This design prevents joint representation of interleaved text‑image documents, multimodal queries, or video‑transcript pairs, limiting applicability in modern AI systems.

**Figure 1.** Performance and efficiency overview of WeMM-Embedding. Left: MMEB-v2 overall performance across different model sizes, compared with representative baselines. Right: Aggregate performance on the MMEB-v2 image and video subsets, a 12-dataset cross-modal retrieval suite, and the 26-task in-house benchmark.

A single dense vector that can encode any combination of text, images, video frames, or document tokens, enabling downstream models to operate without modality‑specific preprocessing.

The industry shift toward universal multimodal embeddings promises tighter integration of heterogeneous data in large‑scale AI services.

Data Construction

We unify massive multimodal data into a single pair format and curate it for balanced, high‑quality training.

Training a universal multimodal model requires billions of heterogeneous examples, yet existing datasets differ in modality, supervision, and scale, making a single training pipeline infeasible.

Think of each example as a bilingual dictionary entry: an instruction (optional) points to a source item, which is linked to one target item and optionally a set of hard negatives, mirroring how a word maps to its correct translation and to plausible confusions.

Pack each example into a tuple $z_i$ as defined above.

During the forward pass, the model encodes $q_i$ and all candidates $\{c_i\}\cup N_i$ into a common embedding space.

Compute similarity scores between $q_i$ and each candidate; the positive target should obtain the highest score.

Apply a contrastive loss that pushes the positive above the negatives, weighted by $y_i$ when present.

Back‑propagate gradients jointly for both examples, sharing the same optimizer step.

This concrete packing shows how a single loss can handle binary matches, graded relevance, and hard‑negative contrast without any architectural changes.

How does this pair‑based format differ from traditional supervised pairwise training?

Traditional pairwise setups fix a single negative per example, often sampled randomly. Our format allows an arbitrary set of hard negatives $N_i$ and an optional graded score $y_i$, enabling richer contrast and ranking signals while keeping the loss unchanged.

Even with a unified format, raw web‑scale data are noisy and semantically imbalanced, so we curate a smaller, higher‑quality subset.

Imagine a library where popular books dominate the shelves; we deliberately pull back those titles and promote rarer ones so readers see a broader range. Here we use a learned “semantic ID” to down‑sample dense regions of the embedding space.

Count occurrences per coarse code: 00 → 2, 01 → 1, 10 → 1.

Assign sampling probabilities inversely proportional to count (e.g., 00 gets 0.3, 01/10 get 0.85).

Randomly keep the first 00 example, discard the second, and retain both 01 and 10 examples.

Resulting batch contains three diverse semantic IDs instead of four redundant ones.

By throttling over‑represented codes, the model sees a richer set of concepts early in training, which improves downstream generalization.

Why not simply random‑sample the raw data instead of using Semantic IDs?

Random sampling ignores the underlying semantic distribution; frequent patterns would still dominate, leaving the model under‑exposed to rare concepts. Semantic‑ID‑guided sampling explicitly balances the representation of semantic clusters.

Hard negatives are like decoy answers on a multiple‑choice test: they look plausible, forcing the model to sharpen its discrimination.

Why not use only in‑batch negatives instead of generating explicit hard negatives?

In‑batch negatives are random and often trivially easy; explicit hard negatives are deliberately chosen to be close to the positive, providing a stronger training signal that improves fine‑grained discrimination.

**Figure 2.** Overview of our multimodal training data. Major data families and representative coverage across diverse task settings and content domains.

Model Architecture

The model encodes heterogeneous multimodal inputs into a single normalized embedding token.

Heterogeneous multimodal streams (text, images, video) lack a common anchor, making downstream tasks struggle to consume a single representation.

We prepend a special token to the multimodal token stream and let the LLM’s causal attention aggregate all preceding modalities into its final hidden state, which becomes the universal embedding.

Token order: [t₁, t₂, t₃, v₁, v₂, v₃, v₄, v₅, v₆,

The LLM processes the sequence; each token’s hidden state is computed by attending to all earlier tokens.

The

We extract $h_{\text{emb}}$ (a 768‑dim vector) from the final layer.

Normalize: $e_D = h_{\text{emb}} / \|h_{\text{emb}}\|_2$, producing a unit‑norm embedding.

This toy example shows that a single token can absorb arbitrarily many modalities simply by being placed at the sequence tail.

How does this differ from the CLS token used in BERT‑style models?

CLS is a learned classification token that attends bidirectionally to the entire sequence, whereas the token uses the causal mask of the generative backbone and therefore only aggregates past tokens, preserving the model’s autoregressive nature.

After obtaining the full‑dimensional embedding $h_{\text{emb}}$, we can truncate its prefix to any supported size $d$ and renormalize, yielding a smaller embedding without extra computation.

For $d=2$, take the prefix $(0.6, 0.8)$.

Compute its norm $\sqrt{0.6^2 + 0.8^2}=1.0$.

Normalize: $e(2) = (0.6, 0.8) / 1.0 = (0.6, 0.8)$.

For $d=4$, the full vector norm is also $1.0$, so $e(4)=h_{\text{emb}}$.

The example shows that truncation does not degrade the vector when the discarded components are already zero, illustrating why early dimensions often capture the bulk of the signal.

Is MRL simply PCA applied to the final hidden state?

No. PCA learns a data‑dependent linear projection, whereas MRL relies on the pretrained backbone’s ordering and merely drops trailing dimensions, followed by a norm‑preserving scaling. The operation is deterministic and requires no extra parameters.

Encoding pipeline for a multimodal instance.

Training Strategy

We train embeddings in two stages: broad alignment then focused fine‑tuning with hard negatives and distillation.

Training on hundreds of millions of multimodal pairs creates two problems: (1) the loss sees many near‑duplicate items that confuse the contrastive signal, and (2) a single generic objective cannot capture the nuanced relevance needed for downstream tasks.

First we teach the model a universal language by aligning massive, noisy multimodal pairs; then we polish that language on curated data, injecting hard negatives, reranker rankings, and teacher‑student distillation to make the embeddings robust for any downstream task.

Duplicate‑aware mask zeros out any candidate whose similarity to a non‑matching source exceeds $\tau$₍dup₎. Here, c₁ is masked for q₂ because s(q₂,c₁)=0.88>0.8.

After masking, the contrastive denominator for q₂ contains only c₂⁺ and c₃, so the softmax denominator shrinks, raising the probability of the true positive.

Compute the InfoNCE loss for q₂: –log( exp(0.92/$\tau$) / (exp(0.92/$\tau$)+exp(0.45/$\tau$)) ). With $\tau$=0.07, the loss ≈ 0.12, much lower than without masking (≈0.45).

Apply the CoSENT‑style ranking loss on a graded‑relevance pair (q₁,c₁⁺) with label y=2 and (q₁,c₃) with label y=0. The weight w = max(|2−0|, $\epsilon$)=2, so the exponential term is amplified, pushing s(q₁,c₁⁺) higher.

Run MRL at dimensions d∈{256,128,64}. For this toy, we truncate embeddings to the first dimension, recompute all similarities, and add the same losses with $\alpha$₍d₎ weights (e.g., $\alpha$₂₅₆=0.5, $\alpha$₁₂₈=0.3, $\alpha$₆₄=0.2).

Masking removes spurious negatives that would otherwise penalize correct pairs, while the weighted ranking term forces the model to respect large relevance gaps; evaluating the same loss at multiple dimensions guarantees that a sliced‑down model retains the learned structure.

How does this two‑stage approach differ from a single‑stage contrastive training pipeline?

In a single stage the model sees only one loss on the same data distribution, so it cannot benefit from high‑quality curated signals or from a teacher’s soft similarity distribution. The two‑stage design first builds a coarse universal space with massive noisy data, then refines it with targeted hard negatives, reranker rankings, and bidirectional KL distillation, which together improve fine‑grained alignment without sacrificing the breadth learned earlier.

Sample a batch from the massive multimodal corpus; each example may be a plain pair or a graded‑relevance item.

For plain pairs, compute InfoNCE contrastive loss; for graded items, compute the CoSENT‑style ranking loss.

Apply duplicate‑aware masking (threshold $\tau$₍dup₎) to drop near‑duplicate sources or targets from the negative pool.

Evaluate the selected loss at every dimension in D₍MRL₎, weighting each with $\alpha_{d}$, and sum to obtain `L_MRL`.

Back‑propagate `L_MRL` and update the model parameters.

Draw a batch from the curated dataset; identify its supervision type (standard pair, graded relevance, or reranker‑scored).

Compute the corresponding task loss: `L_CL`, `L_Rel`, or `L_Rank`, each already wrapped by MRL across dimensions.

If a larger teacher model is available, compute source‑to‑target and target‑to‑source softmax distributions from the teacher.

Calculate bidirectional KL divergence between teacher and student distributions, scale by $\lambda_{E}$mb, and add to the task loss.

Back‑propagate the combined loss `L_Stage2` = `L_Task` + $\lambda_{E}$mb·`L_Emb` and update the student.

The core trick is a two‑stage schedule that first learns a universal embedding on massive noisy data, then refines it with curated hard negatives, reranker rankings, and teacher‑student distillation, all evaluated at multiple dimensions.

Benchmark Results

WeMM-Embedding sets new records on MMEB‑v2 and MMEB‑v3 across model sizes.

The paper’s core premise is that a two‑stage training on massive multimodal data yields embeddings that work universally. This section shows how that claim holds on the MMEB benchmark series.

A standardized collection of 190 tasks spanning images, video, text, audio, and agent‑oriented retrieval, used to measure how well a multimodal embedding works across heterogeneous content.

WeMM‑Embedding 9B tops the official MMEB‑v2 leaderboard with an 80.6 AVG, beating the previous best by 4.7 points.

Table 1 shows the 9B variant achieving 80.6 AVG, while the next‑highest open‑source model (DME‑Large) scores 80.2.

On the more demanding MMEB‑v3, the 9B WeMM‑Embedding reaches 59.5 V3‑All, outpacing the strongest baseline by 2.3 points.

Table 2 reports 59.5 for the 9B variant; the best competing model listed scores 57.2.

Across both benchmark versions, the 4B and 2B variants also lead their size class, with 79.2 AVG (v2) and 58.2 V3‑All (v3) respectively, confirming consistent scaling benefits.

**Table 1.** Benchmarking results on MMEB-v2. Baseline results are taken from the official MMEB leaderboard. †Closed-source leaderboard submission without publicly released model weights or a public inference endpoint. *Proprietary commercial model with an undisclosed parameter count. CLS: classification; QA: question answering; Ret: retrieval; GD: visual grounding; V-Ret: video retrieval; M-Ret: moment retrieval.

**Table 2.** Evaluation results on MMEB-v3. V3-All averages all 190 tasks, comprising the 78 MMEB-v2 tasks, 53 Text tasks, 47 Agent tasks, 11 Audio tasks, and MCMR [10]. Following the evaluation protocol defined in the MMEB-v3 paper [15], Text results are reported using NDCG@5. Unsupported tasks are assigned a score of zero. RR: reasoning retrieval; IF: instruction following; LC: long-context retrieval; MC: multi-condition retrieval; GR: general retrieval.

WeMM‑Embedding consistently outperforms all baselines on both MMEB‑v2 and MMEB‑v3, demonstrating that the two‑stage training yields truly universal multimodal embeddings.

Cross-Modal Retrieval

WeMM‑Embedding 9B tops the cross‑modal retrieval leaderboard.

WeMM‑Embedding 9B achieves the highest average cross‑modal retrieval score of 81.7 across the 12 public benchmarks.

Table 3 shows it outperforms all open‑source baselines and matches leading proprietary models.

Given a query in one modality (e.g., text) and a database in another (e.g., images), the system ranks items by the semantic similarity of their embeddings.

**Table 3.** Cross-modal retrieval results on 12 public benchmarks. AVG denotes the average across the 12 datasets. $^\dagger$Proprietary commercial model with an undisclosed parameter count.

Industrial Application Evaluation

WeMM-Embedding outperforms the baseline on the in‑house benchmark across all tasks.

WeMM‑Embedding achieves an average score of 60.9 on the in‑house benchmark, beating the open‑source baseline on every metric.

Table 4

Across categories—search, cross‑domain matching, article relevance, and video relevance—WeMM‑Embedding consistently exceeds the baseline, with gains ranging from 4 to 7 points per metric.

**Table 4.** Evaluation results on the in-house benchmark. AVG denotes the average over all 26 tasks. Cross-DM denotes cross-domain content matching; Article Rel. denotes article relevance; Video Rel. denotes video relevance.

Ablation and Analysis

We dissect how each component impacts performance and dimensionality trade‑offs.

We evaluate how the Matryoshka Representation Learning (MRL) trick behaves when embeddings are shrunk, and we quantify the impact of each Stage‑1 and Stage‑2 design choice.

MRL trains a single model to emit a hierarchy of nested embeddings, so the same network can be queried for a 64‑dim vector, a 256‑dim vector, or a full 2,048‑dim vector without extra parameters.

How does MRL differ from naïvely training separate low‑dim models?

Separate low‑dim models learn independent parameters for each size, so they cannot share semantic structure and must be trained from scratch. MRL, by contrast, forces a single backbone to serve all sizes, guaranteeing that the low‑dim vectors are consistent projections of the same high‑capacity representation.

**Figure 3.** MRL analysis of WeMM-Embedding-2B on MMEB-v2. Left: Performance retained on the image, video, and visual-document subsets across embedding dimensions. Right: Performance retained for classification (CLS), question answering (QA), and retrieval (RET), each averaged over the corresponding image and video tasks.

At 256 embedding dimensions, MRL retains over 98 % of full‑dim performance on image and video tasks and exceeds 97 % across all task groups.

Figure 3 shows 98.7 % retention for image/video at 256 dims and >97 % retention for classification, QA, and retrieval once the dimension reaches 256.

Cumulative Stage‑2 strategies raise the average MMEB‑v2 score by 2.2 points, reaching 80.7.

Table 6 shows the final configuration (+ expanded visual input budget) achieving an average of 80.7 versus 78.5 for the Stage‑1 checkpoint.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers