NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference

Aurélien Lac, Tony Wu

NeoMME is a multimodal-native bidirectional encoder that unifies text and image processing in a single Transformer.

How can we build a single-tower, multimodal-native encoder that handles both text and images in a shared space without relying on separate vision towers or causal decoders?

Multimodal document retrievers typically rely on separate vision and language towers, creating significant parameter and compute overhead for non-generative tasks. NeoMME replaces these dual-tower architectures with a single bidirectional Transformer that processes raw image patches and text tokens in a shared hidden space, trained from scratch using a masked discrete-diffusion objective. On the ViDoRe v3 benchmark, the 260M-parameter model outperforms all evaluated retrievers of similar size, while hierarchical pooling and quantization compress document embeddings by 255× with minimal quality loss.

Paper Primer

NeoMME treats images as sequences of 32×32 patches and text as standard tokens, feeding both into a shared backbone that uses 2D rotary position embeddings to maintain spatial awareness. By training from scratch with a masked-text denoising objective conditioned on visible image patches, the model learns to integrate visual evidence without needing a pretrained vision encoder.

NeoMME-Retriever 260M achieves state-of-the-art retrieval performance for its size class.

On the ViDoRe v3 benchmark, it scores 0.523 nDCG@10, outperforming all models with fewer than 800M parameters. It delivers 1.97× the throughput of ColModernVBERT at a matched 2048×2048 input resolution.

Late-interaction embeddings can be aggressively compressed without significant retrieval degradation.

Combining hierarchical token pooling and asymmetric quantization reduces embedding size from ~1.5 MB to 6 kB per document. This 255× compression retains over 95% of the baseline nDCG@10 score.

Why use a single-tower bidirectional encoder instead of the standard generative VLM approach?

Sharing the Transformer backbone provides a unified computational path for both modalities, avoiding the overhead of separate vision towers and enabling more efficient fine-tuning and inference for retrieval tasks.

Does this model require OCR or external text extraction to process documents?

No. NeoMME processes raw image patches directly, allowing it to capture layout, tables, and figures that traditional OCR-based pipelines might lose.

Motivation and Contribution

We expose the inefficiencies of multi‑tower multimodal models and motivate a single‑tower bidirectional encoder.

Current multimodal systems stitch together a pretrained vision encoder and a causal language model, which inflates parameter counts and compute for retrieval tasks that do not require generation.

Separate visual and textual towers force the model to carry duplicated parameters and to run two distinct forward passes, making retrieval slow and memory‑hungry.

NeoMME embodies this shift: a bidirectional Transformer encoder that ingests multilingual text and raw 32 × 32 pixel image patches through modality‑specific projections, then processes them jointly.

**Figure 13.** Compiled document-encoding throughput over the resolution grid on one NVIDIA L40S.

**Figure 12.** Quality–storage frontier for the NeoMME-260M late-interaction index on ViDoRe v3. Labels show pool factor, retained quality, compression, and storage.

**Table 18.** Selected image inputs and text generated by the NeoMME-260M model. [...] marks truncation.

The key insight is that moving from multi‑component to single‑tower multimodal encoders yields a leaner, faster, and storage‑efficient foundation for retrieval.

Context and Prior Art

Survey of prior text, diffusion, and multimodal models relevant to NeoMME.

This section situates NeoMME among prior text encoders, diffusion‑based representations, multimodal backbones, and visual‑document retrieval systems.

Early bidirectional pretraining (BERT) and its extensions—Sentence‑BERT, DPR, ColBERT, ModernBERT, EuroBERT, mmBERT—adapt encoder representations for dense and late‑interaction retrieval.

Methods such as LLM2Vec, LFM2.5‑Encoder, and BidirLM convert causal decoder models into bidirectional encoders, enabling retrieval‑oriented representations.

DiffusionBERT, MDLM, MD4, and LLaDA extend masked‑language modeling with a sampled noise trajectory, producing diffusion‑pretrained hidden states useful for retrieval.

Diff Embed, PPLX‑Embed, and Diff Retriever leverage diffusion‑pretrained states to obtain dense or multi‑vector embeddings for retrieval.

CLIP and SigLIP train image and text encoders independently, enabling offline image indexing but limiting cross‑modal interaction.

Flamingo, BLIP‑2, LLaVA, PaliGemma, and Qwen2‑VL insert gated or lightweight cross‑attention modules to fuse vision and language while retaining a separate visual backbone.

Models such as ViLT, UFO, Uni‑Perceiver, OneR, M3AE, VLMo, BEiT‑3, and the shared‑backbone EVE reuse a single Transformer across modalities, often initializing from pretrained vision models.

Recent works (Fuyu, SOLO, NEO, Chameleon, Gemma 4 Unified) feed raw image patches directly to decoder‑oriented backbones, eliminating explicit vision towers.

LayoutLMv3 combines OCR tokens, layout boxes, and visual tokens; DSE, ColPali, and ColQwen2 produce dense or multi‑vector page embeddings for retrieval without OCR.

ModernVBERT, Jina Embeddings v4, and Nemotron ColEmbed V2 extend dense or multi‑vector retrieval pipelines, often integrating pretrained vision encoders.

Late‑Interaction (multi‑vector) embeddings increase representational capacity compared to single‑vector dense embeddings, but incur higher storage and scoring costs; methods such as PLAID, MUVERA, FLASH‑MAXSIM, and MaxSim2 mitigate these costs.

The NeoMME Architecture

NeoMME encodes text and image patches together in a single bidirectional Transformer, eliminating separate vision and language towers.

NeoMME replaces the usual two‑tower setup (separate vision encoder + language decoder) with one bidirectional Transformer that processes text tokens and image patches together.

How does this differ from a dual‑tower retrieval system?

In a dual‑tower system the image and text are encoded separately and only compared after encoding, so cross‑modal interactions happen only at the final similarity step. The single‑tower design lets the Transformer mix modalities throughout the network, enabling richer joint representations and reducing the overall parameter budget.

Instead of fixing a single image resolution, NeoMME samples a longest‑side cap per example, letting the number of image patches grow with the original image size.

The longest side ($800$ px) is below the cap, so the image is kept at $800\times600$ px.

Dividing each dimension by the patch size $32$ yields $\lceil800/32\rceil = 25$ patches horizontally and $\lceil600/32\rceil = 19$ patches vertically, for a total of $25\times19 = 475$ image tokens.

These $475$ tokens are projected into the model dimension and concatenated with the text token stream.

The encoder now attends over $475$ image tokens plus the $N_{\text{text}}$ text tokens, preserving fine‑grained visual detail for this relatively large image.

Token count scales with image size, so larger images receive more visual tokens while smaller images do not waste capacity on unnecessary patches.

Why not simply resize every image to a fixed size?

Fixed resizing forces all images to share the same token budget, discarding detail for large images and adding redundant tokens for small ones. Dynamic resolution adapts the token count to the image’s intrinsic size, preserving detail where it matters and keeping compute proportional to the visual information.

Layer schedule: alternating sliding‑window and global attention.

**Figure 1.** Unlike dual-tower and VLM encoders, NeoMME processes image patches and text tokens in one bidirectional Transformer, without a pretrained vision tower or causal decoder.

**Figure 2.** Dynamic-resolution image processing with a variable side-length cap.

**Figure 3.** Alternating sliding-window and global-attention layers in the NeoMME encoder stack.

**Figure 4.** Pre-normalized attention and MLP paths in one NeoMME encoder layer.

**Figure 5.** Parameter allocation by module group for NeoMME-260M and NeoMME-800M. Counts include the backbone and input paths but exclude the contrastive-retrieval heads.

**Table 1.** NeoMME architecture settings.

**Table 16.** Tokenizer compression on the 14 target languages in FLORES-200 devtest. All tokenizers encode identical texts without added special tokens. Values are tokens per UTF-8 byte, and lower values are better.

**Table.** ViDoRe v3 performance metrics across various tasks.

Pretraining Strategy

Pretraining teaches a masked‑diffusion denoiser that jointly consumes text and visible image patches.

The pretraining phase builds the shared backbone from scratch on a mix of pure‑text and multimodal examples, forcing the model to rely on image evidence rather than textual shortcuts.

Dataset 4.1 draws from fourteen text‑only corpora and a multimodal collection, sampling tokens according to the weights in Table 2 so that 55 % of packed tokens come from text‑only streams and 45 % from multimodal streams.

**Table.** Data composition for training, showing the distribution of text-only and multimodal datasets, their within-stream weights, and the expected number of tokens.

Pack each document into a 16,384‑position stream and enforce variable‑length attention boundaries to keep streams isolated.

Decode images in background workers, split them into patches, and enqueue the patches for the next training step.

Prefetch packed batches while the previous step is still back‑propagating, overlapping I/O with compute.

Apply a compute‑aware schedule that spreads multimodal work evenly across data‑parallel ranks.

Run the forward/backward passes with FlashAttention‑3, Liger Kernel’s fused cross‑entropy, and torch.compile to keep GPU utilization high.

Section 4.3 defines the core pretraining objective: a discrete masked‑diffusion denoiser that predicts masked tokens while always seeing the associated image patches.

Mask a random subset of text tokens, keep image patches visible, and train the model to reconstruct the masked tokens conditioned on those patches.

Mask 60 % of the text tokens → positions {t₂, t₄, t₆, t₈} become the mask token.

Compute weights: $w_i$ = 1 / max(0.6, 0.05) = 1.67 for each masked position.

Run the model to obtain predicted token distributions for the four masked slots.

Calculate cross‑entropy for each slot, multiply by 1.67, sum, and divide by the total weight (4 × 1.67) to obtain $L_j$.

Average $L_j$ across all ranks (here only one rank) to get `L_pretrain`.

Because 60 % of the text is hidden, the model must rely on the visible image patch to resolve the ambiguity, demonstrating the intended cross‑modal learning signal.

How does this pretraining objective differ from BERT’s masked‑language modeling?

BERT uses a fixed 15 % mask rate and a mixture of mask, random, and unchanged tokens; NeoMME draws a segment‑wise corruption rate (up to 100 % for multimodal data), always replaces selected tokens with the mask token, and conditions prediction on visible image patches, thereby encouraging genuine visual grounding.

Section 4.4 lists the hardware, batch, and optimizer choices that enable training at the 260 M and 800 M scales.

Run on AWS p5.48xlarge instances (2 nodes for 260 M, 4 nodes for 800 M) with 16 or 32 H100 GPUs per node.

Pack 1,048,576 tokens per global step and train for 500,000 steps (≈524 B tokens total).

Use NorMuon for matrix‑valued parameters (attention, MLP) and MasterAdamW for embeddings, both with bfloat16 model weights and full‑precision master copies.

Apply a warmup–stable–decay schedule: linear warmup over 300 updates, hold peak learning rate (0.012 for NorMuon, 0.0013 for AdamW on 260 M), then linearly decay to 1 % of the peak over the final 10 % of training.

Synchronize gradients globally while sharding optimizer updates for same‑shaped NorMuon matrices across GPUs.

**Table 3.** Pretraining hardware, batch, and optimization settings.

Section 4.5 investigates how masking severity influences reliance on image patches, reporting a positive image‑gain metric that grows with higher corruption.

**Figure 6.** Pretraining loss and image gain at 90% masking for both model sizes. Lines show raw and smoothed measurements, with dashed lines marking observed loss transitions.

Section 4.6 evaluates image‑conditioned generation: starting from visible patches and a fully masked text canvas, the model iteratively predicts tokens using the same denoising head, producing coherent captions and document text without any OCR fine‑tuning.

Retrieval Objectives and Benchmarks

NeoMME‑Retriever achieves large nDCG gains with both late‑interaction and dense heads.

NeoMME‑260M improves ViDoRe v3 nDCG@10 by +26.1 points over the strongest sub‑300 M baseline.

Table 5 shows NeoMME‑260M at 0.523 nDCG@10 versus the next best sub‑300 M model at 0.262.

Each token of a query and a document is encoded separately, then the best matching token pairs are summed to produce a relevance score.

How does Late‑Interaction differ from a dense bi‑encoder?

Dense bi‑encoders collapse each input to a single vector before scoring, losing token‑level detail. Late‑interaction keeps a vector per token, allowing the model to match specific words or image patches, which yields higher nDCG especially on multimodal documents.

A single pooled vector represents each document, enabling fast approximate nearest‑neighbor search.

Why would one still use a dense head if Late‑Interaction is more accurate?

Dense vectors can be indexed with highly optimized ANN libraries, giving orders‑of‑magnitude faster retrieval on very large corpora. A common two‑stage pipeline first retrieves a shortlist with the dense head, then reranks with the more expensive late‑interaction head.

A suite of visual‑document retrieval tasks that evaluate how well models retrieve PDF page images given textual queries.

What does “MeanMaxSim” mean in the context of ViDoRe?

MeanMaxSim first takes the maximum dot product between each query token and any document token (MaxSim), then averages these maxima over all query tokens. This normalizes the score by query length, preventing longer queries from automatically receiving higher scores.

Retrieval Performance

Key storage‑reduction and speedup results for NeoMME retrieval.

Hierarchical token pooling at pool factor 7 reduces storage by about 7× while retaining >99 % of ViDoRe v3 nDCG@10.

Both NeoMME‑260M and NeoMME‑800M achieve this reduction.

**Figure 11.** Hierarchical token pooling on ViDoRe v3: impact of the pool factor on retained retrieval quality for both NeoMME-Retriever models. (a) Quality by pool factor for both model sizes. (b) Quality versus storage for both model sizes.

**Table 7.** Int8 document representations reduce packed storage by 3.9x while retaining at least 99.96% of float32 nDCG@10, and binary documents reduce storage by 32x. Arrows give the change from the float32 baseline in nDCG@10 points (1 point = 0.01). All results use the evaluation protocol in subsection 5.6.

**Figure 14.** Similarity map for the query token hour, overlaid on a document from the ViDoRe syntheticDocQA energy test set.

Generalization to Other Tasks

NeoMME adapts to language, document, and image tasks despite a single compact backbone.

NeoMME attains a 75.3 mean score on the 17‑task multilingual benchmark.

Within 4 points of the 79.3 reported by the dedicated LFM2.5‑Encoder‑230M text model.

NeoMME exceeds the LFM2.5‑Encoder on several language tasks, achieving higher scores on PAWS‑X, MASSIVE Intent, SeaHorse, MRPC, and WSC, while its 46.4 Matthews correlation on CoLA highlights a weakness in acceptability judgments. The same backbone also reaches 85.5 ± 0.2 accuracy on the 8K‑context LEDGAR task and 89.3 entity F1 in a one‑seed CoNLL‑2003 experiment, demonstrating strong transfer to long‑document classification and token labeling.

For visual transfer, NeoMME’s pretraining does not include a pixel‑reconstruction or image‑level contrastive objective, so frozen probes on ten natural‑image classification tasks average only 13.2 % accuracy. Fine‑tuning, however, raises performance to 77.1 ± 0.3 on Food101, 63.9 ± 0.2 on Oxford Pets, and 46.8 ± 6.3 on Stanford Cars. Document‑image adaptation is stronger: a frozen first‑token probe on RVL‑CDIP attains 51.6 ± 0.3 % accuracy, and fine‑tuning on just 6 k examples (under 2 % of the training set) boosts accuracy to 81.5 ± 0.6 %.

Limitations and Caveats

NeoMME trades the usual multi‑encoder setup for a single‑tower Transformer that jointly encodes text and images.

This work was carried out under limited time and compute, so the reported NeoMME results should be viewed as a preliminary exploration.

First, NeoMME’s pretraining scale is modest: it consumes roughly 524 B tokens (≈290 B text) whereas ModernBERT trains on about 2 T text tokens, i.e., roughly seven times more data.

Longer runs with more text, images, and training steps could reveal how the model scales under larger data regimes.

Second, the visual objective is absent: NeoMME predicts masked text while leaving image patches untouched, providing no direct supervision for image content.

This may explain the weak frozen‑image performance reported in Section 6; controlled studies of image‑token prediction, pixel reconstruction, or image‑level contrastive losses would help isolate the effect.

Third, retrieval supervision is limited: NeoMME sees about 430 K pure‑text queries and 850 K image‑query examples, far fewer than the 660 M contrastive query–document pairs and 16 M hard‑negatives used by the SOTA mLateOn system.

Scaling up the retrieval training set could clarify how much of the BEIR gap stems from data volume versus architectural or objective choices.

Fourth, NeoMME‑Retriever was trained on a mixed‑modality corpus that merges text chunks and page‑image data into a single embedding space, yet we did not evaluate it on established mixed‑modality benchmarks such as UniDoc‑Bench or MixBench.

Future work should benchmark NeoMME‑Retriever on these datasets and extend them to multilingual page‑image collections.

Fifth, there is a scarcity of large multilingual visual‑document datasets; real‑world PDFs are hard to translate while preserving layout.

Generating documents from code, translating their source content, and rendering pages in many languages could address this gap.

Sixth, language coverage for visual retrieval is narrower than for text retrieval, limiting the multilingual robustness of the system.

Synthetic document generation could broaden coverage, especially for low‑resource scripts.

Seventh, the current ablations do not isolate the contributions of architecture, model scaling, data composition, or training regimes.

Controlled experiments under matched compute would pinpoint the drivers of pretraining, retrieval, and transfer performance.

Eighth, we did not employ distillation; both pretraining and retrieval fine‑tuning lack teacher supervision.

Distilling the 800 M model into the 260 M variant at both stages could reveal efficiency‑accuracy trade‑offs.

Ninth, training data may embed social, geographic, topical, and language biases, which were not systematically evaluated.

A dedicated bias evaluation would enable comparison of filtering and balancing strategies.

Tenth, safety concerns arise because iterative masked‑token sampling can generate harmful or private content, even though NeoMME is not a conversational model.

Future work should measure such behavior across text and image inputs and under different decoding methods.

Hard-Negative Sampling

Ablation studies on hard‑negative mining, memory‑efficient training, and tokenizer efficiency.

This appendix reports ablation experiments that isolate the contribution of hard‑negative mining, memory‑efficient training tricks, and tokenizer design.

The retriever first learns with only in‑batch negatives, then uses its own checkpoint to mine a fixed‑size candidate window that supplies harder negatives for a second fine‑tuning pass.

How does this hard‑negative mining differ from standard in‑batch negatives?

Standard in‑batch negatives are drawn randomly from the current mini‑batch, so they are often easy. Here the second stage explicitly selects the hardest candidates from a pre‑mined window, forcing the model to discriminate finer distinctions.

GradCache decouples representation computation from the contrastive loss, allowing the encoder to process large candidate pools in small activation chunks, while LIK replaces the full token‑similarity tensor with an online max reduction.

NeoMME emits substantially more tokens than competing tokenizers, indicating lower compression efficiency.

Across all 204 FLORES‑200 language‑script pairs, NeoMME emits 65.29 % more tokens than mmBERT and 15.28 % more than EuroBERT in aggregate.

Qualitative image‑conditioned generation examples illustrate that NeoMME‑260M can produce coherent captions, though the outputs are not yet benchmark‑level.

Retrieval Ablations

Ablation studies quantify each component’s impact on retrieval performance.

This appendix reports the ablations that answer “does removing this component hurt?” for every major design choice.

Matryoshka training forces a single model to emit dense vectors at several preset widths, like nesting smaller dolls inside a larger one.

How does Matryoshka differ from simply training several independent models of different sizes?

Matryoshka shares a single set of parameters across all widths, so the storage cost is that of one model and the training time is comparable to a single run. Independent models would require separate checkpoints and duplicate computation for each size.

**Table 19.** Visual document retrieval performance on the ViDoRe benchmarks for all compared models. A dash marks a result that was not reported. Emb. dim. is the per-token vector width for late-interaction models and the single-vector width for dense models.

**Table.** BEIR-15 performance results.

**Table 21.** Retrieval quality and uncompressed square-page representation size across image resolutions. All results use the evaluation protocol in subsection 5.6.

**Table 22.** ViDoRe v3 domains. Table 22 reports nDCG@10 for every task and a subset of the compared models. The NeoMME-800M row has a mean score of 0.5560.

**Table 23.** Selected task-level NeoMME-260M$^{15}$ late-interaction results. All results use the evaluation protocol in subsection 5.6.

Joint training of late‑interaction and dense heads improves late‑interaction nDCG@10 by 1.38 points on ViDoRe v3.

Table 25 shows the dual‑head architecture achieving 0.5088 → 0.5226 (↑ +1.38) compared with the late‑interaction‑only baseline.

Joint training harms dense‑head performance on ViDoRe v3, dropping nDCG@10 by 1.85 points.

Table 25 records dense‑head nDCG@10 falling from 0.3906 to 0.3055 (↓ −1.85) under the dual‑head setup.

**Table 25.** Retrieval-head objective ablation for NeoMME-260M. Arrows show nDCG@10-point changes from the corresponding single-objective run. All results use the same evaluation protocol.

Efficiency Analysis

Retrieval efficiency benchmarks highlight NeoMME’s speed and memory advantages.

NeoMME‑260M delivers a 2.4× speedup in document‑encoding throughput on H100 compared to the next‑best model.

Table 27 shows NeoMME‑260M at 76.8 pages/s versus 53.3 pages/s for the closest competitor, ColModernVBERT.

All other baselines fall well behind NeoMME‑260M. On H100, Vultron Flash reaches 35.2 pages/s and Qwen3‑VL‑Embedding‑2B only 15.5 pages/s. Even on the less powerful L40S and M5 Pro GPUs the NeoMME‑260M advantage persists, with 51.3 pages/s and 3.2 pages/s respectively.

NeoMME‑260M achieves a 2.5× lower query‑encoding latency on M5 Pro than competing visual‑language retrievers.

Latency measurements report 15.9 ms for NeoMME‑260M versus roughly 39 ms for the nearest baseline.

Across devices the latency gap remains pronounced: on the L40S NeoMME‑260M records 21.0 ms, while the slowest baseline exceeds 50 ms. On a CPU host the model still finishes in 78.3 ms, well below many GPU‑only pipelines.

Late‑Interaction Kernels add negligible overhead, boosting NeoMME‑260M training throughput by 1.6%.

Table 28 reports 383 614 tokens/s with LIK versus 377 439 tokens/s without.

LIK reduces peak memory consumption by 71% for 4 k‑token documents.

Table 28 shows memory dropping from 672 MB (Naive) to 193 MB (LIK).

Table 29 further demonstrates LIK’s practical impact: ColQwen2‑5 sees a 130× drop in peak memory (7.81 GiB → 61 MiB) and a 2× increase in maximum batch size, while PyLate gains 7 % memory savings and a 1.25× speed‑up in step time.

Retrieval Case Demonstrations

Retrieval case examples illustrate NeoMME‑Retriever 260M performance on ViDoRe v3 tasks.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers