On the Design Fundamentals of Pixel Text Representation Learning

Chaohao Yuan, Ruifeng Yuan, Zhuoxu Huang, Yu Rong, Hong Cheng, Hou Pong Chan, Chenghao Xiao

A unified vision-only encoder that learns robust, multilingual text representations through on-the-fly rendering and multimodal grounding.

How can we design vision encoders to read and retrieve text directly from pixels without the resolution and grounding bottlenecks of traditional CLIP-style models?

Pixel-based text encoders often struggle to generalize because they rely on fixed-resolution training and synthetic data that lacks real-world visual grounding. PIXEL LINGUIST II addresses this by using a native-resolution architecture and a two-stage curriculum that combines unsupervised multilingual pretraining with joint contrastive grounding on natural image-text pairs. This approach sets new state-of-the-art results on Visual STS and document retrieval benchmarks while remaining robust under 80% visual token compression.

Paper Primer

The model treats text as a visual modality, rendering text on-the-fly with diverse fonts, backgrounds, and spatial perturbations. This layout-aware augmentation forces the encoder to prioritize semantic structure over superficial pixel-level shortcuts.

PIXEL LINGUIST II achieves state-of-the-art performance on Visual Semantic Textual Similarity (Visual STS) and Visual Document Retrieval (ViDoRe).

The model outperforms the strongest SigLIP variants by approximately 15% in Spearman correlation on cross-lingual tasks and 16% on multilingual tasks. It achieves these gains while using roughly 1/7 the parameters and 1/87 the training examples of comparable large-scale vision encoders.

The model maintains high semantic fidelity under aggressive token compression.

On ViDoRe benchmarks, the model outperforms uncompressed CLIP baselines even when 80% of visual tokens are discarded.

Why is multimodal grounding with natural images necessary if the goal is primarily text understanding?

Purely synthetic training leads to representation collapse; natural image-text pairs act as a foundational regularizer that grounds textual semantics in real-world visual contexts and diverse layout structures.

How does the model handle high-resolution documents without training on them directly?

The model uses variable natural-image resolutions and rendered font sizes as "spatial proxies," which implicitly teach the encoder to handle multiple spatial frequencies and generalize to dense documents at inference.

Researchers can now achieve robust document-level understanding using a vision-only encoder by prioritizing layout diversity and multimodal grounding over simple data scaling.

Introduction and Motivation

We expose why pixel‑text encoders fail and outline the design principles needed for robust visual text learning.

Pixel‑text encoders today falter on four fronts: they are tied to a fixed‑resolution pretraining regime, they learn cheap visual shortcuts instead of semantics, they lack strong grounding to real‑world visuals, and they struggle with multilingual visual text understanding.

Systematic ablations reveal four design pillars that overcome these bottlenecks: (1) variable image resolutions and rendered font sizes provide spatial proxies for high‑resolution document generalization; (2) natural image‑text pairs are essential for grounding and prevent collapse to text‑only representations; (3) layout‑aware rendering thwarts pixel‑level shortcuts; and (4) a two‑stage multilingual curriculum enables effective cross‑lingual alignment.

Pixel‑text encoders hit a bottleneck because they bind textual meaning to a rigid pixel grid, which encourages the model to exploit low‑level visual cues rather than learn robust, resolution‑agnostic semantics.

Building on these insights, we introduce PIXEL LINGUIST II, a native‑resolution vision encoder trained with on‑the‑fly rendering, unified contrastive grounding, and a multilingual curriculum spanning 280 M examples.

PIXEL LINGUIST II sets new state‑of‑the‑art performance on English, cross‑lingual, and multilingual Visual STS benchmarks as well as the ViDoRe visual document retrieval task, and it stays robust even when 80 % of visual tokens are compressed.

The shift from tokenizer‑based to pixel‑based text encoding eliminates the need for discrete token vocabularies, enabling a unified visual‑language model that learns semantics directly from raw image data.

Design Fundamentals

We expose why fixed‑resolution pretraining hurts real‑world text, then present the native‑resolution engine that solves it.

Pixel‑text encoders inherit a fixed‑resolution bottleneck: they train on 224 × 224 canvases but must later ingest 4 K documents, and they often overfit to static visual templates.

Instead of shrinking every document to a tiny thumbnail, the engine renders each text instance at its original pixel density and feeds the model a canvas that preserves the true spatial frequencies.

How does this differ from ordinary multi‑scale data augmentation?

Standard augmentation randomly rescales the whole image, which changes both content and layout simultaneously. The native‑resolution engine keeps the textual content fixed while only varying the font size and aspect‑ratio of the surrounding image, explicitly teaching the model to separate semantic structure from raw pixel scale.

Step 1: Resize the photograph – longest side → 224 px, shorter side → 150 px, preserving aspect ratio.

Step 2: Sample font size = 18 pt for the word “Hello”. Render the word centered on the canvas.

Step 3: Feed the 224 × 224 image (photo + rendered text) to the encoder; the encoder sees both the 150‑pixel vertical variation and the 18‑pt glyph scale.

Step 4: During a later inference on a 4 K PDF, the encoder receives a 3840 × 5120 canvas; because it has learned to map 224‑pixel variations to scale‑invariant features, it can extrapolate to the larger resolution.

Randomizing font size forces the model to learn a representation that is robust to pixel‑level size changes, which directly mitigates the “resolution paradox” without ever training on full‑resolution documents.

**Table 1.** Deconstructing the Pixel Pretraining Pipeline. We evaluate spatial proxies and multimodal grounding using controlled, small-scale ablations on resolution-sensitive ViDoRe tasks. Performance is measured in nDCG@5.

**Table 2.** Data Curriculum Ablation on Visual STS (Spearman Correlation).

The synthetic rendered‑text curriculum supplies controlled layout diversity that forces the encoder to ignore superficial visual shortcuts, enabling it to learn true semantic structure that transfers to real‑world documents.

PIXEL LINGUIST II Architecture

We detail the PIXEL LINGUIST II architecture, rendering engine, curriculum, and training pipeline.

Pixel‑text encoders suffer from fixed‑resolution bottlenecks and visual shortcuts; we now instantiate PIXEL LINGUIST II to overcome these issues.

**Figure 1.** Overview of PIXEL LINGUIST II. We construct two types of training data: natural image–text pairs and text–text pairs. Textual inputs are rendered on-the-fly into images with diverse layouts using a text-to-image rendering engine. A native-resolution ViT encodes both natural images and rendered visual text within a unified pixel space. The model is trained by a contrastive learning objective.

Instead of forcing every image into a fixed grid, the encoder processes a variable‑size token map that matches the original document layout.

The encoder receives the 16 tokens as separate patches.

The $2\times2$ pooling merges each 2 × 2 block, producing 4 pooled tokens.

Each pooled token now aggregates information from four original patches, preserving the overall shape.

Pooling cuts the token count by 75 % without discarding the spatial pattern needed to read the word.

How does NaViT differ from a standard ViT that resizes inputs to a fixed grid?

Standard ViT interpolates every image to a preset resolution, which can blur small characters. NaViT keeps the original pixel layout, allowing the model to see each character at its native size before pooling.

We render text into images on the fly with diverse fonts and backgrounds, so the model never sees the same visual token twice.

Font Arial on white background yields a clean high‑contrast image.

Font Arial on stone texture adds texture noise, slightly lowering contrast.

Font Times on white background introduces a different serif style.

Font Times on stone texture combines serif style with texture noise.

Even with the same word, the four renderings differ enough that a model cannot rely on a single visual shortcut.

Why not pre‑render a large static dataset instead of rendering on the fly?

Static rendering would let the model memorize exact pixel patterns for each token, defeating the goal of learning true textual semantics from visual cues.

We train the encoder with both text‑text and image‑text pairs, aligning their embeddings so the model retains world knowledge while learning visual text semantics.

How does this contrastive grounding differ from standard CLIP training?

Standard CLIP pairs each image with a single caption; here we also include pure text‑text pairs, which inject additional semantic signal that pure image‑caption data lack.

We first teach the model broad multilingual knowledge, then fine‑tune it on high‑quality semantic pairs, using a two‑stage schedule.

Stage 1: The model sees the two documents, learns multilingual token patterns, and aligns the two image‑text pairs.

Stage 2: The model refines its embeddings using the two high‑quality text pairs while still seeing the two image‑text pairs.

Each stage runs for 2 passes over its data, totaling 8 training steps.

Separating broad pretraining from focused semantic fine‑tuning yields better multilingual visual text performance than a single monolithic pass.

Why not train on all four datasets simultaneously instead of staging?

Staging allows the model to first acquire general language structure without interference from noisy high‑quality pairs, then specialize; simultaneous training can cause conflicting gradients that slow convergence.

We scale training across 64 GPUs using DeepSpeed ZeRO 2, gathering embeddings to compute a large‑batch InfoNCE loss.

Training loop for PIXEL LINGUIST II using DDP and InfoNCE.

Why use InfoNCE loss instead of a simple cross‑entropy classification loss?

InfoNCE treats every other example in the batch as a negative, scaling efficiently with batch size and directly optimizing the embedding similarity needed for retrieval.

With these components combined, PIXEL LINGUIST II learns robust visual text representations that generalize across resolutions, languages, and visual contexts.

Evaluation Benchmarks

PIXEL LINGUIST II outperforms CLIP baselines on visual language tasks.

PIXEL LINGUIST II outperforms the strongest CLIP‑based baseline by 7.5 points on Visual STS.

Table 3 shows the best CLIP baseline averaging 75.30, while PIXEL LINGUIST II reaches 82.87.

Visual STS asks a vision model to judge how semantically similar two pieces of text are, after the texts are rendered as images.

VDR measures how well a model can retrieve the correct document image given a query image, assessing cross‑modal retrieval performance.

**Figure 2.** Visual STS task. Text pairs are rendered as images for models to quantify their semantic similarity.

The gap between PIXEL LINGUIST II and CLIP‑based baselines highlights the advantage of native‑resolution training for visual text understanding.

Downstream Performance

Downstream evaluation shows PIXEL LINGUIST II outperforms baselines on visual STS and VDR tasks.

Recall that the paper’s native‑resolution rendering curriculum lets a vision‑only encoder learn text‑in‑pixel representations without the fixed‑resolution bottleneck.

PIXEL LINGUIST II achieves a 2.75 % average relative improvement over Qwen2.5‑ViT on nine downstream MLLM benchmarks.

Table 8 reports higher scores for PIXEL LINGUIST II on every benchmark, yielding the quoted average gain.

**Table 4.** Encoder performance of PIXEL LINGUIST II on Visual Document Retrieval (VDR) Tasks using ViDoRe subsets (Faysse et al., 2025) in MIEB-lite benchmark (Xiao et al., 2025), compared with SOTA baseline encoder models.

**Table 8.** Performance comparison on downstream tasks under MLLM evaluation.

Across cross‑lingual and multilingual Visual STS (Tables 5 and 13), PIXEL LINGUIST II outperforms the strongest SigLIP variant by roughly 15‑16 % in Spearman correlation, confirming its multilingual robustness despite a vision‑only design.

Visual Token Compression

Ablation studies reveal how visual token compression preserves semantics while shrinking context size.

We probe how aggressively visual token compression impacts semantic fidelity across benchmarks.

Compressing the visual token sequence reduces the number of image patches the model processes, allowing longer textual contexts to fit within the fixed token budget.

**Figure 5.** Performance comparison of models trained with and without natural images.

Related Work

We wrap up PIXEL LINGUIST II’s achievements and note its remaining challenges.

Prior work spans dual‑encoder models, vision‑only encoders, and retrieval‑focused architectures, each addressing different facets of visual‑text alignment.

These models pair a frozen image encoder with a tokenizer‑based text encoder and train them jointly on image‑text pairs, establishing a strong cross‑modal similarity baseline.

Contrastive embedding derived from large‑scale generative pre‑training; it activates language‑level capabilities in a vision model via contrastive fine‑tuning.

Vision‑only model that learns to reconstruct rendered text images, enabling direct pixel‑text representation.

Vision‑only encoder that processes rendered text but lacks native‑resolution support, limiting its effectiveness for document‑centric tasks.

Vision‑centric late‑interaction retriever that refines candidate rankings using visual features after an initial textual retrieval.

Self‑supervised visual learning at web scale that matches language‑supervised methods on downstream benchmarks.

Unified vision encoder that learns text representations directly from pixels using native‑resolution rendering, layout‑aware curricula, and multilingual pre‑training.

PIXEL LINGUIST II demonstrates that careful curriculum design—spatial proxies, multimodal grounding, layout‑aware rendering, and multilingual curricula—yields strong performance on Visual STS and ViDoRe while tolerating heavy token compression.

**Table 6.** Summary of Font Families

Limitations remain: training data is smaller than billion‑scale CLIP/SigLIP corpora, performance on diagram‑heavy scientific documents lags behind, and our synthetic layout diversity does not capture all real‑world noise such as scans, blur, or handwriting.

Extended Results: Compression

Key performance tables for PIXEL LINGUIST II under token compression and across visual STS and VDR benchmarks.

PIXEL LINGUIST II sets new state‑of‑the‑art average scores on visual STS and VDR benchmarks.

Achieves 79.80 avg on English‑only Visual STS (Table 10) and 55.25 avg on VDR (Table 11), surpassing the strongest baselines.

Across nine downstream tasks, PIXEL LINGUIST II maintains or improves performance when visual tokens are reduced to 50 %–60 % of the original count. At 50 % keep, all tasks stay within 95 % of Qwen2.5‑ViT, with six surpassing it; at 60 % keep, seven tasks surpass the baseline. TextVQA is the most compression‑sensitive.

**Table 10.** PIXEL LINGUIST II Encoder Performance on Visual STS Tasks (English-only), which renders traditional STS tasks in NLP as image-only tasks, assessing vision models' text-on-image semantic understanding.

**Table 11.** PIXEL LINGUIST II Encoder performance on Visual Document Retrieval (VDR) Tasks, using ViDoRe benchmark, compared with SOTA baseline encoder models.

**Table 12.** PIXEL LINGUIST II Encoder Performance on Visual STS Tasks (Cross-lingual Tasks)

**Table 13.** PIXEL LINGUIST II Encoder Performance on Visual STS Tasks (Multilingual Tasks)

Extended Results: STS

Appendix D lists the best Visual STS scores and related ablations.

google/siglip-so400m-patch14-384 achieves the highest average Visual STS score.

Average (Avg) = 56.38, surpassing the next best model (laion/CLIP‑ViT‑bigG‑14‑laion2B‑39B‑b160k) at 43.19.

Across the Visual STS table, most encoders score between 25 and 40 Avg, with the larger CLIP‑ViT‑H and CLIP‑ViT‑bigG families reaching the high‑30s. The standout model, google/siglip‑so400m‑patch14‑384, exceeds the 50‑point threshold, indicating strong image‑only semantic alignment.

The ablation study shows that stripping away the textual pathway (vision‑only) harms every task dramatically, with drops ranging from ~19 points on DocVQA to over 53 points on Gov., confirming the necessity of multimodal cues.

Qwen2.5‑VIT’s raw scores (e.g., 0.94 on the second metric) are modest compared with the encoder models above, underscoring the gap between specialized vision‑language encoders and generic vision backbones.

The mid‑training versus finetuned comparison reveals consistent gains after finetuning: most metrics improve by 1–3 points, with the largest jump on Metric 2 (from 29.87 to 31.66), highlighting the benefit of task‑specific adaptation.

Table 11 reports VDR results on the ViDoRe benchmark; scores vary by language pair, with the highest average (21.40) achieved by openai/clip‑vit‑base‑patch32, while other models follow a similar range.

Extended Results: VDR

Key performance numbers for PIXEL LINGUIST II on Visual STS benchmarks.

PIXEL LINGUIST II sets a new state‑of‑the‑art average of 22.07 on Visual STS.

Table 13 reports an average of 22.07 across 12 languages, surpassing the previous best of 20.93.

By contrast, CLIP‑based baselines range from 10.54 (openai/clip‑vit‑base‑patch16) to 31.51 (laion/CLIP‑ViT‑bigG‑14‑laion2B‑39B‑b160k), while the best SigLIP model reaches 48.30. Even strong multilingual encoders such as EVA02‑CLIP‑bigE‑14 achieve only 41.43 on average.

Font and Rendering Details

Acknowledgments and detailed font resources used in the study.

We thank the anonymous reviewers and meta‑reviewer for their valuable feedback. This work was supported in part by a Multi‑year Research Grant from the University of Macau (Grant No. MYRG‑SRG2026‑00032‑FIC) and by the Research Grants Council of Hong Kong SAR, China (No. CUHK 14206625).

We utilize a diverse set of fonts to ensure the robustness of our rendering pipeline. The font library consists of 393 unique font files spanning multiple scripts and weights (100‑900). We allocate more font varieties to languages that dominate the dataset.

Chinese fonts include Noto Sans/Serif SC, Liu Jian Mao Cao, Long Cang, Ma Shan Zheng, Zhi Mang Xing, and the ZCOOL series. Japanese fonts include Noto Sans/Serif JP, DotGothic16, Kiwi Maru, Potta One, and Reggae. Korean fonts include Noto Sans/Serif KR, Gothic A1, Do Hyeon, Jua, Yeon Sung, and the Nanum series.

Arabic fonts include Noto Sans Arabic, Amiri, and Cairo. English fonts include Roboto, Open Sans, Lato, Montserrat, Nunito, Playfair Display, Poppins, Quicksand, Raleway, PT Sans, Ubuntu, Lora, Merriweather, Libre Baskerville, Anton, Josefin Sans, Caveat, Dancing Script, Pacifico, Shadows Into Light, Great Vibes, Allura, Cookie, Courgette, Lobster, Parisienne, Sacramento, Satisfy, Tangerine, Yellowtail. Code fonts include JetBrains Mono, Fira Code, Roboto Mono, Source Code Pro, Ubuntu Mono, Inconsolata, and Space Mono.

Other script families cover Armenian, Bengali, Devanagari, Ethiopic, Georgian, Gujarati, Gurmukhi, Hebrew, Kannada, Khmer, Lao, Malayalam, Math, Mongolian, Myanmar, Tamil, Telugu, Thai, and various symbols, all provided via Noto Sans variants.

Table 6 presents a concise summary of all font families grouped by script.

The following references are cited throughout the paper.

Training Configuration

This section details the rendering hyperparameters and two‑stage training schedule used for end‑to‑end MLLM experiments.

Table 7 lists the on‑the‑fly rendering hyperparameters that drive data generation. The canvas is fixed at $224 \times 224$ pixels, while font size is sampled uniformly from $U(16,28)$. Other stochastic augmentations include background texture probability $p_{\text{bg}}=0.5$, rotation in $[-15^\circ,+15^\circ]$, and Gaussian blur probability $p_{\text{blur}}=0.2$ with radius $[0.5,1.2]$.

Downstream evaluation uses a suite of nine benchmarks spanning OCR/document understanding (InfoVQA, DocVQA, ChartQA, TextVQA, LiveXivVQA) and general vision (AI2D, MMBenchEN, POPE, RealWorldQA, MMStar) via the LMMs‑Eval framework. Table 8 reports that PIXEL LINGUIST II consistently outperforms the baseline Qwen2.5‑ViT across all tasks.

We also explore visual token compression ranging from 10 % to 90 % of the original token budget. Table 9 shows that even at high compression rates the performance gap remains modest, confirming the robustness of the native‑resolution curriculum.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers