Scalable Visual Pretraining for Language Intelligence

Yiming Zhang, Zhonghan Zhao, Wenwei Zhang, Haiteng Zhao, Tianyang Lin, Yunhua Zhou, Demin Song, Kuikun Liu, Haochen Ye, Haian Huang, Yuzhe Gu, Haijun Lv, Qipeng Guo, Bin Liu, Gaoang Wang, Kai Chen

Visual Pretraining (VP) improves scientific reasoning by learning directly from raw document images instead of extracted text.

Does pretraining language models on raw document images (visual pretraining) improve scientific reasoning compared to training on text extracted from those same documents?

Foundation models are typically trained on text-only corpora, which forces scientific documents—rich with figures, equations, and complex layouts—to be converted into plain text. This textualization process is inherently lossy, discarding the geometric and structural cues essential for scientific reasoning. Visual Pretraining (VP) addresses this by training the model to predict document patches in latent space directly from raw page images. It treats these pages as ordered visual sequences, allowing the model to internalize document-native relations without requiring image-text pairing supervision. Across multiple backbones, VP consistently outperforms text-only pretraining on scientific reasoning benchmarks while consuming only 25% of the token budget.

Paper Primer

The core move is an autoregressive objective that treats rendered document pages as sequences of visual latents. The model uses a frozen vision tower to extract foreground patches, which are then fed into the LLM to predict the next visual latent in raster-scan order, effectively integrating visual structure into the model's predictive process.

VP significantly improves scientific reasoning performance over text-only baselines.

On the GPQA Diamond benchmark, Qwen-3.5 improved from 81.56 to 90.21, while Llama-3.2 Vision improved from 50.60 to 60.64. Up to 3.22 points on GPQA and 5.4 points on ChartQAPro.

VP achieves superior scaling efficiency compared to text-only pretraining.

VP processes approximately 20B visual tokens compared to 80B text tokens for the same document corpus, while reaching higher downstream performance. 2.02× normalized gain on GPQA and 2.88× on AIME-25.

Why does this approach outperform text-only pretraining if the underlying document source is the same?

Text extraction (e.g., OCR or parsing) discards geometric constraints, symbolic topologies, and spatial layouts that are critical for scientific reasoning. VP preserves these native visual relations, allowing the model to learn from evidence that textualization weakens or destroys.

Does VP require specialized multimodal data or annotations to achieve these gains?

No. VP is an unsupervised framework that trains on raw, unlabeled document pages. It improves cross-modal alignment and visual reasoning as an emergent property of learning to predict the next visual latent in the document sequence.

Researchers should treat raw visual documents as a high-fidelity source of supervision for foundation models, as visual-latent prediction provides a more efficient and structurally aware alternative to traditional text-only pretraining for knowledge-dense domains.

The Case for Visual Pretraining

We expose the loss of visual information in text‑only pretraining and propose Visual Pretraining.

Large foundation models are built on massive text corpora, yet this approach assumes that every piece of knowledge can be captured as a linear sequence of tokens. In reality, scientific documents embed critical information in figures, equations, and page layouts that text extraction inevitably discards. Visual Pretraining (VP) challenges this assumption by training models directly on raw document images, preserving the visual cues that text‑only pipelines lose.

TP trains models on text extracted from documents, treating the content as a flat token stream.

**Figure 1.** Matched text and visual pretraining from the same scientific-document corpus. In the TP pathway, PDF pages are parsed into text, tokenized, and trained with text-token prediction, which can discard or distort page-level structure such as diagrams, equations, tables, and layout. In the VP pathway, the same pages are rendered as images, filtered into foreground visual patches, and trained with next visual latent prediction in a frozen visual-feature space. The t-SNE [31] projections on the right illustrate the representation change quantified in Table 2: VP brings matched visual and textual document embeddings closer in the shared representation space.

The core information gap lies between raw document pages and the extracted text that current pretraining discards.

The Visual Pretraining Pipeline

How Visual Pretraining turns raw pages into a compact, learnable visual stream.

Text‑only pretraining discards the rich layout and diagram information that scientific papers rely on, yet feeding an entire page as dense image tokens would explode the sequence length. The method below solves that tension by keeping only the informative foreground patches and training the model to predict the next visual latent just like it predicts the next word.

VP augments standard text pretraining with a frozen vision encoder that extracts visual latents from raw document pages, filters out blank regions, and trains the language model to autoregressively predict the next foreground latent.

How does VP differ from simply appending raw image patches to the language model?

VP first discards blank background patches, dramatically shrinking the visual token count, and then predicts continuous visual latents with a contrastive loss rather than treating each patch as an independent discrete token. This keeps the sequence length manageable and leverages the frozen vision encoder’s pretrained features.

Instead of feeding every image patch, we keep only patches that contain content (non‑blank) and order them left‑to‑right, so the visual stream stays short while preserving the page’s spatial layout.

Retain foreground patches: $u_1 = z_2$, $u_2 = z_4$.

Raster order of the retained patches is $(\text{top-right},\ \text{bottom-right})$, so positional indices become $[1,2]$.

Project each $u_t$ with the visual projection matrix $W_{\text{proj}}$ to obtain LLM‑compatible embeddings.

The resulting visual token sequence length $L=2$ replaces the original $N=4$ tokens, cutting the visual token budget by 50%.

This toy illustrates how a simple variance filter can halve the token count while still preserving the right‑hand side of the page, which often contains figures or equations.

Why not just downsample the whole page instead of masking?

Downsampling reduces resolution everywhere, discarding fine‑grained details in figures and equations. Masking preserves full resolution for informative regions while still achieving a comparable token budget.

The model treats each foreground visual token like a word: given the LLM hidden state at position $t$, it predicts the next visual latent $z_{t+1}$ under a causal mask.

Compute $\hat{z}_2 = W_{\text{out}}\,h_1$ (linear map back to visual space).

Calculate cosine similarity between $\hat{z}_2$ and the true $z_4$ and between $\hat{z}_2$ and all other batch latents.

Apply softmax with temperature $\tau$ to obtain probabilities $p_{ij}$.

Loss contribution for this position is $-\log p_{ii}$, encouraging $\hat{z}_2$ to be closest to $z_4$.

The loss operates on the similarity of continuous vectors, not on a discrete token mismatch, which makes learning efficient even with a tiny visual vocabulary.

Why use a contrastive (InfoNCE) loss instead of an $L_2$ regression loss?

Contrastive loss scales with the number of in‑batch negatives, providing a richer supervisory signal that pushes the correct latent to be uniquely closest, whereas $L_2$ would only penalize distance without encouraging discrimination from other patches.

Render each PDF page as an image $\mathcal{I}$.

Pass $\mathcal{I}$ through the frozen vision tower $E_v$ to obtain dense features $\mathcal{Z}$.

Compute a foreground mask $m_i$ using patch variance and luminance; keep only $m_i=1$ patches.

Order retained patches in raster‑scan order to form $\mathcal{U}$ and reassign positional indices.

Project $\mathcal{U}$ into the LLM embedding space with a learned linear layer.

Concatenate visual embeddings with text tokens and feed the sequence to the LLM under a causal mask.

From the LLM hidden state at each visual position, predict the next visual latent $\hat{z}_{t+1}$.

Compute the contrastive loss $\mathcal{L}_{VP}$ using in‑batch negatives.

Combine $\mathcal{L}_{VP}$ with the text cross‑entropy loss $\mathcal{L}_{CE}$ to obtain the total loss $\mathcal{L}$.

Back‑propagate gradients to update the LLM, the visual projection, and the prediction head; keep $E_v$ frozen.

**Figure 3.** The main VP setting balances visual-token budget and rendering resolution. Left, with the training context fixed for each forward pass, varying the visual token budget shows that the 1x setting (8,192 foreground visual tokens per batch) offers the best trade-off. Right, with the visual budget fixed at 8,192 tokens, lowering the maximum resolution preserves strong performance, indicating the efficiency of VP under compressed visual input.

Performance and Scaling Findings

VP beats TP on scientific reasoning while using far fewer tokens.

Visual Pretraining (VP) consistently outperforms Text Pretraining (TP) on scientific reasoning benchmarks under matched corpora.

Table 1 shows VP improving GPQA by up to 3.22 points and MMLU‑Pro by up to 2.1 points across four backbones.

**Table 1.** Effectiveness: VP improves scientific language reasoning

**Figure 2.** Visual pretraining scales with retained PDF visual tokens and benefits most from structure-heavy pages. (a), VP reaches a more favorable SFT trajectory after comparable CPT loss. (b), Increasing the retained visual-token budget consistently raises downstream gains, normalized by the TP-over-base improvement. (c), VP's advantage is largest on examples with high visual-structure density, where figures, equations, tables, and layout carry more of the evidence. (d), Better next-token visual-feature prediction is associated with larger multimodal gains over TP, linking visual-space modeling quality to downstream transfer.

**Table 1.** (a) Cross-modal alignment. (b) Multimodal benchmark performance.

**Figure 4.** Visual-token reasoning attends to the same semantic evidence as text reasoning. We compare foundation models with TP or VP on the same math problem under text and image inputs. Top, textual reasoning: the problem and chain-of-thought solution are given as plain text, and attention is measured from the final answer sentence to previous tokens. Bottom, visual reasoning: the same problem is rendered as an image and fed through the visual pathway, and attention is measured from visual tokens in the answer region to retained page patches. Both views concentrate on the semantic regions highlighted in red, including the question constraint and intermediate computation steps.

VP consistently outperforms TP on scientific reasoning.

Mathematical Pipeline Details

The pipeline builds a compact visual token sequence and trains the LLM to predict the next visual feature.

The visual‑pretraining pipeline turns a rendered page into a short, content‑rich token stream and teaches the language model to predict the next visual token, mirroring next‑word prediction in text.

Instead of feeding every image patch to the language model, we keep only patches that contain visual content and stitch them into a compact, ordered sequence.

Patch 1: $\sigma$²₁ = 0.005, ℓ₁ = 0.95 → b₁ = 1 (background, near‑white).

Patch 2: $\sigma$²₂ = 0.12, ℓ₂ = 0.45 → b₂ = 0 (foreground).

Patch 3: $\sigma$²₃ = 0.003, ℓ₃ = 0.08 → b₃ = 1 (background, near‑black).

Patch 4: $\sigma$²₄ = 0.20, ℓ₄ = 0.60 → b₄ = 0 (foreground).

After merging (each patch is its own merged token), compute mⱼ = max(1 − bᵢ): m₁ = 0, m₂ = 1, m₃ = 0, m₄ = 1.

Raster ordering keeps tokens 2 and 4, yielding the sparse sequence (u₁, u₂) with L = 2 ≪ N = 4.

This foreground‑filtering discards empty margins while preserving the visual order, shrinking the token count dramatically without losing layout information.

How does this foreground‑filtering differ from simply cropping out blank margins?

Cropping removes whole regions and destroys the raster order, whereas the mask works at patch granularity, keeping the relative positions of all retained content. The max‑pooling step also ensures that a merged token survives if any of its constituent patches carries information, so layout continuity is preserved.

Implications for Scientific Intelligence

Visual Pretraining lets models ingest raw scientific pages, preserving layout that text alone loses.

Large language models learn from text‑only corpora, yet scientific knowledge is often expressed through equations, figures, tables, and layout. Converting these documents to plain text discards the visual structure that can be crucial for reasoning.

Visual Pretraining (VP) addresses this gap by training the shared autoregressive backbone on foreground visual tokens ordered by their page positions, using a causal next‑latent prediction objective. This encourages the model to capture document‑native relations such as equation topology and figure‑text correspondence.

Empirically, VP improves scientific reasoning over matched text pretraining across backbones, with the largest gains on visually dense examples where figures, tables, and equations dominate the evidence. Moreover, cross‑modal alignment improves without any paired image‑text supervision, indicating that the visual objective reshapes the shared representation space.

Unlike most multimodal pretraining approaches that rely on image‑text pairs, captioning losses, or OCR supervision, VP treats rendered scientific pages as unlabeled visual sequences and learns via next‑latent prediction without dense annotations. OCR‑free models such as Donut and Pix2Struct also operate directly on document images, but they are optimized for parsing or image‑to‑text generation rather than joint visual‑language reasoning.

Our study has three main limitations: VP is not fully independent of language pretraining; coordination between visual‑latent and text decoding remains an open problem; and the experiments focus on high‑knowledge‑density scientific PDFs, leaving open the transferability to natural images or video. Future work should explore loss scheduling, foreground‑aware token selection, and scalable visual‑only pretraining streams with lightweight text alignment.

Generative Evaluation and Decoder Variants

Evaluating the generative decoder’s impact on cross‑modal alignment.

We assess the generative decoder by measuring how well visual pretraining aligns images with text using a conditional‑perplexity retrieval protocol.

Adding the generative decoder (Visual Pretraining) raises Recall@1 from 64.0 % to 99.0 %.

Table 3 shows the VP (Ours) column achieving 99.0 % versus 64.0 % for Text Pretraining.

Visual pretraining reduces off‑diagonal score variance from 0.168 to 0.043, eliminating hub‑attractor effects.

Analysis of the 100 × 100 score matrix reports the variance drop after applying the generative decoder.

**Table 3.** Generative image-to-text retrieval with conditional perplexity. Each image is matched against 100 candidate texts using the PMI-style score defined in Section B; higher Recall@K and MRR indicate better cross-modal compatibility.

**Figure 5.** Architecture comparison between the generative-decoder variant and decoder-free visual pretraining. (Left) The generative-decoder variant (VP with Decoder) adds a MAR decoder and a frozen VAE decoder to reconstruct pages from LLM-refined latents, providing additional pixel-level supervision at the cost of extra parameters and computation. (Right) Our decoder-free formulation (VP w/o Decoder) predicts foreground patch latents autoregressively without pixel reconstruction.

Implementation and Evaluation Configs

Implementation details cover data, model, training, cost, evaluation, and cross‑modal alignment analysis.

The training corpus combines three components: a standard text corpus for continued pretraining, a scientific‑PDF corpus for matched visual‑text pretraining, and instruction‑tuning SFT data. For the matched text‑pretraining baseline, each PDF page is converted to text with MinerU2.5 [32]; for Visual Pretraining (VP) the same pages are rendered as images and used as unlabeled visual supervision.

From the same PDF pages the pipeline yields roughly 20 B retained visual tokens for VP and about 80 B parsed text tokens for TP, ensuring a matched construction that varies only in representation. VP image records are stored as JSONL entries with page‑image paths and optional precomputed latent‑feature paths; each page is loaded, converted to RGB, resized to a fixed square resolution, and normalized before processing.

Blank regions are removed by a foreground mask computed from patch variance and luminance (variance < 0.02, high‑luminance > 0.95, low‑luminance < 0.15). A merged visual token is kept if any of its constituent raw patches is foreground.

The VP model shares the LLM backbone with the text‑pretraining baseline. In Qwen 3.5 the visual pathway is a frozen ViT‑style encoder (27 layers, hidden size 1152, 16 heads, patch size 16, spatial merge 2). A visual projector maps frozen visual features into the LLM hidden space, and a two‑layer MLP prediction head maps LLM hidden states back to the frozen visual‑feature space for next‑visual‑latent prediction.

During continued pretraining, one VP batch is attached to every text‑training step, yielding a fixed mixing ratio. The text branch uses standard autoregressive cross‑entropy; the visual branch uses the next‑visual‑latent contrastive loss (InfoNCE with temperature $\tau = 0.07$) weighted by 0.1. The LLM backbone, visual projector, and prediction head are updated jointly, while the vision tower remains frozen.

Multiple sparse foreground sequences are packed into fixed‑length contexts, and sequence boundaries are tracked to prevent attention across different document pages. For Qwen 3.5 runs, text sequences are hard‑packed to 32,768 tokens and trained for 12,000 steps with global batch size 1024, AdamW $3\times10^{-5}$ learning rate, weight decay 0.1, a 1,000‑step warmup, and linear decay to $10^{-6}$. The VP head uses AdamW $4\times10^{-5}$ with the same betas and weight decay, and the visual VP context is capped at 8,192 retained foreground tokens.

Across the main Qwen and Llama VP/TP continued‑pretraining runs, each run typically required about 1.5–3 days of wall‑clock time on a distributed cluster with 128 accelerators, depending on backbone, token budget, context length, and modality. The flagship Qwen 3.5‑35B‑A3B run took roughly 36 hours, followed by an additional 10 hours for the SFT stage.

All models are evaluated zero‑shot after SFT initialized from CPT. Text‑only reasoning benchmarks (Table 1) use chain‑of‑thought prompting (“think step by step”) and report pass@8 for GPQA, average score over 32 runs for AIME‑25, and pass@1 for MMLU‑Pro and HLE. Multimodal benchmarks (Table 2) use a direct‑answer template without explicit reasoning guidance and report pass@1.

Cross‑modal alignment analysis constructs $N = 100$ matched document image–text pairs from a held‑out scientific‑PDF set. Each image is processed by the visual pathway and each text by the shared tokenizer; hidden states from the last Transformer layer are mean‑pooled over foreground visual tokens and non‑padding text tokens, yielding paired embeddings $\{(v_i, t_i)\}_{i=1}^{N}$ before and after VP.

Across the three metric families, VP yields substantial gains: centroid separation drops by 60 % (1.665 → 0.661), pairwise cosine similarity rises from 0.631 to 0.907, linear CKA increases from 0.657 to 0.745, and $k$‑NN overlap improves at all $k$ (e.g., $k=1$: 0.140 → 0.310). These results show that visual pretraining reshapes the shared representation space, enhancing compatibility without explicit image–text pair supervision.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers