Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Junlin Han, Shengbang Tong, David Fan, Minghao Chen, Philip Torr, Filippos Kokkinos, Mike Lewis

Unified multimodal pretraining requires early joint training and modality-specific feed-forward networks to prevent vision laziness.

How do language, visual understanding, and visual generation tasks interact when trained together in a single unified model?

Unified multimodal models often treat vision as a secondary module to be aligned with a pre-existing language model, which bottlenecks performance and forces visual signals to conform to a rigid language space. The authors demonstrate that native multimodal capabilities emerge only when vision and language co-evolve from the start, using shared attention to foster synergy and modality-specific feed-forward networks to isolate capacity competition. This approach achieves strong generative performance using only 5% of the compute budget compared to standard late-alignment heuristics.

Paper Primer

The paper identifies "vision laziness" as a primary failure mode in late-alignment regimes, where a hardened language trunk prevents the vision pathway from fully committing to dense visual representations. By systematically varying the timing of visual integration and the architecture of parameter sharing, the authors isolate the "physics" of how modalities interact.

Early joint training is strictly superior to sequential or late-alignment strategies.

Controlled experiments across 1T tokens show that delaying visual integration consistently degrades performance on general, OCR, and vision-centric benchmarks. Late alignment triggers a "vision laziness" bias where the model relies on language priors rather than learning native visual faculties.

Decoupling feed-forward networks (FFN) while sharing attention layers maximizes cross-modal synergy.

Splitting FFNs mitigates capacity competition, while shared attention and normalization act as bridges for representational alignment. This architectural recipe improves language perplexity by -0.211 and lowers visual diffusion loss by -0.0168 compared to fully shared dense baselines.

Why does the paper argue against the common practice of pretraining a language model first?

Late alignment forces visual signals to conform to a pre-existing language manifold, which bottlenecks the model's ability to learn native visual representations and triggers "vision laziness."

Does visual generation help or hinder language and understanding tasks?

Visual generation acts as a neutral observer; it does not provide a strong catalyst for language or understanding, but it also does not severely interfere with them, unlike the strong positive transfer observed from language to vision.

Introduction to Multimodal Physics

We expose the under‑studied dynamics of early‑fusion multimodal pretraining.

Current multimodal research largely treats vision as an add‑on to language models, either fusing late or retrofitting pretrained encoders, leaving the fundamental interaction mechanisms opaque.

Instead of training a language model first and then grafting vision onto it, the model learns language, visual understanding, and visual generation together from the first step, allowing all three streams to shape each other’s representations.

The experimental gap is that prior work has not quantified *when* vision must be introduced nor *how* modality interactions scale with data complexity, leaving designers to rely on heuristics.

The shift from conditional vision to natively unified multimodal models is the key to unlocking true cross‑modal competence.

Experimental Framework

We detail the model training, visual tokenization variants, and evaluation benchmarks used in our experiments.

This section enumerates the concrete training pipeline, the four visual tokenization variants we explore, and the suite of benchmarks used to assess language, visual understanding, and generation performance.

Initialize a decoder‑only Transformer following the Llama‑3 design (16 layers, hidden size 2048, 1.5 B parameters) with SwiGLU, RoPE ($\theta$ = 500 000), pre‑RMSNorm, grouped‑query attention, QK‑norm, and FlashAttention.

Integrate vision via the Transfusion framework, which unifies next‑token prediction for text and continuous flow matching for image generation.

Configure modality‑specific split feed‑forward networks for text and image tokens.

Select one of four visual tokenization pipelines (RAE, Raw Pixels, CLIP + VAE, AR UniTok) to encode images.

Optimize with AdamW ($\beta$₁ = 0.9, $\beta$₂ = 0.95, weight decay = 0.1) and gradient clipping at 1.0.

Apply a loss weighting of 3.0 to the diffusion flow‑matching term relative to the text cross‑entropy term.

Schedule training with cosine decay and a linear warm‑up over the first 8 000 steps.

Train on 350 M image‑text pairs from Shutterstock‑Image together with DCLM text data, using a context length of 4096 tokens in bf16 precision under FSDP‑2.

Fine‑tune on VQA by running one epoch of supervised training on the Cambrian‑7M instruction dataset.

Fix random seed to 0 and set decoding temperature to 0 for all evaluations to guarantee reproducibility.

Encoder produces 256 tokens representing the image.

Flow‑matching loss drives the model to predict the clean latent $x_0$ for each token.

During generation, the predicted $x_0$ is converted on‑the‑fly to a velocity $v = (x_0 - x_t)/(1 - t)$ for each Euler ODE step.

The Representation Autoencoder reconstructs the image from the latent tokens, yielding a final 224 × 224 output.

The RAE pipeline reuses the same token grid for both understanding and generation, enabling a single model to share visual representations across tasks.

Assess language ability by averaging accuracy over 11 downstream benchmarks and reporting validation perplexity on DCLM and C4.

Measure visual understanding using 16 VQA benchmarks grouped into General, Knowledge, OCR & Chart, and Vision‑Centric categories.

For each VQA category, run the corresponding suite of datasets (e.g., GQA for General, ScienceQA for Knowledge).

Evaluate visual generation with GenEval for compositional text‑to‑image quality, DPG‑Bench for dense‑prompt generation, and CLIP similarity scores across short, medium, and long prompts.

Report the held‑out diffusion loss on a 1 000‑sample validation set as an overall generation quality proxy.

Demystifying Modality Knowledge Flow

We detail the experimental design for probing modality knowledge flow across language, vision, and generation.

Modality Knowledge Flow describes how learning signals travel directionally between language, visual‑understanding, and visual‑generation modules when they are trained jointly.

Is Modality Knowledge Flow the same as generic multitask learning?

No. Multitask learning merely shares parameters across tasks, whereas Modality Knowledge Flow emphasizes *directional* transfer effects that can be asymmetric and concept‑dependent.

**Figure 4.** Overview of the synthetic CLEVR testbed. We extend the standard CLEVR vocabulary across five conceptual axes: color, shape, spatial relation, size, and object count. To isolate knowledge flow, specific target concepts (highlighted in red) are systematically ablated from targeted modality training streams.

Pre‑train a vision encoder/decoder on 50 B tokens of SSTK image–text pairs (fixed vision allocation).

Pre‑train a language model (DCLM) on 50 B tokens of web text (fixed language allocation).

Create mixtures by adding language tokens at ratios {0 %, 20 %, 40 %, 60 %, 80 %}, corresponding to {0, 12.5, 33, 75, 200} B additional tokens, while keeping the vision allocation constant.

For each mixture, train the model from scratch and evaluate visual‑understanding on General, Vision‑Centric, Knowledge, and OCR & Chart benchmarks.

Also evaluate visual‑generation (unconditional and text‑conditional) by measuring diffusion loss and text‑to‑image alignment.

Repeat the same procedure swapping the role of language and vision: add visual‑understanding tokens (same token sweep) to a fixed 50 B language base, and similarly for generation tokens.

Generate CLEVR scenes covering five concept axes: color, shape, spatial relation, size, and object count.

For each target concept (e.g., “red” in the Color axis), remove all occurrences of that value from the training stream of the chosen modality (visual‑understanding, language, or generation).

Train a multimodal model on the mixed data with the specified ablation applied, keeping total token budget constant.

Evaluate per‑concept performance on a held‑out CLEVR test set that includes all values.

Compare the performance drop across modalities to infer the direction and strength of knowledge flow for that concept.

Training data: 900 k images without red + 10 k images with red (from generation), plus 1 M language captions.

After training, the model’s color‑identification accuracy on a held‑out CLEVR test set drops from 92 % to 78 %.

The language‑to‑vision downstream task (e.g., VQA) shows a 5‑point BLEU loss, confirming that visual‑understanding color knowledge contributed to language‑driven visual performance.

This concrete drop demonstrates that visual‑understanding supplies color knowledge to the language and generation pathways; removing it isolates the causal contribution of that concept.

Concept-Dependent Knowledge Transfer

Zero‑shot transfer fails for low‑level attributes but structural concepts transfer asymmetrically.

Zero‑shot transfer fails for low‑level concepts while structural concepts transfer asymmetrically between visual understanding and generation.

Figure 5 shows near‑zero generation accuracy for color/shape and up to +15 % accuracy gain for relation when learned via understanding.

All experiments keep the model architecture, total number of training steps, and overall data volume identical; the only variable is which modality stream is stripped of a given concept.

Transfer success hinges on whether a concept is low‑level (color, shape) or structural (relation, size, count); low‑level attributes need direct supervision in each modality, whereas structural attributes can cross modalities, often only one way.

How is Concept‑Dependent Transfer different from generic multimodal transfer?

Generic transfer assumes all knowledge can be shared uniformly across modalities. Concept‑Dependent Transfer asserts that only structural concepts transfer readily, while low‑level attributes require task‑specific exposure; the transfer is therefore asymmetric and concept‑specific.

**Figure 5** Zero-shot concept transfer results on CLEVR. Left (Color, Shape): Low-level (Color, Shape) attributes fail to transfer in either direction. Right (Relation, Size, Count): Structural concepts exhibit an asymmetric transfer. Understanding helps zero-shot generation, whereas generation largely fails to help understanding, with a minor exception for counting.

**Figure 6. Concept recovery via fine-tuning.** We measure how quickly models learn a missing low-level concept. Top row: Prior exposure via visual understanding provides no acceleration for color generation, but leaves a usable prior that accelerates shape generation. Bottom row: Prior exposure via visual generation acts as a booster, accelerating visual understanding learning across both color and shape.

Low‑level concepts (color/shape) do not transfer; structural concepts (relation/size) exhibit asymmetric transfer.

Synergy and Competition in Pretraining

Simple tasks boost cross‑modal performance while complex tasks trigger competition, shaped by sharing attention and FFNs.

Simple visual or linguistic tasks act as cross‑modal boosters, whereas complex tasks create capacity competition; sharing attention and normalization layers preserves synergy while modality‑specific feed‑forward networks mitigate competition.

In the `split_ffn` configuration, language perplexity drops by 0.211 compared to unimodal baselines, whereas the fully dense configuration raises perplexity by 0.272, demonstrating that decoupling FFNs removes interference while shared attention sustains cooperation.

**Figure 1.** Impact of scaling language data on visual understanding and generation. Increasing the language ratio universally improves both vision capabilities.

**Figure 2.** Impact of scaling visual understanding data. It significantly benefits visual generation but degrades pure language performance.

**Figure 3.** Impact of scaling visual generation data. Adding visual generation causes minor fluctuations in language and most understanding tasks.

**Figure 7** Overview of data complexity progressions. Examples of visual (top) and language (bottom) data used to evaluate the impact of task complexity on modality interactions, ranging from simple synthetic patterns to complex real-world distributions.

**Figure 8.** Impact of task complexity on modality interaction. Left: Escalating visual task complexity gradually turns synergy into competition. Simple visual tasks (e.g., backgrounds, noise) improve language modeling, whereas complex visual distributions (SSTK, video) degrade text perplexity. Right: Introducing language universally aids visual generation, but the simplest linguistic distribution provides the maximum synergistic boost.

The Necessity of Early Unification

Early simultaneous multimodal training prevents vision laziness and maximizes cross‑modal synergy.

Recall that the paper’s central claim is that multimodal models must integrate language, visual understanding, and visual generation early and jointly, otherwise the visual pathways become under‑utilized.

Integrating visual and textual streams at the start of training is like mixing flour and water before adding eggs to a batter—once the mixture is already baked, later ingredients can’t change its texture.

How does “early unification” differ from the common practice of pretraining a large language model first and then adding vision?

In early unification the visual feed‑forward layers remain randomly initialized when the language trunk is still learning, so language priors can shape vision from the outset. In the language‑first recipe the vision branch is added to a fully‑trained language model; the hardened language weights push the image‑wrapper embeddings toward zero and silence the `img_ffn`, leading to the vision‑laziness effect described in § 5.3.

Fix the total token budget at 1 T tokens for every run.

Choose a language‑warm‑up length B ∈ {0, 200, 400, 600, 800} B tokens.

Consume B language tokens using a pure‑language objective.

Switch to a unified objective with a 50 %/25 %/25 % split (language / visual‑understanding / visual‑generation) for the remaining tokens.

For the sequential‑vs‑joint study, define a three‑stage schedule covering all six permutations of (L, U, G) and run each with and without a 12.5 % replay buffer of previously seen modalities.

Evaluate on the full suite of language, visual‑understanding, and visual‑generation benchmarks after training completes.

Step 1: Randomly initialize the transformer backbone and the image‑side FFN (`img_ffn`).

Step 2: Train on language‑only data for 400 B tokens; the language loss drops from 7.2 → 2.1 bits, and the embedding rows for

Step 3: Switch to the unified mix; after the first 100 B unified tokens the `img_ffn` L2 norm rises modestly to 0.45, then plateaus around 0.48 for the remaining 500 B.

Step 4: At evaluation, the model’s attention on image tokens averages 12 % per layer, compared with 22 % for the 0 B baseline.

This trial shows that even a moderate language warm‑up leaves the vision branch under‑activated, confirming the “vision laziness” trend.

**Figure 11** Timing of unification training. The x-axis represents the number of pure language tokens consumed before visual data is introduced to the training mix. While extending the initial pure language phase yields marginal improvements in unimodal text metrics like language accuracy and perplexity, it triggers a steep and consistent decline in performance across all visual understanding and visual generation benchmarks.

**Figure 12.** Impact of sequential versus joint pretraining across various modality orderings. The charts display the performance of six distinct sequential training paths. Solid bars denote strict sequential training, while patterned bars indicate training with a 12.5% replay buffer of previously seen modalities. The horizontal dashed line represents the simultaneous joint training baseline. The results clearly show that joint training dominates all sequential approaches across almost every metric. Although data replay slightly mitigates catastrophic forgetting, it fails to match the cross-modal synergies.

**Figure 13.** Results of vision laziness in late alignment. We measure the visual pathway's commitment across varying language starting points (0B to 800B tokens) for visual generation (V-Gen) and understanding (V-Und). (A) Training-time activation L2 norms of vision-specific FFNs consistently decrease across layers as the language warm-up lengths. (B) Embedding norms of special image-wrapper tokens shrink in late-aligned models, showing weaker integration into the language manifold. (C) Inference-time activations of the vision FFNs drop drastically when visual unification is delayed. (D) The fraction of attention placed on image tokens at inference time, from image queries during generation and from text queries during understanding, likewise drops with longer language warm-ups, meaning the model focuses less on the image itself. Across all metrics, longer pure-language pretraining reduces the model's utilization of its visual parameters.

Unified Pretraining Recipes

Actionable recipes for unified multimodal pretraining, covering data mixing, architecture, and timing.

We now turn the empirical insights into concrete pretraining recipes. Because language acts like a universal power outlet while visual generation is a modest battery, allocating most tokens to language and understanding drives the whole system efficiently.

A recipe specifies how many tokens of each modality to feed, which architectural components to share, and when to introduce vision during training.

How does this recipe differ from the naïve “balanced” 50/25/25 split?

The balanced split wastes the majority of tokens on generation, which provides little backward transfer. By skewing the mix to L70/U25/G5 we keep language and understanding tokens abundant, letting them bootstrap generation through shared attention while using far fewer generation tokens.

Define three sweep axes: “Fix MM” (vary L 10‑90 % with U = G), “Fix Lan” (fix L = 50 % and vary U vs. G), and “Next” (fix L = 70 % and fine‑tune U/G).

Run 1 T‑token pretraining runs for each configuration, keeping total token count constant.

Evaluate each model on language perplexity, visual‑understanding metrics (V‑Ctr, OCR, etc.), and visual‑generation metrics (DiffLoss, DPG, GenEval).

Collect the results in Table 1 and identify the configuration that maximizes language and understanding while preserving generation quality.

Language tokens are streamed first; after every 10 M tokens the scheduler injects a batch of 250 k visual‑understanding tokens.

Every 2 M tokens a small batch of 5 k generation tokens is interleaved, keeping the generation proportion at 5 %.

The model updates shared attention weights on each batch, while the modality‑specific FFNs receive gradients only from their own token type.

At the end of training the model has seen 700 B language contexts, 250 B visual‑understanding contexts, and 50 B generation contexts, matching the L70/U25/G5 ratio.

This schedule shows that a tiny generation token budget can still yield strong image‑to‑text performance because the language and understanding streams already encode the necessary visual priors.

**Figure 9. Impact of parameter sharing on cross-modal performance.** Fully shared (dense) parameters force modality competition, degrading both language and vision. Decoupling solely the FFNs (`split_ffn`) perfectly mitigates this competition while leveraging shared attention to foster strong synergy. Decoupling attention (`split_ffn_attn`) or normalization (`split_ffn_norm`) significantly diminishes these improvements, and fully isolating all parameters (`split_all`) yields identical results to baselines.

**Figure 10** Impact of vision encoder designs on modality synergy. Left: The impact of pairing pure background images with language across different encoder configurations on language perplexity ($\Delta$ PPL). Right: The relative change in diffusion loss (%) for conditional and unconditional generation when paired with simple language. Modality synergy consistently occurs across all four visual tokenization designs.

Having identified the optimal data mix, we scale the design to 13.5 B‑parameter MoE models and test three orthogonal variables: data mix, architecture, and unification timing.

Train a 13.5 B MoE model (256 experts, top‑16 routing) with the L70/U25/G5 mix (Full recipe).

Train a dense 3.5 B baseline under the balanced 50/25/25 mix with early unification.

Train a MoE model with early unification but replace it with late‑fusion (vision tokens introduced after 60 % of steps) while keeping the balanced mix.

All runs use a 2 T‑token budget and the same optimizer settings.

Report language accuracy, visual‑understanding average, and visual‑generation diffusion loss (Table 2).

Decoupling FFNs mitigates competition while maintaining synergy.

Related Work

Survey of unified multimodal models, vision integration strategies, knowledge flow, and architecture trade‑offs.

Unified models have moved beyond treating vision as a mere conditional input, aiming instead for architectures that jointly perform visual understanding and generation.

Early work quantized images into discrete tokens for autoregressive modeling, while recent hybrid systems such as Transfusion combine continuous diffusion with language prediction to set $a$ strong baseline for joint modality modeling.

Late‑fusion pipelines stitch pretrained large language models with separate visual encoders via adapters, but this forces rich visual signals into a pre‑existing language space, creating a performance bottleneck.

Early‑fusion approaches integrate vision from the start of training, allowing visual and linguistic representations to co‑evolve and supporting deeper synergy for tasks like interleaved generation and robotics.

Empirical studies show that language consistently boosts visual generation, while visual understanding strongly drives generation; visual generation contributes little back, revealing a highly asymmetric knowledge flow.

Design choices around parameter sharing shape the balance of synergy versus competition: monolithic transformers maximize shared attention, whereas modality‑specific feed‑forward networks or separate attention blocks reduce competition.

Recent work highlights that shared attention layers foster cross‑modal synergy, while decoupled feed‑forward networks mitigate competition, supporting the effectiveness of mixture‑of‑experts designs.

Additional Results and Verification

Supplementary transfer experiments confirm modality‑flow findings across tokenizers and datasets.

We replicate the real‑world modality‑transfer experiments using three alternative vision pipelines—raw pixels, CLIP + VAE, and AR (UniTok)—to test whether the asymmetry and neutrality of knowledge flow depend on the visual tokenizer.

**Figure 14** Modality transfer results using Raw Pixels. We replicate the real-world knowledge flow transfer experiments using raw pixels. The results mirror the main text: language universally boosts vision (top), understanding significantly benefits generation (middle), and generation yields a highly stable, neutral effect on language and visual understanding tasks (bottom).

**Figure 15.** Modality transfer results using CLIP + VAE. We replicate the real-world modality transfer experiments utilizing a SigLIP-2 encoder for visual understanding and an SD3 VAE for generation. The transfer dynamics remain highly consistent, showing that language acts as a universal booster, understanding acts as a strong prior for generation, and generation has a stable, neutral, or slightly fluctuating impact on other abilities.

**Figure 16. Modality transfer results with AR modeling.** We replicate the real-world modality transfer experiments utilizing discrete visual tokenization and autoregressive next-token prediction. Consistent with our continuous diffusion findings, the results demonstrate that language acts as a universal booster (top), visual understanding serves as a strong prior to drive generation (middle), and visual generation exerts a highly stable, neutral effect on other capabilities (bottom). Note that in the language-to-vision transfer experiments (top row), we record slightly different data points.

The CLEVR synthetic suite is extended with a richer color palette (red, blue, green, gray, brown, purple, cyan, yellow) and additional shapes (cross, ellipse, pentagon, rectangle, triangle) to stress low‑level concept transfer.

Five target concepts—color, shape, relation, size, count—are probed via balanced VQA pairs derived directly from scene‑graph metadata, ensuring noise‑free ground truth for both understanding and generation tasks.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers