Scaling Native Multimodal Pre-Training from Scratch

Haoyuan Wu, Aoqi Wu, Hai Wang, Jiajia Wu, Jinxiang Ou, Bei Yu

Native multimodal pre-training follows distinct scaling laws for language and vision, enabling compute-optimal model design.

How should compute resources be optimally allocated between model size and training tokens when training native multimodal models from scratch?

Native multimodal models are often built by patching vision encoders onto pre-trained language models, creating optimization asymmetries that hinder deep cross-modal integration. The authors train decoder-only Transformers from scratch on interleaved multimodal data, decoupling the language and multimodal objectives to derive independent compute-optimal scaling laws. They find that while language scaling is invariant to data composition, multimodal scaling is highly sensitive to it, necessitating a joint Pareto frontier to guide optimal resource allocation.

Paper Primer

The core mechanism hinges on modeling the language objective ($L_{\text{text}}$) and multimodal objective ($L_{\text{mm}}$) as separate functions of model size ($N$) and token count ($D$). By using IsoFLOP profiles and training-curve envelopes, the authors identify that text-heavy mixtures require larger model capacities to remain compute-efficient, effectively shifting the optimal allocation toward parameter expansion.

Native multimodal pre-training induces positive cross-modal transfer, specifically enhancing pure-text spatial reasoning.

Models trained with multimodal tokens consistently outperform text-only baselines on abstract spatial reasoning subtasks (e.g., MazeNav, SpatialMap) despite the queries lacking visual input. The performance gap between multimodal and text-only models widens substantially as model scale increases up to 3B parameters.

Multimodal in-context learning emerges naturally from native pre-training as a function of scale and data.

Performance gains from few-shot templates are absent in small models but scale positively with model capacity and training duration. At the 3B parameter scale, accuracy improves by 2.43 points in 3-shot settings compared to 0-shot baselines.

Why does this paper decouple the language and multimodal objectives instead of using a single aggregate loss?

The authors observe that the two modalities follow fundamentally divergent scaling regimes: language learning is composition-invariant, while cross-modal alignment is highly sensitive to the multimodal data ratio. Decoupling allows them to isolate these dynamics and avoid over-parameterized, data-starved architectures.

Does adding multimodal data degrade the model's core language capabilities?

No. The authors report that aggregate performance across 16 pure-text benchmarks remains consistent across all multimodal data ratios, with average scores deviating by less than one percentage point.

Motivation and Approach

We expose the missing scaling laws for native multimodal pre‑training and define the compute‑optimal gap.

Large language models excel at text‑only reasoning, but this limits their ability to perceive the physical world. Native multimodal pre‑training trains models from scratch on multimodal inputs, achieving deep cross‑modal integration and removing the asymmetry of late‑fusion pipelines.

Despite these advantages, the scaling behavior of native multimodal pre‑training is unknown: how should model size and token count be allocated under a fixed compute budget? This missing compute‑optimal scaling law constitutes the central gap this work addresses.

Training a transformer from scratch on combined vision‑language data lets the model learn a single shared representation, rather than stitching together separately‑trained language and vision modules.

The shift from late‑fusion to native multimodal pre‑training unlocks scalable cross‑modal learning.

Compute-Optimal Scaling Framework

Define how to split a fixed compute budget between model size and token count.

When a research budget is fixed, the biggest obstacle is deciding how much of it to spend on a larger model versus feeding the model more data.

Think of a fixed amount of money that must be split between buying a faster GPU (larger $N$) and buying more training data (larger $D$); the optimal split maximizes performance for the given spend.

Compute the loss for each pair (using a toy loss curve where loss decreases with both $N$ and $D$): $L(8,4)=0.30$, $L(4,8)=0.28$, $L(2,16)=0.35$.

The arg‑min selects $(N=4,D=8)$ because it yields the lowest loss $0.28$.

According to the power‑law $N_{\text{opt}}\propto C^{a}$ with $a=0.5$, the predicted optimal $N$ is $\sqrt{C/6}\approx\sqrt{32}\approx5.7$, rounded to the nearest feasible integer $N=6$, which is close to the empirical optimum $N=4$.

This illustrates that the optimal allocation balances model size and data rather than pushing all budget into one side.

The optimal point is not at the extremes (all compute in $N$ or all in $D$); the trade‑off curve has a clear minimum where increasing one side further harms performance.

Why not simply double the model size and keep the token count fixed?

Doubling $N$ forces $D$ to halve because $6ND$ must stay constant. The loss curve shows diminishing returns for larger $N$ when data is scarce, so the overall loss actually rises compared to the balanced allocation.

To locate the minimum of the loss curve for any budget we use IsoFLOP profiles: we fix $C$, vary $N$, and plot loss versus $\log N$.

**Figure 1.** IsoFLOP curves (language objective). For a range of model sizes, we adjust the number of training tokens to maintain a constant final FLOPs, setting the cosine cycle length to match this target compute budget. The distinct valley in the loss curve demonstrates that an optimal model size exists for any given FLOP budget. Based on the locations of these minima, we project the compute-optimal model size and token count for larger scales.

The second estimator aggregates training curves across many model sizes, extracts the lower envelope, and reads off the minimal loss for each compute value.

**Figure 2.** Training curve envelope (language objective). Training curves are shown for all runs across varying values of $r$, encompassing various model sizes. By extracting the envelope of minimal loss per FLOP from these curves, we estimate the optimal model size and training token allocation for a given compute budget.

Along the envelope the loss follows a simple power law in compute, which we call the compute frontier.

**Figure 3.** Compute-optimal allocation (language objective). (a) Optimal model size $N_{opt}$ and (b) optimal token count $D_{opt}$ as a function of the compute budget $C_{text}$ for each $r$. Solid lines and stars represent the IsoFLOP fits and their respective minima, while thin dashed lines and scattered points provide a cross-check using the training envelope. The two estimation methods align closely, and the slopes exhibit minimal variation with respect to $r$, demonstrating that the compute-optimal allocation is composition-invariant.

Our experiments use 250 B text tokens and 75 B multimodal tokens, trained with a MoE decoder‑only Transformer that projects images directly via a single patch embedding layer and omits any auxiliary loss.

Scaling Native Multimodal Models

Derive how to split compute between text and multimodal data using the mixture ratio.

Native multimodal pre‑training lacks a principled way to split a fixed compute budget between text and visual tokens, leading to either over‑parameterized models or data‑starved training.

r is the fraction of training tokens that come from the multimodal stream; it directly controls how much compute is devoted to visual data versus pure text.

Multimodal tokens: $D_{\text{mm}} = 0.2 \times 100 = 20$ M.

Text tokens: $D_{\text{text}} = 0.8 \times 100 = 80$ M.

If the model has $N = 1$ B parameters, the compute spent on multimodal data is $C_{\text{mm}} = 6 \times 1\text{B} \times 20\text{M}$, and on text it is $C_{\text{text}} = 6 \times 1\text{B} \times 80\text{M}$.

Increasing $r$ shifts a larger share of the fixed FLOP budget toward multimodal data, which quickly dominates the scaling behavior of the overall system.

The frontier captures the best achievable pair of language loss $L_{\text{text}}$ and multimodal loss $L_{\text{mm}}$ for a given total compute $C_{\text{total}}$, by sweeping the mixture ratio $r$.

**Figure 4.** **IsoFLOP curves (multimodal objective).** For a range of model sizes, we adjust the number of training tokens to maintain a constant final FLOPs, setting the cosine cycle length to match this target compute budget. The distinct valley in the loss curve demonstrates that an optimal model size exists for any given FLOP budget. Based on the locations of these minima, we project the compute-optimal model size and token count for larger scales.

**Figure 5.** Training curve envelope (multimodal objective). Training curves are shown for all runs across varying values of $r$, encompassing various model sizes. By extracting the envelope of minimal loss per FLOP from these curves, we estimate the optimal model size and training token allocation for a given compute budget.

**Figure 6.** **Compute-optimal allocation (multimodal objective).** (a) Optimal model size $N_{opt}$ and (b) optimal token count $D_{opt}$ as a function of the compute budget $C_{mm}$ for each $r$. Solid lines and stars represent the IsoFLOP fits and their respective minima, while thin dashed lines and scattered points provide a cross-check using the training envelope. The two estimation methods align closely, and the slopes vary significantly with respect to $r$, demonstrating that the compute-optimal allocation is composition-variant.

**Figure 7.** **Joint Pareto frontier.** (a) Pareto frontier demonstrating the trade-off between multimodal loss $L_{mm}$ and language loss $L_{text}$ by sweeping the mixture ratio $r$. The curves are bounded by fixed total compute budgets $C_{total}$, plotting actual data points alongside extrapolated frontiers for larger compute scales. (b) Optimal model size $N_{opt}$ and (c) optimal token count $D_{opt}$ as a function of compute $C_{total}$. The distinct scaling exponents, $a(r)$ and $b(r)$, are compared between the base text objective (r-free) and the multimodal objective across different mixture ratios.

Performance and Capabilities

Multimodal training preserves text ability while boosting spatial reasoning and few‑shot learning.

The paper’s core claim is that native multimodal pre‑training can be scaled without sacrificing language ability, while it endows models with spatial reasoning and few‑shot capabilities.

Multimodal few‑shot prompting adds up to +2.43 points over the 0‑shot baseline at the largest scale (A3B), demonstrating emergent in‑context learning.

Table 13 shows a 3‑shot gain of +2.43 points for the A3B model when trained with $r=0.3$ multimodal data.

**Figure 8.** Text capabilities are preserved under native multimodal pre-training. Average accuracy across 16 text benchmarks with varying multimodal data ratios $r$ (fixed 250B text token budget). (a) Text performance across model sizes $N$. (b) A3B text performance over training tokens $D_{text}$. Consistently overlapping curves indicate multimodal data integration does not degrade core language abilities.

**Figure 9.** Multimodal pre-training enhances pure-text spatial reasoning. Accuracy on SpatialEval's text-only abstract spatial-reasoning sub-tasks (Wang et al., 2024a), comparing baseline ($r = 0$) and multimodal ($r = 0.3$) settings. (a) Accuracy across model sizes $N$. (b) A3B accuracy over training tokens $D_{text}$. Multimodal models consistently outperform text-only baselines, with the performance gap widening at larger scales.

**Figure 10.** Multimodal in-context learning emerges with model scaling. (a) Average multimodal accuracy for 0-, 1-, and 3-shot settings across model sizes N. (b) Relative performance gains of k-shot settings over the 0-shot baseline. The increasing gains demonstrate that multimodal in-context learning emerges with model scaling.

After pre‑training, a model can answer new image‑text queries by simply prepending a few example pairs, without any parameter updates.

How does multimodal in‑context learning differ from standard few‑shot prompting in text‑only LLMs?

In text‑only LLMs the prompt contains only tokens; here each example also includes a visual token stream, so the model must attend jointly across image and text modalities. The shared encoder‑decoder learns to align visual patches with textual queries, enabling the few‑shot templates to convey spatial structure that pure text cannot.

Multimodal training improves spatial reasoning without degrading text performance.

Text Benchmark Data

Scaling token budgets dramatically lifts text benchmark performance.

Increasing the pre‑training token budget from 25 B to 250 B raises the average text benchmark score from 25.40 to 44.77 points.

Table 3 reports the average across 21 benchmarks for each token budget; the 250 B row shows 44.77 versus 25.40 at 25 B.

Across both tables, every benchmark shows a steady rise as the token budget grows, with the most pronounced gains on coding and commonsense reasoning.

**Table 3.** Performance on text benchmarks at different pre-training token budgets for A3B models. The models are trained with 250B text tokens and 25B multimodal tokens.

Extended Text Benchmark Data

Scaling token budgets dramatically lifts text benchmark performance.

Increasing the pre‑training token budget from 25 B to 300 B (with 50 B multimodal tokens) raises the average text benchmark score by 20.21 points.

Average score climbs from 25.00 to 45.21 (Table 4).

Across both tables, every benchmark improves as the pre‑training token budget increases, with especially steep gains on coding (HumanEval+, MBPP+) and spatial reasoning (SpatialEval).

**Table 4.** Performance on text benchmarks at different pre-training token budgets for A3B models. The models are trained with 250B text tokens and 50B multimodal tokens.

**Table 5.** Performance on text benchmarks at different pre-training token budgets for A3B models. The models are trained with 250B text tokens and 75B multimodal tokens.

Multimodal Benchmark Results (25B)

Table 6 shows multimodal performance scaling with token budget, peaking at 33.12 % average.

Average multimodal benchmark accuracy improves with larger pre‑training token budgets, reaching 33.12 % at 275 B tokens.

Table 6 reports the average score climbing from 25.19 % at 25 B tokens to 33.12 % at 275 B tokens.

**Table 6.** Performance on multimodal benchmarks at different pre-training token budgets for A3B models. The models are trained with 250B text tokens and 25B multimodal tokens.

Multimodal Benchmark Results (50B)

Scaling token budgets raises the average multimodal benchmark score by nine points.

Increasing the pre‑training token budget from 25 B to 325 B raises the aggregate multimodal benchmark score by 9.09 points.

Table 7 shows the average score climbing from 26.03 at 25 B to 35.12 at 325 B.

All benchmark families improve monotonically as the token budget grows, with the steepest jumps in Spatial Reasoning (e.g., CountBench, OmniSpatial) and Vision Knowledge categories.

**Table.** Performance of models across various benchmarks as a function of the number of pre-trained tokens (ranging from 25B to 325B).

Multimodal Benchmark Results (75B)

Table 8 shows multimodal performance scaling with token budget, while Table 9 reports text benchmark results.

Increasing multimodal pre‑training tokens from 25 B to 300 B raises the average multimodal benchmark score from 25.91 to 35.90.

Table 8 reports the average across all multimodal tasks for each token budget, showing a monotonic rise.

**Table 8.** Performance on multimodal benchmarks at different pre-training token budgets for A3B models. The models are trained with 250B text tokens and 75B multimodal tokens.

In contrast, Table 9 shows that text‑only benchmarks achieve comparable scores across model sizes, suggesting that allocating more multimodal tokens does not degrade textual capabilities.

Comprehensive Text Benchmark Results

Increasing multimodal tokens boosts average text performance across model sizes.

Increasing multimodal tokens from 25 B to 75 B raises the average text benchmark score of the 3 B model.

Table 10 reports 44.77, while Table 12 reports 46.03 for the same model.

Across all six model sizes (71 M → 3 B) every benchmark category—coding, mathematics, logic reasoning, knowledge, and commonsense—shows a monotonic gain as multimodal tokens increase, typically between 0.5 and 2.0 points per step.

**Table 10.** Performance on text benchmarks. Model sizes (N) range from 71M to 3B activated parameters, with all models trained on 250B text tokens and 25B multimodal tokens.

**Table 11.** Performance on text benchmarks. Model sizes (N) range from 71M to 3B activated parameters, with all models trained on 250B text tokens and 50B multimodal tokens.

**Table 12.** Performance on text benchmarks. Model sizes (N) range from 71M to 3B activated parameters, with all models trained on 250B text tokens and 75B multimodal tokens.

Comprehensive Multimodal Benchmark Results

Scaling to 3 B parameters boosts average multimodal accuracy to 36.05 in the 3‑shot setting.

Tables 13‑15 report performance of Native Multimodal models across 0‑, 1‑, and 3‑shot evaluation regimes, spanning model sizes from 71 M to 3 B activated parameters.

Increasing model size to 3 B parameters raises the overall 3‑shot average score to 36.05, a +10.41 improvement over the 71 M model.

Table 15

**Table 15.** Performance on multimodal benchmarks in a 3-shot setting. Model sizes (N) range from 71M to 3B activated parameters, with all models trained on 250B text tokens and 75B multimodal tokens.

**Table.** Performance comparison across various benchmarks for different model sizes (A71M to A3B).

Related Work and Conclusion

We contextualize prior multimodal approaches and summarize our scaling contributions.

Most vision‑language models follow a late‑fusion paradigm, attaching a pretrained language model to a separately pretrained vision encoder via a lightweight projection layer. This modular design creates an asymmetry: vision and language streams are learned independently from distinct data distributions and objectives, limiting deep cross‑modal integration — like plugging a pre‑made Lego brick into a custom‑built structure, which is easier than molding the brick as part of the structure from the start. By contrast, native multimodal pre‑training trains a single model from scratch on interleaved data, establishing a unified representational space without a dedicated vision encoder.

Scaling laws provide a predictable link between performance, model size, data, and compute, guiding resource allocation under fixed budgets. Prior work on unimodal language models revealed power‑law loss trends and later refined compute‑optimal frontiers using IsoFLOP profiles and training‑curve envelopes. Existing analyses, however, focus on single‑modality settings, leaving the joint allocation for heterogeneous modalities unresolved; we decouple language and multimodal components, showing divergent scaling regimes that can be reconciled by a compute‑optimal frontier accounting for data mixture.

We established compute‑optimal scaling laws for native multimodal pre‑training, demonstrating that language scaling is largely composition‑invariant while multimodal scaling is composition‑variant and sensitive to the image‑text data ratio. Our analysis, limited to models up to 3 B active parameters and a single image‑text family, provides a principled framework for designing compute‑efficient multimodal foundations and motivates validation at larger scales and across diverse modalities.

Implementation Details

Implementation specifics for native multimodal pre‑training, covering optimizer, schedule, and hyperparameters.

We train the native multimodal models with the Muon optimizer, applying a weight decay of 0.1 and clipping gradients at a norm of 1.0. A warmup‑stable schedule drives the learning rate, while a global batch of 16 M tokens and a maximum context of 4096 tokens define the compute envelope. Crucially, the loss is evaluated only on text tokens; vision tokens are excluded via masking.

Training‑loop skeleton illustrating optimizer setup, learning‑rate schedule, and loss masking.

**Table 1.** Pre-training hyperparameters. We detail the hyperparameters used for pre-training different model configurations to derive scaling laws.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers