Scaling Laws for Language Models
Language model performance scales predictably as a power law with model size, dataset size, and compute.
How do language model performance, model size, dataset size, and compute budget relate to each other in a predictable, quantitative way?
Researchers often struggle to determine how much data or compute is required to improve a language model, leading to inefficient training runs that either overfit or fail to reach potential performance. This paper establishes empirical power-law scaling relations for Transformer language models, showing that performance improves smoothly as model size, dataset size, and compute are scaled up. The authors demonstrate that optimally compute-efficient training involves scaling up model size significantly while increasing data and training time much more slowly.
Paper Primer
The core mechanism is the identification of universal power-law dependencies for the cross-entropy loss. By measuring performance across seven orders of magnitude, the authors show that loss is governed by three primary factors: the number of non-embedding parameters ($N$), the dataset size ($D$), and the training compute ($C$).
Model performance depends strongly on scale but weakly on architectural shape.
Empirical measurements across varying depths, widths, and head counts show that total parameter count ($N$) is the primary driver of performance, with architectural hyperparameters having minimal impact. Trends span more than six orders of magnitude without deviation.
Large models are significantly more sample-efficient.
As model size increases, the amount of data required to reach a specific loss target grows sub-linearly ($D \propto N^{0.74}$), meaning larger models require fewer optimization steps and fewer data points to reach the same performance as smaller models. Every 8x increase in model size requires only a 5x increase in data to avoid overfitting.
Why does this paper matter for practitioners?
It provides a predictive framework for resource allocation, allowing researchers to determine the optimal model size and compute budget before starting a training run, rather than relying on trial and error.
What is the scope of these scaling laws?
The laws apply to Transformer-based language models trained with cross-entropy loss; while the authors conjecture they may apply to other generative tasks, they explicitly note that the trends must eventually flatten as performance approaches the non-zero entropy of natural language.
Compute-efficient training requires prioritizing larger models over longer training times or massive datasets. Practitioners should stop training significantly before convergence to maximize the utility of their compute budget.
Introduction and Motivation
We expose predictable power‑law scaling of language‑model loss with size, data, and compute.
Language modeling offers a massive, well‑defined arena for studying artificial intelligence, yet we lack a principled way to forecast how performance improves as we scale models, data, and compute. This gap hinders efficient allocation of limited training resources. The paper addresses the gap by showing that cross‑entropy loss follows a smooth power‑law relationship with the three scale factors $N$, $D$, and $C$.
We need a reliable formula that predicts a model’s loss from its size ($N$), the amount of training data ($D$), and the compute budget ($C$) so we can choose the most efficient training configuration.
The key insight is that scaling laws make language‑model performance reliably predictable across size, data, and compute.
Empirical Scaling Laws
Scaling laws let us predict loss from model size, data, and compute without exhaustive experiments.
Predicting test loss for a new language model traditionally required costly sweeps over size, data, and compute. The authors observe that loss follows a simple power‑law in each factor, turning a multi‑dimensional search into a handful of exponent estimates.
Loss drops as a fixed‑exponent power of any bottleneck resource—model parameters, training tokens, or compute—so each resource contributes an independent, predictable term.
Compute the parameter term: $\bigl(N_c/N\bigr)^{\alpha_N}= \bigl(8.8\!\times\!10^{13}/2\!\times\!10^{6}\bigr)^{0.076}\approx 0.42$.
Compute the data term: $\bigl(D_c/D\bigr)^{\alpha_D}= \bigl(5.4\!\times\!10^{13}/5\!\times\!10^{9}\bigr)^{0.095}\approx 0.31$.
Sum the contributions: $L(N,D)\approx0.42+0.31=0.73$ nats.
If we double $N$ to $4\times10^{6}$, the parameter term drops to $\approx0.36$, reducing total loss to $0.67$ nats.
If we instead double $D$ to $1\times10^{10}$, the data term falls to $\approx0.27$, yielding $L\approx0.69$ nats.
The example shows that each resource contributes a separate, diminishing term; improving the dominant bottleneck yields the biggest loss reduction.
**Figure 1.** Language modeling performance improves smoothly as we increase the model size, dataset size, and amount of compute used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.
Even when the Transformer’s depth, width, or head count change, the loss follows the same power‑law in $N$; shape only shifts the constant, not the exponent.
**Figure 5.** Performance depends very mildly on model shape when the total number of non-embedding parameters $N$ is held fixed. The loss varies only a few percent over a wide range of shapes. Small differences in parameter counts are compensated for by using the fit to $L(N)$ as a baseline. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an $(n_{layer}, d_{model}) = (6, 4288)$ reaches a loss within 3% of the $(48, 1600)$ model used in [RWC+19].
Why does the combined scaling law use a sum of two power‑law terms instead of a single term?
Each term captures a distinct bottleneck. If the model is too small, the $N$‑term dominates; if the dataset is insufficient, the $D$‑term dominates. Adding them lets the formula automatically select the limiting factor without hand‑tuning a piecewise rule.
The Infinite Data Limit
Performance stabilizes when data size ceases to be the bottleneck.
Test loss across model sizes follows the power‑law L(N,D) with a fit quality R² > 0.99, confirming the scaling law even when data size varies.
Figure 9 shows the empirical loss curves aligning with the fitted L(N,D) surface; Table 2 lists the fitted exponents.
We train each model with a uniform 10 % dropout rate and stop training as soon as test loss ceases to improve, ensuring comparable optimization across scales.
When the training dataset is so large that adding more tokens no longer reduces loss, the model’s performance is bounded solely by its size.
How does the “infinite data limit” differ from simply training on a larger dataset?
Increasing $D$ reduces the $ (D_c/D)^{\alpha_D}$ term, but once $D$ is large enough that this term is negligible, further data adds no benefit; only the $ (N_c/N)^{\alpha_N}$ term remains, so performance is driven exclusively by model size.
**Figure 8** **Left:** Generalization performance to other data distributions improves smoothly with model size, with only a small and very slowly growing offset from the WebText2 training distribution. **Right:** Generalization performance depends only on training distribution performance, and not on the phase of training. We compare generalization of converged models (points) to that of a single large model (dashed curves) as it trains.
Scaling Laws for Model Size
Scaling laws reveal sublinear data growth and optimal compute trade‑offs across model size.
We study the relative loss $\delta$L(N,D) = L(N,D)/L(N,∞) − 1, which isolates the excess loss due to finite data.
At large $D$ the expression admits a series expansion in $1/D$, confirming that data‑size effects decay inversely with token count.
Dataset size grows sub‑linearly with model size, following $D \approx 5\!\times\!10^{3}\,N^{0.74}$, so larger models need proportionally fewer tokens per parameter.
Fits to models up to $10^{9}$ parameters on the 22 B‑token WebText2 corpus show a consistent exponent of 0.74 with residual error ≈ 0.02 loss units.
Consequently, models smaller than $10^{9}$ parameters can be trained on the 22 B‑token WebText2 dataset with negligible over‑fitting, whereas the largest models begin to show mild over‑fitting.
We now turn to scaling laws that incorporate training time, introducing a universal step count $S_{\min}$ that accounts for sub‑optimal batch sizes.
Training at $B \approx B_{\text{crit}}$ yields a near‑optimal time‑compute trade‑off: it requires roughly $2 S_{\min}$ steps and processes $2 E_{\min}$ examples.
Fitting this form to our Adam‑optimized runs yields the parameters shown in Table 3 ($\alpha_{N}$ = 0.077, $\alpha_{S}$ = 0.76, $N_c$ = 6.5·10¹³, $S_c$ = 2.1·10³), producing the loss curves in Figure 4.
**Figure 11** When we hold either total compute or number of training steps fixed, performance follows $L(N,S)$ from Equation (5.6). Each value of compute budget has an associated optimal model size that maximizes performance. Mediocre fits at small $S$ are unsurprising, as the power-law equation for the learning curves breaks down very early in training.
Using the fitted law we can also bound the earliest step at which early‑stopping becomes safe when data is limited.
Empirically, this bound aligns well with observed early‑stopping points, validating the joint model‑size‑time scaling framework.
Optimal Compute Allocation
Allocate compute between model size and data to maximize performance.
We previously saw loss scaling with compute when batch size is fixed. Here we correct for the critical batch size and derive the optimal way to split compute between model size and data.
Allocate most of the extra compute to growing the model, while using only modest increases in data and batch size.
Model size: $N \approx 10^{0.73}=5.4\;\approx5$ million parameters.
Critical batch size: $B_{\text{crit}} \approx 10^{0.24}=1.7\;\approx2$ million tokens per batch.
Serial steps: $S_{\min} \approx 10^{0.03}=1.07\;\approx1$ epoch (one pass over the data).
The total number of data tokens processed is $2B_{\text{crit}}S_{\min}\approx4$ million, far fewer than the $10$ million tokens that would be needed if we kept batch size fixed.
This toy calculation shows that most of the compute budget should be spent on enlarging the model; data and steps increase only marginally.
Why not allocate all extra compute to data instead of the model?
Because the loss reduction per unit of compute is proportional to the exponent: $0.73$ for model size versus $0.03$ for data. Scaling the model therefore yields a far larger drop in loss for the same amount of compute.
The batch size that makes training most compute‑efficient grows slowly with total compute and is essentially independent of model size.
$B_{\text{crit}} \approx 100^{0.24}=3.8\;\approx4$ million tokens per batch.
If we kept the original batch size $B=1$ million, we would waste roughly 75 % of the compute potential.
The example illustrates that even large compute budgets only modestly increase the optimal batch size.
Does $B_{\text{crit}}$ depend on the number of parameters $N$?
No. Empirical measurements across models from 3 M to 85 M parameters show the same $B_{\text{crit}}$–loss curve, confirming that the critical batch size is a function of loss (and thus compute) rather than model size.
Compute‑optimal allocation given a compute budget $C$.
**Figure 3.** As more compute becomes available, we can choose how much to allocate towards training larger models, using larger batches, and training for more steps. We illustrate this for a billion-fold increase in compute. For optimally compute-efficient training, most of the increase should go towards increased model size. A relatively small increase in data is needed to avoid reuse. Of the increase in data, most can be used to increase parallelism through larger batch sizes, with only a very small increase in serial training time required.
**Figure 10** The critical batch size $B_{crit}$ follows a power law in the loss as performance increase, and does not depend directly on the model size. We find that the critical batch size approximately doubles for every 13% decrease in loss. $B_{crit}$ is measured empirically from the data shown in Figure 18, but it is also roughly predicted by the gradient noise scale, as in [MKAT18].
**Figure 15** Far beyond the model sizes we study empirically, we find a contradiction between our equations for $L(C_{min})$ and $L(D)$ due to the slow growth of data needed for compute-efficient training. The intersection marks the point before which we expect our predictions to break down. The location of this point is highly sensitive to the precise exponents from our power-law fits.
Compute-Efficient Training Details
Ablation analysis quantifies the cost of deviating from compute‑efficient training and optimal model size.
This appendix evaluates how the training schedule and model‑size choices affect the three core resources: number of parameter updates, model size, and total compute.
We first compare the efficient schedule (using $f\approx10\%$) to the more common practice of training until a much smaller convergence factor $f=2\%$.
Training with the compute‑efficient schedule reduces the number of parameter updates by 7.7× relative to the $f=2\%$ baseline.
Derived from equations (B.11)–(B.14) where $f=10\%$ versus $f=2\%$ yields a $7.7$‑fold reduction in update count.
The same efficient schedule permits using 2.7× more non‑embedding parameters for the same loss.
From the ratio $N_f/N_f = 1/\alpha_N \approx 2.7$ in (B.12) when $f=10\%$ versus $2\%$.
Overall compute required drops by 65 % under the efficient schedule.
Equation (B.14) gives $C_f/C_f \approx 0.35$, i.e. a 65 % reduction.
Next we examine the impact of deviating from the optimal model size while keeping the compute budget fixed.
Choosing a model size between $0.6\times$ and $2.2\times$ the optimal incurs at most a 20 % increase in compute budget.
From equation (B.15) after eliminating $L$ and evaluating the ratio $C(N,\text{Neff})/C(\text{Neff},\text{Neff}) = \alpha_S/\alpha_N$.
A $2.2\times$ larger model cuts required training steps by 45 % while increasing total compute by 20 %.
Using (B.16)–(B.17) the step‑ratio $S(N,\text{Neff})$ shows a $45\%$ reduction in steps; the compute ratio remains $+20\%$.
Supplemental Figures and Early Stopping
Supplemental analyses of early stopping, recurrent transformers, batch size, sample efficiency, context effects, learning‑rate schedules, fit quality, and generalization.
Section D.1 revisits the early‑stopping analysis of §5.3, showing both train and test loss curves for a fixed model size across multiple dataset sizes.
**Figure 9** The early-stopped test loss $L(N, D)$ depends predictably on the dataset size $D$ and model size $N$ according to Equation (1.5). **Left:** For large $D$, performance is a straight power law in $N$. For a smaller fixed $D$, performance stops improving as $N$ increases and the model begins to overfit. (The reverse is also true, see Figure 4.) **Right:** The extent of overfitting depends predominantly on the ratio $N^{\alpha_N/\alpha_D}/D$, as predicted in equation (4.3). The line is our fit to that equation.
Section D.2 compares standard Transformers to recurrent Transformers that reuse parameters, observing a modest gain in performance versus $N$ but a slight loss when measured against compute budget $C$.
**Figure 17.** We compare recurrent Transformers [DGV+18], which re-use parameters, to standard Transformers. Recurrent Transformers perform slightly better when comparing models with equal parameter count, but slightly worse when accounting for reuse and comparing per FLOP.
Section D.3 reports measurements of the critical batch size $B_{\text{crit}}(L)$, derived from the data shown in Figure 18.
**Figure 18.** These figures demonstrate fits to Equation (5.1) for a large number of values of the loss $L$, and for two different Transformer model sizes. These fits were used to measure $B_{crit}(L)$ for Figure 10.
Section D.4 highlights that larger models reach a given loss level in far fewer serial steps, as visualized in Figure 19, implying a roughly 100× boost in sample efficiency from smallest to largest models.
**Figure 19** The number of minimum serial steps needed to reach any fixed value of the test loss decreases precipitously with model size. Sample efficiency (show here for training far below the critical batch size) improves greatly as well, improving by a factor of almost 100 when comparing the smallest possible model to a very large one.
Section D.5 examines token‑wise loss behavior across context positions $T$ (Figure 20) and across model sizes (Figure 21), revealing that later tokens benefit more from scaling while early tokens improve modestly.
**Figure 20** This figure provides information about the performance per token as a function of model size and training time. **Left:** Loss per token as a function of its position $T$ in the 1024-token context. Loss scales predictably as a power-law in $T$. **Right:** Test loss per token as a function of training step.
**Figure 21** In addition to the averaged loss, individual tokens within the 1024-token context also improve smoothly as model size increases. Training runs with shorter context $n_{ctx} = 8$ (dashed lines) perform better on early tokens, since they can allocate all of their capacity to them.
Section D.6 surveys a variety of learning‑rate schedules (Figure 22) and concludes that, provided the summed learning rate is large enough and a warm‑up plus final decay are present, the exact schedule has negligible impact on final loss.
**Figure 22** We test a variety of learning rate schedules including cosine decay, linear decay, as well as other faster/slower decays schedules on a 3 million parameter model, shown on the left. For these experiments we do not decay to zero, since we find that this tends to give a fixed improvement close to the end of training. We find that, as long as the learning rate is not too small and does not decay too quickly, performance does not depend strongly on learning rate. Run-to-run variation is at the level of 0.05 in the loss, so averaging multiple runs is necessary to validate performance changes smaller than this level.
Section D.7 reports that power‑law fits to $L(N)$, $L(C)$, and $L(D)$ consistently outperform alternative forms such as logarithmic fits, even after excluding outlier models (e.g., single‑layer extremes).
Section D.8 shows that, when total parameter count is held constant, depth has negligible effect on generalization (Figure 24); performance correlates almost exclusively with training‑distribution loss.
**Figure 24.** We show evaluations on a series of datasets for models with approximately 1.5 Billion parameters. We observe no effect of depth on generalization; generalization performance depends primarily on training distribution performance. The 12-layer model overfit the Internet Books dataset and we show the early-stopped performance; we have not seen this surprising result in other experiments.
Related Work and Discussion
We situate our scaling findings within prior work and outline broader implications and future directions.
Power‑law scalings have been reported in density estimation, random‑forest models, and many other domains, suggesting a broad phenomenon.
One interpretation treats the exponent as roughly the inverse of the number of relevant data features.
Early work linked performance to dataset size with a power‑law, while more recent studies examined joint scaling of model size and data.
Notably, HNA+17 reported super‑linear scaling of data with model size, whereas we observe a sub‑linear relationship.
Our findings echo the optimal compute‑allocation literature (e.g., Kom19) and the EfficientNet observation that depth and width scale exponentially.
For language models the width‑to‑depth exponent is close to one, implying that keeping width/depth fixed yields the observed scaling.
Across many experiments architectural hyperparameters (depth, width, activation choices) have far less impact than the overall scale of non‑embedding parameters.
VWB16 argued that deep networks behave like ensembles of shallower ones, offering a possible explanation for the scale‑dominance we see.
Related image‑model work (ZK16) showed wide ResNets can outperform deeper variants, reinforcing the limited role of depth alone.
Studies that fix computation per example typically find it proportional to the number of model parameters, whereas we vary both model size and total compute.
Research on highly over‑parameterized models identified a “jamming transition” when model size matches dataset size, a phenomenon we do not observe.
Our data indicate that required training tokens grow sub‑linearly with model size, avoiding the abrupt transition reported elsewhere.
Expansions at large width (JGH18, LXS+19) and noisy quadratic models of learning curves (ZLN+19) provide useful lenses for interpreting our scaling relations.
Accurate predictions also depend on characterizing the Hessian spectrum, as discussed in recent theoretical work.
Empirically we find the cross‑entropy loss scales as a power‑law with non‑embedding parameter count $N$, dataset size $D$, and optimized compute $C_{\min}$.
Performance shows only weak sensitivity to most architectural and optimization hyperparameters, leading to diminishing returns as scale increases.
The WebText2 corpus contains roughly 1.4 tokens per word and 4.3 characters per token, providing concrete dataset statistics.
After our work, RRBS19a independently arrived at similar loss‑dependence predictions for model and data size.
The resulting scaling relations resemble the ideal‑gas law: a simple, universal formula linking macroscopic quantities.
We conjecture that analogous power‑law behavior will hold for other generative tasks such as image, audio, and video modeling.
Future work should test these relations across domains and seek a statistical‑mechanics theory that derives them from first principles.
Larger language models continue to improve and become markedly more sample‑efficient, suggesting “big models” may matter more than “big data”.
Scaling to larger models raises practical concerns about parallelism: pipeline parallelism (HCC+18) splits parameters depth‑wise, while wide networks parallelize more easily (SCP+18).
Techniques such as sparsity (CGRS19, GRK17), branching architectures (KSH12), and dynamic network growth (WRH17, WYL19) offer additional routes to efficient large‑scale training.
Summary of Scaling Laws
Provides concise tables and formulas summarizing the paper’s scaling relationships and compute‑efficient training prescriptions.
This appendix gathers the closed‑form scaling formulas and the numeric tables that support the paper’s predictive framework. Readers can use them as a quick reference when planning model size, data budget, or compute allocation.
**Table 1.** Summary of scaling laws under different constraints.
This table presents power law coefficients and their corresponding tokenization-dependent scales.
**Table.** Compute-Efficient Value, Power Law, and Scale parameters.