Understanding Reasoning from Pretraining to Post-Training

Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun, Micah Goldblum, Matus Telgarsky, Pavel Izmailov

A controlled chess testbed reveals how pretraining scale and reinforcement learning compute interact to shape reasoning.

How do pretraining choices (compute, model size, data) quantitatively determine the efficiency and performance of subsequent RL post-training for reasoning tasks?

Large language model training splits compute between pretraining and reinforcement learning (RL), but it is unclear how these stages interact or how to optimally allocate resources between them. The authors use chess as a controlled testbed to systematically sweep pretraining and RL compute, establishing a joint scaling law that predicts post-RL performance from pretraining loss and data scale. They find that while RL improves performance, its effectiveness is initialization-dependent: as total compute grows, the optimal strategy shifts toward allocating a larger fraction to RL.

Paper Primer

The authors instantiate a standard training pipeline—pretraining on human games, supervised fine-tuning (SFT) on synthetic reasoning traces, and RL on verifiable puzzles—to isolate the mechanics of reasoning. This setup allows them to measure how pretraining choices, such as model size and token count, dictate the subsequent returns on RL compute.

Pretraining loss is a strong predictor of post-RL performance.

Across 36 pretraining-RL combinations, lower pretraining validation loss consistently corresponds to higher pass@1 performance after RL, with Spearman correlation reaching $|\rho| = 0.99$ at high compute levels. The relationship is non-linear, with pretraining providing the necessary initialization strength for RL to be effective.

RL does not uniformly sharpen the policy; its effects are state-dependent.

On easy puzzles, RL amplifies correct moves already preferred by the SFT policy; on hard puzzles, it surfaces previously absent correct moves but also reinforces incorrect ones. This heterogeneous redistribution explains why RL improves pass@1 without consistently improving pass@k.

Why use chess to study reasoning instead of natural language?

Chess provides a compact, verifiable action space where move quality can be checked exactly, allowing the authors to isolate policy changes at individual reasoning steps—a task that is opaque in natural language due to vast, ambiguous action spaces.

Does this scaling law hold for language models?

Yes; the authors trained a 1B-parameter language model on a math-domain corpus and observed the same pattern: longer pretraining checkpoints achieved higher post-RL performance and exhibited steeper RL scaling slopes.

Introduction: The Pretraining-RL Gap

We expose how pretraining quality governs the returns of reinforcement‑learning fine‑tuning.

Reinforcement Learning (RL) has become central to improving large language models on complex reasoning, yet RL post‑training is studied almost entirely apart from the preceding pretraining phase. This leaves two basic questions open: (1) how do pretraining choices—model size, data volume—shape the returns to RL compute, and (2) what does RL actually do to the inherited policy? The difficulty stems from massive, uncontrolled pretraining corpora and the prohibitive cost of systematic compute sweeps across both stages.

RL post‑training performance is not an independent boost; it is a predictable function of the loss and compute invested during pretraining.

To isolate these effects we adopt chess as a compact, fully observable testbed that mirrors the standard LLM pipeline: we pretrain models ranging from 5 M to 1 B parameters on tokenized human games, apply Supervised Fine‑Tuning (SFT) on synthetic reasoning traces, and finally run RL on chess puzzles with verifiable rewards.

Across 36 pretraining‑RL combinations we discover a joint scaling law: the post‑RL performance at a given RL compute level is well‑predicted from the pretraining loss, and the local RL reward‑curve slope improves roughly linearly with the logarithm of pretraining token count. Moreover, RL does not merely sharpen the SFT policy; on easy puzzles it amplifies moves the SFT already prefers, while on hard puzzles it can surface correct moves that were nearly absent, albeit sometimes reinforcing incorrect ones.

The key insight is the disconnect: RL gains depend critically on the quality of the pretrained prior, so compute allocation must consider this pretraining‑RL interface.

The Chess Reasoning Testbed

We introduce a chess‑based framework that links pretraining, synthetic reasoning, and RL to study policy evolution.

Pretraining on human games yields a strong move‑generation prior, yet it never learns to articulate a reasoning process. Consequently, downstream RL struggles to improve because the policy has no explicit trace of why a move is correct. Our framework injects a synthetic reasoning stage that bridges this gap, enabling RL to refine policies with verifiable, step‑wise feedback.

The framework treats chess like a language‑model pipeline: first learn move distributions from human games, then teach the model to write its own “thought process” before committing to a move, and finally let RL polish the policy using a binary puzzle reward.

All three share the first move a₁ (or c₁), so we create a root node with two children: one for a₁ and one for c₁.

Under the a₁ branch we add two leaves: one leaf holds the suffix a₂, the other holds b₂.

Under the c₁ branch we add a single leaf with suffix c₂.

Depth‑first serialization yields r =

This compact tree captures all distinct continuations while avoiding repeated storage of the shared prefix, which is why the SFT model can learn to reason over many possibilities without exploding sequence length.

**Figure 1. Overview.** (a) We introduce a synthetic framework for studying reasoning from pretraining to post-training in the chess domain. (b) Using this framework, we establish a joint pretraining–RL scaling law, showing that pretraining performance provides predictive signal for RL performance under a fixed compute budget. (c) Through mechanistic analysis of policy evolution, we show that RL can surface moves that were nearly absent under the SFT policy.

How does the synthetic reasoning trace differ from a standard chain‑of‑thought prompt?

Standard CoT asks the model to generate a single linear explanation after seeing the input. Our trace first samples multiple plausible continuations, merges shared prefixes into a tree, and then serializes the whole tree. The model therefore learns to reason over a set of alternatives rather than a single imagined line, which aligns better with the RL environment where only the exact ground‑truth move earns reward.

Scaling Laws for Reasoning

Scaling analysis links pretraining loss and compute to RL performance gains.

RL compute share grows from 5 % to 32 % as total compute increases for the 20 M model, a 27 pp rise.

Figure 2 shows the Pareto frontier points and labels the RL‑compute fraction at each point.

Pretraining loss sets a baseline reward, while the amount of pretraining data determines how quickly RL can improve that reward as compute grows.

How does this joint law differ from the previously proposed sigmoid RL scaling law?

The sigmoid law models the entire RL curve and includes a saturation plateau, requiring long runs to estimate the asymptote. Our joint law replaces the sigmoid with its linear approximation valid before saturation and adds explicit dependence on pretraining loss and token count, making it predictive under realistic compute budgets.

**Figure 2.** Empirical frontier of puzzle benchmark performance across pretraining-RL sweeps for four model sizes (20M, 50M, 200M, 680M). Each point is a checkpoint evaluated on the puzzle benchmark. Color indicates pretraining compute; open rings mark the pre-RL baseline. Black curves mark the Pareto frontier. For the pass@1 curves, we label the fraction of RL compute at which each run first reaches the frontier (excluding the smallest and largest compute ranges). For pass@1, RL consistently improves performance and the labeled RL ratio shows an increasing trend as total compute increases across models. For pass@16, gains from RL are smaller and additional pretraining is often more effective.

**Figure 3.** Pretraining properties predict local RL scaling behavior. (a) $R_{N,T}^{ref}$, the fitted post-RL performance (pass@1 metric) at a reference RL compute level, versus pretraining validation loss. Each curve corresponds to a different reference compute level; the fit tightens as RL compute increases ($\rho=-0.93$ to $-0.99$). (b) $B_{N,T}$, the local slope measuring performance gain per decade of RL compute, versus pretraining tokens. More tokens predict faster RL improvement. (c) Joint fit of the slope using both $\log T$ and $\log N$, achieving higher $R^2$.

**Figure 4.** Extrapolated compute-optimal frontier across model sizes using the fitted law. (a) Simulated compute-optimal reward ($R_N^\star(C)$) for each model size. The black curve is the global simulated frontier, dots mark model-size transitions, and the dashed gray curve is the empirical frontier. The two frontiers closely agree. (b) Compute-optimal RL share ($\rho_{RL} = C_{RL}^\star / C_{total}$). The optimal RL fraction increases with total compute, from approximately 20% at 50M to 28% at 680M; the red dashed curve shows the continuous-(N) optimum. (c) Optimal pretraining tokens relative to the Chinchilla allocation at the same pretraining compute. The results do not show systematic deviation from Chinchilla scaling.

Pretraining loss reliably predicts RL performance, and more pretraining data accelerates RL gains.

Policy Evolution During RL

We dissect how RL reshapes the move policy and reasoning traces, revealing a targeted evolution mechanism.

RL post‑training does more than tweak a temperature; it systematically reshapes the move distribution and the reasoning process that generates those moves. Understanding this reshaping is essential before we can improve downstream performance.

RL acts like a sculptor refining a rough statue: it sharpens the peaks of the pre‑RL policy (amplifying already‑preferred moves) while sometimes over‑emphasizing flaws (reinforcing wrong moves) and carving out new correct features from the low‑probability tail.

Apply a power exponent $\alpha$ = 1.5: raise each probability to 1.5 → [0.253, 0.164, 0.089, 0.032].

Normalize the transformed scores: sum = 0.538, yielding RL probabilities [0.47, 0.30, 0.17, 0.06].

With k = 3, the top‑k set after RL is {a, b, c}; the correct move c has moved from rank 4 to rank 3 (tail discovery).

Move a remains the top probability; if a is incorrect, this illustrates wrong‑mode amplification.

Ground‑truth amplification would occur if c had already been in the top‑k and its probability increased further.

This toy example shows how a single exponent $\alpha$ can simultaneously promote a previously hidden correct move and boost an existing wrong preference.

How does this differ from the usual temperature scaling applied after supervised fine‑tuning?

Temperature scaling multiplies all logits by a constant $\tau$, preserving their relative ordering; the power transform raises each probability to $\alpha$ and then renormalizes, which can change the ranking because low‑probability entries are suppressed more strongly than high‑probability ones. Hence RL can promote a correct move from the tail, something pure temperature scaling cannot achieve.

Fitting the power‑transform slope $\alpha$ to the data (Table 12) shows a clear upward trend: $\beta$ grows from 0.60 at the pre‑RL stage to ≈1.13 after 750 RL steps, confirming overall sharpening. However, the modest R² values (≈0.5) and large per‑state variance indicate that RL’s effect is highly state‑dependent.

RL sharpening slope $\beta$ exceeds 1 after 750 steps, meaning the policy becomes sharper than the SFT baseline.

Table 12 reports a global $\beta$ of 1.13 with R² ≈ 0.68 at the `RL_750` stage.

**Figure 5** RL reshapes the move policy in qualitatively different ways across puzzle difficulty. Each panel shows the proportion of puzzle states assigned to one policy-update category (Table 13) across difficulty bins B1-B5, at RL training steps until 750. On easy puzzles, ground-truth amplification (a) dominates. On harder puzzles, tail discovery (b) and wrong-mode amplification (c) both increase, showing that RL simultaneously surfaces previously absent correct moves and reinforces incorrect ones.

Beyond move probabilities, RL also reshapes the chain‑of‑thought dynamics: the width‑to‑depth ratio of the reasoning tree grows, indicating broader search, while the maximum depth stays flat, showing limited long‑horizon planning.

Transfer to Natural Language

We show that chess‑derived scaling laws predict RL gains for a 1B‑parameter language model.

Recall that RL post‑training performance is not independent of pretraining; it scales predictably with pretraining loss and compute. We now test whether this relationship transfers to natural‑language models.

Across 14 checkpoints spanning 10 B–200 B pretraining tokens, post‑RL performance correlates tightly with pretraining loss.

Figure 6 (left panel) shows a monotonic decline of $R_T^{ref}$ with loss for each compute bucket, with high $R^2$ values.

**Figure 6.** The predictive pattern extends to the math domain. Across 1B OLMo-2 checkpoints from 10B to 200B pretraining tokens, lower pretraining loss consistently predicts higher post-RL performance $R_T^{ref}$, with the fit tightening as RL compute increases. The slope $B_T$ increases nearly linearly with $\log_{10} T$. The $R_T^{ref}$ relationship with pretraining loss becomes tighter as RL compute increases, mirroring the chess results (Fig. 3).

The study measures whether the scaling relationship discovered in chess also predicts RL gains for a 1 B‑parameter language model trained on mixed math and general text.

The chess‑derived scaling laws hold for 1 B‑parameter language models.

Asymptotic RL Performance

We fit logistic RL curves to 20 M runs and extrapolate compute‑optimal allocations.

We fit a per‑run logistic curve to each 20 M‑parameter run to estimate the RL asymptote and then use the fitted law to explore compute‑optimal allocations.

RL compute share $\rho_{\text{RL}}$ grows from ≈ 20 % at 50 M to ≈ 28 % at 680 M as total compute increases.

Figure 20 shows a monotonic upward trend of the optimal RL fraction across the compute range.

**Figure 19** Asymptote $A_{\infty}$ of the per-run logistic RL learning curve for the 20M-parameter family, plotted against three pretraining covariates. Each point is one of the ten 20M runs; markers are coloured by the pretrain-tokens fraction $\alpha$, vertical bars are 90% parametric-bootstrap CIs (500 resamples per run), and the dashed line is an ordinary-least-squares fit on the displayed sample. (**left**) ceiling versus pretrain eval loss $L$: better-pretrained checkpoints support strictly higher RL asymptotes (Spearman $\rho=-0.73$, $R^2=0.90$). (**middle**, **right**) ceiling versus $\log_{10}(T/N)$ and $\log_{10} T$: both show a positive trend ($\rho=+0.73$, $R^2=0.80$ each)

**Figure 20** $\epsilon$-sensitivity of the compute-optimal RL fraction. Black: argmax $\rho_{RL}$ along the continuous-$N$ optimum. Shaded: the spread of $\rho_{RL}$ over allocations within $\epsilon$ reward of the optimum, $\mathcal{A}\epsilon = \hat{R} \geq \hat{R}_{max} - \epsilon$, for $\epsilon = 0.001$ (dark) and $0.005$ (light). The reward surface is flat near its peak; however, the whole band shifts upward with compute, so the increasing-RL-share trend is robust.

Policy Evolution Metrics

Ablations reveal that RL mainly sharpens the SFT policy and reshapes move distributions across training stages.

All policy‑evolution metrics are evaluated in move space: each token prefix maps to a legal board state, and every legal move has a unique token serialization.

Power‑sharpening is like turning up the volume on the moves the model already prefers – the distribution becomes more peaked without inventing new moves.

Is power‑sharpening just a temperature scaling of the SFT logits?

No. Temperature scaling multiplies all logits by the same constant, which preserves their relative ordering but does not change the normalisation constant per state. Power‑sharpening first raises each probability to $\alpha$, then renormalises, which can amplify differences more aggressively for high‑probability moves while suppressing low‑probability tails.

Global sharpening coefficient $\alpha^*$ rises from 0.57 before SFT to 1.35 after RL₇₅₀, a net increase of +0.78.

Table 11 reports $\alpha^*$ = 0.57 for the pre‑train → SFT transition and $\alpha^*$ = 1.35 for the SFT → RL₇₅₀ transition.

ExplainedSharp improves from –0.01 to 0.17 across the same span, adding +0.18 of the RL‑SFT divergence.

Table 11 shows ExplainedSharp = –0.01 for pre‑train → SFT and 0.17 for SFT → RL₇₅₀.

The centered‑logit scaling slope $\beta$ grows from 0.60 to 1.13, a gain of +0.53, confirming logit‑space sharpening.

Table 12 lists $\beta$ = 0.60 for pre‑train → SFT and $\beta$ = 1.13 for SFT → RL₇₅₀.

R² of the linear fit rises from 0.40 to 0.56, a modest +0.16 improvement in explanatory power.

Table 12 reports R² = 0.40 for the pre‑train → SFT stage and 0.56 for the SFT → RL₇₅₀ stage.

**Figure 21. Policy evolution on a hard puzzle (B5) across training stages.** Left: board position with ground-truth move b6e3 (green arrow). Pretrain: the model assigns ~95% mass to a wrong move (d6e5), with b6e3 nearly absent. SFT: probability spreads across many candidates but b6e3 (red) remains low. RL: over training steps 50–750, b6e3 rises to become the top move (tail discovery), while a competing wrong move (b6b4) also retains significant mass (wrong-mode amplification).

**Figure 22. Structured reasoning traces evolve during RL training.** For the 20M and 50M models, panels report per-step prompt means on the puzzle benchmark. RL modestly increases search breadth, improves move quality scored by Stockfish rank for both model moves (solid) and predicted opponent replies (dashed), and increases ground-truth move coverage in the parsed CoT tree. DFS consistency decreases and revisit rate increases, indicating more re-exploration of earlier prefixes.

**Figure 23.** Reasoning length dynamics over RL training.

**Figure 24** **Search shape and selected move quality over RL training for the 20M and 50M models.** Curves show mean metrics for the 20M and 50M models, with standard error across prompts for the selected moves quality. Columns report effective branching factor $b_{\text{eff}} = N^{1/D}$, maximum tree depth $D$, and commit quality, a per-prompt z-score measuring the Stockfish rank of the committed move relative to both the best move and the best considered move. Branching increases modestly, depth decreases slightly, and commit quality improves monotonically.

**Figure 25** Search-tree coverage of ground-truth continuations by depth for the 20M and 50M models. For each rollout, we measure whether the parsed CoT search tree contains a root-to-depth-k path whose player moves match the first k target plies, allowing arbitrary legal opponent replies. Panels report coverage for k = 1, 2, 3, 4, excluding rollouts with shorter targets. RL improves coverage at all depths, but gains decay sharply with depth; even at step 1000 for the 50M model, few rollouts recover the full 4-ply target line.

Table 13 formalises six policy‑update categories (ground‑truth amplification, tail discovery, top‑k correction, ground‑truth regression, wrong‑mode amplification, other) that capture the qualitative effects observed in the $\alpha^*$ and $\beta$ trends.

Discussion and Limitations

We outline the key constraints limiting the generality of our chess‑based findings.

Our testbed follows the standard LLM pipeline (pretraining on human data, SFT on reasoning traces, RL with verifiable rewards), but chess differs from natural language in vocabulary size (81 tokens), exact verification, and lack of world‑knowledge entanglement. Consequently, the scaling exponents and category proportions we report describe the structure of the pretraining‑to‑RL interface in a controlled setting, not quantitative predictions for language models, and the binary‑reward puzzles we use are a restricted verification form compared to the partial‑credit or open‑ended rewards common in language tasks. Moreover, our models top out at 1 B parameters, our tree‑based CoT serialization may bias dynamics, and the fitted scaling law (see Section G.8) may not hold at larger scales or with alternative reasoning‑trace formats.

Implementation Details

This section details the datasets, model architectures, and training configurations used for pretraining, SFT, and RL.

The experimental pipeline relies on a curated set of Lichess data, ranging from raw game logs for pretraining to difficulty-stratified puzzles for post-training. All models follow a consistent Qwen-style decoder-only Transformer architecture, scaled from 5M to 1B parameters using grouped-query attention.

**Table 1. Test benchmark details.** The Lichess puzzle subsets evaluate tactical performance across Elo difficulty, puzzle themes, and solution depths.

**Table 2. Model architecture details.** All models use a Qwen-style decoder-only Transformer architecture with grouped-query attention.

Training proceeds in three stages: pretraining on 54B tokens, Supervised Fine-Tuning (SFT) on reasoning traces, and Reinforcement Learning (RL) using Group Relative Policy Optimization (GRPO). The SFT objective optimizes the model to predict synthetic reasoning traces, while GRPO refines the policy by maximizing binary rewards based on puzzle solutions.

This table lists the configuration parameters used for training.

To teach the model how to reason, we construct synthetic traces by rolling out multiple move sequences from a starting position and serializing them into a prefix tree.

FLOP Estimation

Quantifies compute costs and fits scaling laws for pretraining and RL.

We estimate training compute in FLOPs using the dense‑transformer rule $C_{\text{train}}(N,T) \approx 6NT$, which counts the dominant matrix multiplications in each forward and backward pass.

We swept 11 compute budgets from $6.5\times10^{16}$ to $6.5\times10^{19}$ FLOPs across 10 model sizes (≈ 200 M–52 B tokens). Each budget exhibits a clear validation‑loss valley, confirming an optimal parameter‑token allocation. Compared with the Chinchilla rule $D/N=20$, chess pretraining prefers smaller models and more tokens; the gap narrows at higher compute.

**Figure 8** IsoFLOP curves for pretrained models across a sweep of 11 compute budgets and 10 model sizes. (a) Pretraining validation loss (NLL) on held-out human games. (b) Pass@1 and (c) Pass@16 on the puzzle benchmark. The starred points mark the compute-optimal model size at each budget. The solid line connects these optima, while the dashed line represents the Chinchilla allocation (D/N = 20). Over the plotted range, the chess-optimal locus lies at smaller models and more tokens than this rule, with the two converging at higher compute.

We fit the Hoffmann‑style parametric loss $L(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}$ to 64 $(N,D,L)$ triples below loss 1.0, obtaining $\alpha = 0.4006\pm0.0399$, $\beta = 0.6789\pm0.0291$, and RMSE 0.0097. The exponents imply compute‑optimal scaling $N_{\text{opt}}(C)\propto C^{0.63}$ and $D_{\text{opt}}(C)\propto C^{0.37}$.

Table 5 lists the fitted parameters alongside Hoffmann et al.’s language‑modeling estimates, highlighting the markedly larger data exponent $\beta$ for chess.

For supervised fine‑tuning we compare (1) SFT without reasoning traces and (2) SFT with synthetic reasoning traces. Across all model sizes, adding reasoning traces boosts Pass@8 and Pass@16 substantially, whereas the no‑reasoning variant improves Pass@1 only and leaves higher‑k scores flat.

**Figure 9** Effect of SFT with and without reasoning traces across four model sizes. Each panel shows puzzle benchmark performance (B1–B5, macro-averaged) as a function of pretraining compute. The star marks the compute-optimal pretraining budget from the isoFLOP fit.

**Figure 10** Effect of SFT with and without reasoning traces across four model sizes evaluated with pass@4 and pass@8. Each panel shows puzzle benchmark performance (B1-B5, macro-averaged) as a function of pretraining compute. The star marks the compute-optimal pretraining budget from the isoFLOP fit.

Statistical Scaling Framework

Statistical model linking pretraining scale, loss, and RL compute.

We model the post‑RL reward as a function of three pretraining quantities—model size $N$, token count $T$, and validation loss $L_{\text{pt}}(N,T)$—plus the RL compute $C_{\text{RL}}$.

The RL reward curve can be expressed as a baseline set by pretraining loss plus a linear gain that scales with RL compute, where both baseline and gain are predicted from $N$, $T$, and $L_{\text{pt}}$.

Pretraining loss follows a power‑law surface in $(N,T)$ that predicts the optimal token‑per‑parameter ratio for a given compute budget.

Leave‑one‑out (LOO) validation over the 36 runs shows a strict Chinchilla‑L RMSE of 0.0194 reward; switching to observed loss raises RMSE by 0.0099 (≈ 49 %). The increase isolates the contribution of loss‑prediction error.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers