Loop the Loopies!

Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, Bryan Dai

Loopie uses recurrent layer-looping and compute-matched scaling to outperform vanilla MoE models under fixed pre-training budgets.

How can we improve parameter efficiency in Mixture-of-Experts models by reusing layers within a single forward pass?

Looped Transformers are often less compute-efficient than vanilla models because repeating a model $N$ times multiplies pre-training costs by $N$, making them difficult to scale against larger, non-recurrent baselines. Loopie addresses this by using a "layer-loop" pattern—iteratively refining hidden states within each layer before moving to the next—and a scaling recipe that converts the resulting memory savings into larger microbatch sizes and increased model capacity. On compute-matched benchmarks, Loopie-20B-A2B consistently outperforms vanilla Mixture-of-Experts (MoE) baselines after 600 billion training tokens, demonstrating that recurrent computation can be a scalable alternative to depth scaling.

Paper Primer

The core innovation is the Loopie Recipe: a procedure that constructs a recurrent model by halving stored layers, executing each twice, and reinvesting the saved activation memory into a doubled per-device microbatch size. This approach aligns the model's training cost with a non-recurrent reference, ensuring that performance gains are driven by effective scaling rather than simply adding more compute.

Layer-looping acts like a local refinement loop: instead of unrolling the entire model stack, each block performs local iteration on hidden states, which improves execution locality and allows for more coherent parameter sharing across effective depth.

Loopie-20B-A2B outperforms compute-matched vanilla MoE baselines.

Benchmark performance on eight downstream tasks (including MMLU and ARC) shows Loopie overtaking the 30B-A3B vanilla baseline after 600B tokens. Consistent advantage maintained across a scaling ladder from 0.15B to 1B parameter baselines.

To reach frontier-level reasoning, the authors introduce Supervised Pre-training (SPT), which applies an SFT-style supervised objective at the scale of pre-training (128M tokens per global batch). This regime avoids the catastrophic forgetting and overfitting typically associated with conventional supervised fine-tuning.

Why does Loopie use only two recurrent steps ($R=2$)?

In the large-scale pre-training regime, the marginal benefit of additional recurrence decays rapidly; $R=2$ provides the optimal trade-off between iterative refinement and training throughput under a fixed compute budget.

How does SPT differ from standard supervised fine-tuning (SFT)?

SPT uses the same loss objective as SFT but operates at the massive scale of pre-training (large batch sizes and token budgets), which prevents the narrow memorization and catastrophic forgetting common in standard SFT.

Introduction and Motivation

We expose why looping models need compute‑matched scaling to beat dense baselines.

Looped Transformers reuse the same computation across multiple recurrent steps, which multiplies the pre‑training FLOP budget by the number of loops. Consequently, a naïve comparison that only matches parameter counts unfairly favors dense models, because the looped model expends far more compute for the same parameter budget.

Loopie models apply a single learned layer‑loop operator repeatedly, so only the parameters of that operator are active at any moment, dramatically shrinking the active‑parameter footprint while retaining the expressive power of a full MoE.

The key shift is from scaling sheer parameter counts to leveraging parameter‑efficient recurrence for high‑performance reasoning.

The Loopie Architecture

Introduces layer‑loop and model‑loop recurrences, explains their benefits, and presents the Loopie scaling recipe.

Loopie’s core contribution is a new recurrence schedule—layer‑loop—paired with a compute‑matched scaling recipe. We first define the two recurrence patterns, then illustrate why layer‑loop is preferable, and finally detail the recipe that turns the insight into a concrete model.

Each transformer layer is applied repeatedly before the hidden state moves to the next layer, so the same parameters refine the representation locally multiple times.

Apply Layer 1 to $h^{(0)}$ → $h^{(1)}$.

Apply Layer 1 again to $h^{(1)}$ → $h^{(2)}$ (first layer‑loop iteration completed).

Pass $h^{(2)}$ to Layer 2 → $h^{(3)}$.

Apply Layer 2 again to $h^{(3)}$ → $h^{(4)}$ (second layer‑loop iteration).

Because the same layer processes the state twice in a row, it can correct its own mistakes before higher‑level transformations see the result.

How does layer‑loop differ from a standard recurrent neural network that also feeds its output back into the same module?

In a standard RNN the recurrence spans the entire network depth at each time step, mixing all layers’ parameters each step. Layer‑loop confines recurrence to a single transformer layer, so only that layer’s parameters are reused consecutively, preserving depth‑specific functionality.

The whole stack of transformer layers is executed, then the entire stack is repeated, so each layer sees a much larger jump in effective depth between its two invocations.

Layer 1 processes $h^{(0)}$ → $h^{(1)}$.

Layer 2 processes $h^{(1)}$ → $h^{(2)}$.

Layer 3 processes $h^{(2)}$ → $h^{(3)}$ (first full pass).

Layer 1 processes $h^{(3)}$ → $h^{(4)}$ (second pass, now at effective depth $L+1$).

Layer 2 processes $h^{(4)}$ → $h^{(5)}$.

Layer 3 processes $h^{(5)}$ → $h^{(6)}$.

The two invocations of Layer 1 occur after very different amounts of computation, forcing the same parameters to serve distinct functional demands.

Why might sharing a single set of parameters across widely separated effective depths be problematic?

Transformer representations evolve non‑uniformly with depth; early layers capture low‑level patterns while later layers encode high‑level semantics. A parameter that must serve both regimes may be forced into a compromise that harms performance.

**Figure 1.** Illustration of the contrast between layer-loop and model-loop. The top panel shows layer-loop, the loop schedule adopted by Loopie, in which each Attention/MoE layer is applied recurrently before passing its output to the next layer. We demonstrate that this schedule achieves better performance on MoE backbones while also training more efficiently. The bottom panel shows the traditional whole-model recurrence pattern, which we call model-loop, where the entire layer stack is traversed and then repeated. This pattern is used by prior looped models such as Ouro and Huginn (Zhu et al., 2025; Geiping et al., 2025).

Layer 1 $arrow$ Layer 1 $arrow$ Layer 2 $arrow$ Layer 2 $arrow$ Layer 3 $arrow$ Layer 3

Layer‑loop brings three concrete benefits: superior long‑term scaling, execution locality that eases pipeline scheduling, and a more coherent parameter‑sharing pattern.

**Figure 2.** Comparison of the average downstream benchmark scores between the layer-loop and model-loop variants of Loopie-6B-A0.6B. Although layer-loop lags behind model-loop early in training, it surpasses model-loop later.

Halve the number of stored layers to form a recurrent seed model.

Apply each stored layer twice using layer‑loop (recurrent depth = 2).

Use the freed memory to double the per‑device microbatch size, then reinvest the resulting training‑step speedup into additional model capacity while keeping optimizer‑step time matched to the non‑recurrent reference.

Performance and Scaling

Loopie models deliver higher accuracy and throughput for the same training time.

Loopie‑20B‑A2B runs at 38 % higher training throughput than the Qwen3‑30B‑A3B baseline while using the same wall‑clock step time.

Figure 5 shows 261.53 TFLOPS/s for Loopie versus 189.65 TFLOPS/s for the baseline.

A baseline is built by adjusting hidden dimension and layer count so that its total FLOP budget matches that of a Loopie candidate while keeping the per‑device microbatch size fixed.

**Figure 3.** Comparison of Loopie-20B-A2B with our reproduction of Qwen3-30B-A3B under matched per-step pre-training wall-clock time in Megatron-LM. The left panel reports the average score across eight downstream benchmarks, while the right panel reports the highest average pre-training throughput achieved for each model during large-scale benchmarking. In the right figure, Loopie-20B-A2B uses EP = 8 and MBS = 2, while Qwen3-30B-A3B uses EP = 8 and MBS = 1. These configurations achieve the highest throughput in a large-scale TP/PP/EP/MBS grid search; the optimal settings may vary across different GPU.

**Figure 4.** Scaling ladder for Loopie. The horizontal axis denotes the parameter count of the non-recurrent baseline, and the vertical axis reports the average score across eight downstream benchmarks. The Loopie models are not plotted at the parameter counts indicated on the horizontal axis; instead, each Loopie model is sized according to the Loopie Recipe so that its per-step pre-training wall-clock time exactly matches that of the corresponding baseline.

**Figure 5.** Layer-loop ablation for Loopie-6B-A0.6B. We report the average score across eight downstream benchmarks for Loopie-6B-A0.6B and Loopie-6B-A0.6B-Ablation. Loopie-6B-A0.6B-Ablation matches Loopie-6B-A0.6B in all architectural details except for the removal of layer-loop recurrence.

**Figure 6.** Comparison between $N \times$ layer-loop and $N \times$ stored-layer scaling. We report the average score across eight downstream benchmarks after training each model on 250B tokens while keeping all other architectural details fixed. The shaded gray region denotes the gap between the $N \times$ layer baseline and the $N \times$ loop model.

Loopie models achieve superior reasoning performance per unit of wall‑clock training time.

Pre-Training and Annealing

We pre‑train Loopie models in two stages and detail the high‑quality annealing mixture.

Loopie models are pre‑trained in a two‑stage regimen designed to maximise exposure to high‑quality text while keeping the training dynamics stable.

Data Annealing builds a curated mixture of the most reliable sources—synthetic reasoning, STEM, code, mathematics, and web text—to fine‑tune the model after the bulk pre‑training phase.

Stage 1: train on the Nemotron‑CC‑v2‑HQ corpus (≈ 570 B unique tokens) for four epochs, yielding ≈ 2.28 T training tokens.

Stage 2: construct a 1.26 T‑token annealing pool (Figure 9) that blends SFT‑style, synthetic, code, web, and math data according to the percentages shown.

Initialize embeddings $E$ and language‑model head $W_{\text{lm}}$ with variance‑scaled Gaussian ($\mathcal{N}(0,d^{-1})$) and hidden weights $W$ with SmallInit ($\mathcal{N}(0,2/(5d))$).

Apply AdamW with peak LR $5\times10^{-5}$, warmup (6 k steps for 20B‑A2B, 2 k steps for 6B‑A0.6B), then keep the LR constant for the remainder of both stages.

Run training on the Megatron‑LM framework using a global batch size of 1024 and sequence length 8192.

After each stage, evaluate checkpoints with the LM Evaluation Harness on eight benchmarks (ARC‑Challenge, ARC‑Easy, BoolQ, CommonsenseQA, HellaSwag, MMLU, OpenBookQA, WinoGrande) and report the mean score.

**Figure 7.** Composition of the Stage-2 high-quality annealing data pool: multiple data sources make up the 1.26T-token annealing recipe.

Supervised Pre-Training

How Loopie Thinking is built from a supervised pre‑training boost and a large‑scale RL polish.

Loopie models reuse a learned internal operator (the layer‑loop) inside a single forward pass, cutting active‑parameter cost dramatically. After the high‑quality annealing stage, we finish the model with two post‑training phases that add broad task competence and refined reasoning.

**Figure 8** Overview of the Loopie training pipeline. The model is first pre-trained and annealed to produce Loopie Base, which then undergoes supervised pre-training and Math RL to produce Loopie Thinking.

SPT keeps the fine‑tuning loss (only target tokens count) but scales the optimization to full pre‑training batch sizes and token budgets, letting the model absorb massive supervised data without forgetting its earlier knowledge.

Mask vector for each sequence: $[0,0,1,1]$ (only positions 3 and 4 contribute to loss).

Cross‑entropy for sequence 1: $-\log p_\theta(t_3|t_{<3}) - \log p_\theta(t_4|t_{<4})$.

Cross‑entropy for sequence 2 computed analogously; the batch loss is the sum of the four target‑token terms.

Because the batch size is 2 × 4 = 8 tokens, the gradient averages over eight target‑token contributions, illustrating the “large‑batch” effect on a tiny scale.

Even with a minimal toy batch, SPT’s loss only sees the supervised targets, yet the gradient aggregates over the full batch, demonstrating how scale and masking are orthogonal.

How does SPT differ from ordinary supervised fine‑tuning (SFT) if both use the same loss mask?

Both mask out prompts, but SPT runs the masked loss at pre‑training scale (global batch ≥ 1 024, sequence length up to 128 K, token budget ≥ 2 T). SFT uses tiny batches and short contexts, which leads to rapid over‑fitting and catastrophic forgetting. The scale, not the mask, is the decisive difference.

Table 2 shows that SPT improves both pre‑training metrics (↑) and reasoning metrics (↑) while avoiding the over‑fitting observed in SFT; PT improves only the former and leaves reasoning unchanged.

GSPO treats an entire generated response as a single decision unit, normalising its probability ratio by length and clipping the whole sequence’s advantage, which aligns the optimisation granularity with the reward signal.

Why does GSPO clip the sequence‑level ratio instead of the usual token‑level PPO clip?

Because the reward is defined for the whole response, clipping at the sequence level preserves the alignment between the unit of optimisation (the response) and the unit of feedback (the reward). Token‑level clipping would distort this alignment and introduce unnecessary variance.

GSPO training loop – one batch of prompts.

Tables 3 and 4 demonstrate that after the GSPO stage, Loopie Thinking matches or exceeds much larger MoE baselines while using a fraction of the pre‑training token budget.

Related Work

We survey prior architectures and analyses that reuse a shared transformation across depth.

We call a “looped model” any architecture that reuses a learned internal operator across a single forward pass instead of stacking distinct layers. Early examples include Neural GPUs, Adaptive Computation Time (ACT), Universal Transformers (UTs), and Deep Equilibrium Models (DEQs), all of which treat depth as a repeated application of a shared transformation. More recent designs such as ALBERT, tied Transformers, depth‑adaptive Transformers, feedback memory, recurrent‑memory Transformers, and block‑recurrent Transformers continue this lineage, and specialized variants like MoEUT, Relaxed Recursive Transformers, and implicit state‑space language models further explore parameter‑efficient depth reuse.

Looped depth introduces inductive biases absent from a stack of independent Transformer layers. Sharing the recurrent operator forces a reusable update rule, which aligns with fixed‑point iteration, gradient‑based in‑context learning, normalized gradient descent, and power iteration. The bias also decouples computation from parameter count, enabling iterative refinement, and it encourages algorithmic thinking, knowledge re‑access, compositionality, and the emergence of a latent scratchpad that can be allocated adaptively to harder inputs.

Theoretical work shows that shared‑depth computation can execute instruction‑level programs, simulate latent chains of thought, and achieve superior approximation rates via timestep modulation. Analyses of normalized‑gradient updates for in‑context logistic regression and power‑method dynamics under layer normalization illustrate concrete algorithmic capabilities. At the same time, compressed recurrent states create a memory‑budget separation, and set‑valued or graph‑structured tasks expose limits around fixed‑point convergence and halting criteria. Reinforcement‑learning perspectives further clarify how internal computation expands the class of compute‑bounded policies.

Recent architectures treat recurrence as a third scaling axis. Huginn and Ouro pre‑train recurrent‑depth language models; retro‑fitted recurrence and LoopUS convert feed‑forward models into latent‑refinement systems. HRM‑Text, CHERRY, Hyperloop, LoopMoE, sparse looped layers, and universal expert pools combine iterative depth with conditional capacity. Dynamic‑depth routing appears in Mixture‑of‑Recursions, CoTFormer, AdaPonderLM, Think‑at‑Hard, Chain‑of‑Layers, and LoopFormer, while stability‑focused designs such as Parallel Loop Transformers, LT2, MELT, LASER, Parcae, CART, and stochastic stopping address drift and convergence at high loop counts.

Looped Transformers provide a natural substrate for in‑context algorithmic learning. Bayesian, gradient‑descent, preconditioned‑gradient, linear‑model, and causal‑structure accounts interpret each recurrent step as an additional internal update, while program‑simulation and normalized‑gradient analyses demonstrate concrete learning dynamics. By iterating a small rule set, these models excel at graph‑algorithm simulation and length generalization, especially when loop count scales with problem size; multi‑resolution recursion and depth‑recurrent compositional models further mitigate the degradation of a stationary update at deeper depths.

Abstract‑reasoning benchmarks such as ARC highlight the need for iterative abstraction. Systems like HRM, TRM, URM, CosmicFish‑HRM, Recursive Inference Machines, Fixed‑Point Reasoners, and various generative, probabilistic, and denoising recursive models demonstrate that repeated application of a compact core can acquire new transformations from few examples. Equilibrium and attractor reasoners, Tiny Autoregressive Recursive Models, memory‑augmented UTs, LoopViT, and interaction‑locality analyses further explore how recurrence supports systematic compositionality beyond memorization.

Conclusion and Limitations

We discuss current limitations, future research avenues, and recap Loopie's key contributions.

Due to computational constraints, our post‑training experiments focus primarily on mathematical and code reasoning tasks. We do not extensively explore other important capabilities studied in prior work, such as scientific question answering, instruction following, alignment with human conversational preferences, or agentic task‑solving. A natural direction for future work is to continue to post‑train Loopie on a broader set of tasks and preference signals, which may further improve its practical utility and general‑purpose capabilities.

Focusing post‑training on math and code yields strong performance on those tasks but leaves broader capabilities under‑explored; balancing task‑specific depth with general‑purpose ability is a key trade‑off for future work.

Also, due to limited computational resources, we were unable to conduct a sufficiently comprehensive ablation study of supervised pre‑training, which we leave for future work.

In addition, our study focuses primarily on matching compute budgets during pre‑training, and we have not yet conducted systematic studies of inference‑time computation. Matching and optimizing inference‑time compute remain important directions for future work; prior work such as the Parallel Loop Transformer (Wu et al., 2025) provides a promising example of this approach.

Finally, our study intentionally focuses on applying looped Transformers to a relatively clean and well‑controlled base architecture, Qwen3‑30B‑A3B, in order to avoid confounding factors introduced by architectural variations. As a result, we do not investigate several recent architectural advances that may be complementary to our method. Exploring how Loopie interacts with these newer designs remains an important direction for future work.

We introduced Loopie, a family of looped MoE language models that makes recurrent depth competitive under a matched pre‑training compute budget. By combining layer‑loop recurrence with a hardware‑aware scaling recipe, Loopie consistently outperforms compute‑matched vanilla Transformer baselines across model scales.

A large‑scale post‑training pipeline based on supervised pre‑training and reinforcement learning further equips Loopie with strong mathematical reasoning and coding abilities. These results suggest that recurrent computation, when jointly optimized with architecture and training efficiency, can serve as a practical scaling axis for large language models.

Loopie demonstrates that recurrent computation, paired with hardware‑aware scaling, offers an efficient path to improve large language models without increasing overall compute.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers