Decoding Looped Transformers Better for (Almost) Free
Weihao Liu, Huangjie Zheng, Tianrong Chen, Rohit Dilip, Richard He Bai, Yizhu Jiao, Yuyang Wang, Ruixiang Zhang
LoopCD turns intermediate recurrent states into contrastive guidance, boosting reasoning accuracy while cutting compute.
How can we improve the performance of Looped Transformers by using their intermediate recurrent states to guide final predictions?
Looped Transformers repeatedly execute a shared block to scale computation, but standard decoding discards all intermediate states except the final one. LoopCD treats the first recurrent pass as a weak reference and the final pass as a strong prediction, contrasting them to guide token selection along the model's own refinement trajectory. This training-free method consistently improves reasoning and code generation, enabling models to match full-depth performance with half the recurrent iterations and up to 48% fewer FLOPs.
Paper Primer
LoopCD implements contrastive decoding by extrapolating the difference between an early, "weak" recurrent state and the final, "strong" state. It operates via two mechanisms: LoopCD-Logits contrasts the final output distributions, while LoopCD-Hidden combines representations before the final layers to eliminate extra output overhead.
LoopCD significantly improves reasoning performance at full recurrent depth.
On the AIME 2024 benchmark, Ouro-2.6B-Thinking's pass@1 accuracy increased from 61.88% to 73.33%.
LoopCD enables substantial compute savings by reducing required recurrent iterations.
Models operating at half their original recurrent depth with LoopCD match or exceed the accuracy of full-depth unguided baselines.
Why does this approach work without auxiliary models or extra training?
Because looped Transformers repeatedly apply the same parameter block, the recurrent trajectory inherently produces aligned weak-to-strong prediction pairs within the same semantic space, which LoopCD simply exploits at inference time.
How does the adaptive guidance strength rule function?
It scales the guidance coefficient based on the probability margin between the top two token candidates, applying stronger guidance when the model is uncertain and withholding it when a decision is already settled.
Abstract
Looped Transformers reuse intermediate states to guide decoding, yielding higher quality with less compute.
Looped Transformers execute the same block repeatedly, producing intermediate states that can be decoded for the next token. These early states act as cheap “weak” predictions, while the final state provides a “strong” prediction. LoopCD contrasts the weak and strong predictions, delivering sizable accuracy gains (e.g., Ouro‑2.6B‑Thinking AIME 2024 from 61.88 % to 73.33 %) while cutting inference FLOPs by up to 48.2 %.
A Looped Transformer repeatedly applies a single neural block, feeding its own output back as input, so each pass refines the same token representation.
**Figure.** Change from the unguided model against the fixed logit strength, five models, on GSM8K, MMLU-Pro, HumanEval, and MBPP; a symmetric-log axis keeps the fall beyond $\omega \approx 0.6$ in frame. Huginn's GSM8K runs use chain of thought.
LoopCD improves performance by reusing intermediate states.
The Case for Recurrent Guidance
Motivates exploiting intermediate states of looped Transformers for contrastive decoding.
Looped Transformers repeatedly apply a shared block of layers, scaling effective depth with recurrent iterations while keeping the parameter count fixed. Existing decoding discards the hidden states produced at intermediate passes, even though each can be decoded into a valid token distribution. This gap motivates the introduction of LoopCD, a training‑free contrastive decoding method that leverages the weak‑to‑strong trajectory of intermediate representations.
LoopCD: Contrastive Decoding Mechanism
How LoopCD refines predictions by contrasting early and final recurrent states.
Recurrent passes generate a “weak” prediction after few iterations, while the fully‑run state yields a “strong” prediction—but running the full pass is costly. LoopCD exploits the cheap weak prediction as a reference to sharpen the strong one without extra compute.
LoopCD treats the earliest recurrent state as a weak reference and pushes the final state away from it, producing a guided prediction that is stronger yet almost free.
Compute the difference $h_{2} - h_{0} = (0.4,\,0.4)$.
Scale the difference by $\omega$: $0.5 \times (0.4,\,0.4) = (0.2,\,0.2)$.
Add to the reference: $h' = h_{0} + (0.2,\,0.2) = (0.3,\,0.4)$.
Feed $h'$ through the coda and head to obtain guided logits $z'$ (e.g., a softmax over vocabulary).
The guided hidden state $h'$ lies midway between the weak and strong states, preserving information from both while biasing the final prediction toward the stronger signal.
How does LoopCD differ from classifier‑free guidance used in diffusion models?
Classifier‑free guidance perturbs an unconditional generation toward a conditional one, whereas LoopCD uses an *earlier* recurrent state of the *same* model as the reference. The contrast is internal to the language model and operates on either hidden or logit representations, not on separate conditional/unconditional streams.
Ouro repeats its entire decoder stack multiple times, making the whole model looped. Huginn inserts the recurrent block between a prelude and a coda, allowing the looped block to receive a fixed prelude representation. Parcae learns its own recurrent dynamics, while Looped‑Qwen3 applies damped updates to a frozen middle‑layer window.
$\omega$ determines how aggressively the final prediction is pushed away from the weak reference; larger $\omega$ yields a stronger contrast but risks over‑amplifying noise.
**Figure.** (a) Schematic of the LoopCD mechanism, showing how an earlier hidden state $h_1$ is used to guide the final prediction. (b) Bar chart comparing baseline performance with LoopCD at the same depth across various benchmarks and models. (c) Scatter plot showing performance versus computational cost (TFLOPs), demonstrating that LoopCD achieves comparable results with half the iterations.
**Figure 1.** Ouro repeats its full stack; Huginn, Parcae, and Looped-Qwen3 repeat a shared block (blue) between fixed prelude and coda layers (gray). Layer and iteration counts are for Ouro-1.4B, Huginn-0125, Parcae-1.3B, and Looped-Qwen3 (Table A1).
**Figure.** LoopCD-Hidden combines $h_1$ and $h_R$ (Eq. (2)) and evaluates the coda and `lm_head` once; LoopCD-Logits evaluates both states through them and combines the logits (Eq. (3)), in two passes. Rust marks the reference, sage the final state, steel the guided result.
Empirical Performance Evaluation
LoopCD delivers sizable accuracy gains and cuts forward FLOPs by up to half.
We benchmark LoopCD on four looped Transformer families under three protocols, first confirming strong full‑depth gains and then probing whether half the recurrent loops suffice.
LoopCD raises HumanEval pass@1 by +9.15 percentage points when applied to the 1.4 B‑parameter Ouro model.
Baseline 23.17 % → LoopCD‑adaptive 31.71 % (Table 1)
**Table 1.** LoopCD improves mathematical reasoning and code generation at full recurrent depth.
**Figure 4.** With LoopCD, half the recurrent iterations match or beat full depth for less compute.
Validating the Recurrence Trajectory
Ablation study reveals which recurrence components drive LoopCD gains and their impact.
Recall that LoopCD refines the final output by contrasting a weak prediction from an early recurrent pass with the strong final prediction.
Using the first recurrent state as the LoopCD reference yields the largest gain; removing it cuts performance by 2.9 points for Huginn.
Huginn’s first iteration gives +3.07 points, while the sixteenth iteration gives only +0.17 points.
Referencing deeper recurrent states dramatically reduces LoopCD‑Logits gain, losing 2.48 points for Parcae‑1.3B.
Parcae‑1.3B’s gain drops from +3.50 points with the first state to +1.02 points with a later state.
Guidance adds up to +13.3 points on the least‑confident fifth of ARC‑Challenge questions, but only +0.4 points on the most‑confident fifth.
Grouping questions by the probability margin shows gains ranging from +6.4 to +13.3 points for low‑confidence items and ≤+0.4 points for high‑confidence items.
Extended Results
Extended LoopCD results across depths and benchmarks with detailed hyperparameters and performance tables.
Table A6 shows that the same fixed‑strength guidance lowers pass@1 by –2.91 points while raising pass@10 by +3.26 points for Looped‑Qwen3, illustrating the trade‑off between single‑sample accuracy and multi‑sample robustness.
**Table 2.** Every logit configuration improves its seven-benchmark mean under LoopCD.
Table A4 lists the hyperparameter caps used for the standard‑budget experiments, distinguishing fixed caps (0.5) from adaptive caps (varying per task) for both LoopCD‑Logits and LoopCD‑Hidden settings.
Table A5 reports LoopCD‑Logits results on GSM8K and MMLU‑Pro; most configurations either maintain or improve the baseline, with the largest MMLU‑Pro gain of +2.04 points for Ouro‑1.4B under the adaptive setting.
Reduced‑depth experiments (see Table A8) follow the distinct settings described here, confirming that LoopCD benefits persist even when the recurrent depth is limited.
FLOP Accounting and Reduced Depth
Quantitative ablations of LoopCD‑Logits versus LoopCD‑Hidden across reduced‑depth settings.
This appendix expands the ablation study introduced in §4.3, reporting FLOP accounting for the two guidance forms and evaluating how halving the recurrent iteration budget impacts both performance and compute.
LoopCD‑Hidden incurs essentially no extra FLOPs (multiplier ≈ 1) while LoopCD‑Logits adds between +0.97 % and +25.11 % overhead depending on the model.
Table A7 shows the per‑model multipliers and the breakdown of head and coda contributions.
Halving the recurrent iteration budget degrades the unguided baseline, but LoopCD recovers the loss across all six configurations, yielding mean $\Delta$ improvements of +0.83 points (logits) and +0.85 points (hidden) on the seven‑benchmark average at $\omega$ = 0.5.
Performance tables (A8–A9) report the recovered scores after applying LoopCD with fixed strength.
When the model’s coda is shallow (e.g., Huginn), LoopCD‑Hidden matches or exceeds LoopCD‑Logits; with deeper coda (Parcae‑1.3B) hidden‑state guidance retains only ~60 % of the logit gain.
Table A10 compares hidden‑state versus logit gains for each architecture.
**Figure.** One bar per configuration, scaled to its unguided full-depth pass: prelude, recurrent iterations, then coda and head. Dashed blocks are the iterations a half-depth run skips; the outlined block at the right is LoopCD-Logits’ second coda and head. LoopCD-Hidden adds no visible block. Analytic FLOPs at a 512-token prefill.
Re-ranking Close Decisions
LoopCD delivers up to +7.3 points gain on the hardest decisions while flipping under 15 % of answers.
LoopCD concentrates its guidance gain on the least‑confident fifth of decisions, delivering up to +7.3 points improvement on ARC‑Challenge while altering fewer than 15 % of answers.
Table A12 shows the least‑confident fifth contributes +7.3 pts (Ouro‑2.6B) and the highest‑confidence fifth contributes 0 pts; answer‑change rates range from 3.9 % to 14.9 % across models.
Guidance acts as a tie‑breaker: it only shifts a final decision when the injected contrast exceeds the margin between the top two candidates. In architectures where the loop feeds directly into the head, LoopCD‑Hidden reproduces the orthogonal re‑ranking component of LoopCD‑Logits, as confirmed by cosine similarities of 0.96 (Ouro) and 0.89 (Huginn). Table A13 illustrates a concrete ARC‑Challenge question where the weak‑to‑strong momentum is amplified to flip the prediction.
Guidance Strength Dynamics
We dissect how LoopCD’s guidance strength and temperature sharpening shape benchmark performance.
When forced to use the first hidden state, Huginn’s alignment falls to 0.62 because the initial state is dominated by random noise rather than a coherent re‑ranking signal.
LoopCD‑Hidden mirrors orthogonal re‑ranking, while LoopCD‑Logits retain a temperature‑sharpening component that can be either beneficial or harmful depending on the benchmark.
On HellaSwag, aggressive sharpening degrades performance: the full logit update adds +0.76 points for Ouro‑1.4B, whereas the pure re‑ranking component adds +1.72 points, and the hidden‑state form gains +1.73 points.
Conversely, on ARC‑Challenge the sharpening term is advantageous; the pure re‑ranking component needs a much larger strength $\omega$ to reach its peak +2.65 points, while the full logit update peaks efficiently at $\omega$ with +2.47 points.
Thus, selecting between LoopCD‑Logits and LoopCD‑Hidden reduces to a dataset‑dependent decision about whether temperature sharpening is helpful, typically shifting net accuracy by one or two percentage points.
Across a seven‑benchmark suite, LoopCD‑Hidden consistently outperforms on HellaSwag but trails LoopCD‑Logits on ARC‑Challenge, ARC‑Easy, SciQ, and MMLU.
Generation benchmarks exhibit a narrow viable range for $\omega$; beyond the optimal positive window performance collapses catastrophically (e.g., Ouro‑1.4B loses 8 points at $\omega$ = 0.8 and 20 points at $\omega$ = 1.3 on GSM8K).
Negative strengths interpolate backward along the contrast vector, uniformly degrading multiple‑choice scores and converging to the reference accuracy at $\omega$ = −1.0, confirming the contrast vector’s role as a pure direction of improvement.
Table A14 illustrates the trade‑off between guidance strength and confidence fifths: as $\omega$ increases, a larger fraction of answers changes, but gains on the least‑confident fifth diminish, eventually receding at $\omega$ = 1.0.
The figure shows the ARC-Challenge accuracy (%) across four recurrent iterations for two models: Ouro-2.6B and Ouro-1.4B.
**Table A12.** The gain comes from the least confident questions and fades toward the most confident.
Ouro-1.4B, 25-shot: per-character log-likelihood of each option after the first pass, at the final pass, and under LoopCD-Logits ($\omega = 0.5$, reference $h_1$); bold marks the leading option.
**Figure a.** Verdicts that differ, HellaSwag, $\omega$ = 0.5
**Figure b.** Direction of the hidden update in logit space
Conclusion
Looped Transformers refine predictions with minimal extra cost.
Looped Transformers generate intermediate states at successive recurrent depths, yielding weak and strong predictions that LoopCD contrasts to refine output quality with minimal overhead.
At equal depth, LoopCD boosts Ouro’s mean pass@1 on mathematical reasoning by 5–7 points and improves every multiple‑choice and generation benchmark across four families; with half the iterations it matches or exceeds deeper unguided models in six settings while using 22.5%–48.2% fewer FLOPs.
The improvement stems from recurrence settling predictions early, so the first iteration records the direction of movement and subsequent passes re‑rank close decisions toward the option recurrence was raising, a benefit that may grow for models trained with intermediate predictions like Ouro.
Implementation Details
Appendix A expands the model configurations, layer counts, and the probability form of LoopCD.
This appendix expands Sections 2 and 3 by detailing the state‑indexing and output‑interface mechanisms that underlie Eq. (3).
Four families of Looped Transformers are evaluated: Ouro, Huginn, Parcae, and Looped‑Qwen3, each with a distinct way of exposing intermediate states to the final prediction head.
Table A1 lists the layer counts for each evaluated model; counts include prelude, recurrent block, and coda layers, while FLOP counts are omitted because layer widths differ.
Ouro‑1.4B uses no prelude or coda layers, repeats its decoder stack 24 times, and applies the prediction head after 96 layer applications.
Huginn‑0125 inserts a two‑layer prelude, repeats a block of 32 layers four times, and adds two coda layers, totaling 132 layer applications; it initializes a recurrent state with Gaussian noise.
Parcae‑1.3B features eight prelude layers, eight repeated blocks of eight layers, and eight coda layers, yielding 80 layer applications; its recurrent output is projected before entering the coda.
Looped‑Qwen3 repeats a fifteen‑layer prelude, a four‑layer window block eight times, and a seventeen‑layer coda, for 64 layer applications; each substep follows $x_{t+1}$=$f_{\text{window}$}($x_t$)+\alpha\,\Delta.
The experimental setup matches each LoopCD run with an unguided baseline that shares checkpoint, prompt, shot count, generation limit, stopping rule, answer extractor, and evaluator; recurrent depths are strictly aligned except in explicit ablations.
Evaluation spans multiple‑choice benchmarks (ARC‑Challenge/Easy, HellaSwag, WinoGrande, PIQA, SciQ, MMLU) and greedy‑generation benchmarks (HumanEval, MBPP, GSM8K, MMLU‑Pro), all scored by likelihood using the language‑model evaluation harness.
All reported scores are percentages; a subscript $\Delta$ denotes the signed percentage‑point change from the matched unguided baseline, and “Mean $\Delta$” averages these changes with equal weight across relevant columns.
Experimental Setup
Appendix B records the benchmark suites, prompt configurations, and hyper‑parameter choices used in the experiments.
B.1 defines the two recurrence depths used throughout the study, R = 16 and R = 32, and explains the notation conventions for the tables that follow.
B.2 introduces the $\Delta$ symbol for contrast magnitude, then outlines the LoopCD experimental pipeline, including the datasets (AIME 2024/2025 and the OlympiadBench subset) and the unbiased pass@k estimator used to compute pass@1 and pass@10.
Tables A2 and A3 list the few‑shot prompt configurations for multiple‑choice and generation benchmarks; Table A2 shows the number of demonstrations per model per task, while Table A3 records the number of demonstrations for GSM8K, MMLU‑Pro, HumanEval(+), and MBPP(+).
Hyperparameters were selected via screening sweeps described in Appendix G.1 (Figures 9 and 10), and the chosen values for the standard recurrent depth are recorded in Table A4.
B.3 details the prompting and sampling settings, repeating the recurrence depths (R = 32 and R = 16) for the various evaluation runs and noting the † symbol for runs that use the adaptive strength variant.
B.4 describes the strength‑allocation strategies: fixed strengths (e.g., $\omega$ = 0.5, 0.3, 0.2) are used for logit‑space LoopCD, while adaptive caps are applied to hidden‑state configurations and reduced‑depth experiments.
**Figure 8.** (a) An illustrative contrast $z_R - z_1$ splits into a part parallel to $z_R$, which rescales the logits like a temperature, and an orthogonal part, which re-ranks tokens. (b) HellaSwag, the two Ouro models: accuracy change as the strength grows, re-ranking part alone (thick) and full update (thin). (c) ARC-Challenge questions in five equal groups by the unguided prediction's confidence, least confident first: accuracy change per group, LoopCD-Logits at $\omega = 0.5$, four models.
**Table A1.** Layer structure of the evaluated models.
Reference Selection and Initialization
Analysis of reference selection, geometric convergence, and intra-iteration dynamics in looped architectures.
To determine what makes an effective reference for guidance, we examine how standalone accuracy and disagreement drive selection. Empirical results demonstrate that guidance gain scales reliably with the reference’s disagreement from the final prediction, rather than its standalone competence.
For instance, Parcae-1.3B’s sixth step achieves a highly competent standalone accuracy of 41.04% compared to the final prediction’s 40.36%. However, because it has largely converged with the final state, it yields only a single point of gain. Conversely, Huginn’s second step is weak, yet because its disagreement has already shrunk, it provides only half the guidance gain of the initial step. Consequently, the first iteration universally serves as the optimal logit reference across stable architectures.
This reliance on divergence direction also explains a notable architectural anomaly: Huginn’s hidden-state reference performance. Huginn initializes its recurrent state from Gaussian noise and moves it on a hypersphere; the first step is nearly as long as the radius itself, making the path 6.0 times longer than the net displacement. LoopCD-Hidden directly amplifies this unprojected noise, rendering the first hidden state detrimental.
These visualizations highlight distinct architectural dynamics. Ouro exhibits a sharp geometric reset, where similarity to the final state drops dramatically after every pass boundary, followed by a steady recovery. In contrast, Looped-Qwen3 incorporates damped substeps that constrain updates, resulting in a much smoother, gradual climb toward the final representation.
Our sweep of internal layers revealed that extracting any internal state severely degrades ARC-Challenge performance, dropping by up to 24.9 points for Ouro-2.6B below ordinary decoding. Unlike standard dense networks where intermediate layers can be contrasted against the final layer, looped architectures do not form trained exit points, making intra-iteration states unstable for contrastive guidance.