LEV: Thinking Fast and Deep with Looped Decision Models
Dingjie Song, Zhiling Yan, Yuxuan Zhang, Lichao Sun
LEV uses natively looped language models to trade latency for accuracy per question without generating text.
Can we improve decision model accuracy by running the same decoder stack multiple times (looping) and dynamically choosing when to stop?
Decision models are fast but spend the same fixed computation on every question, forcing them to waste resources on easy tasks or fail on hard ones. LEV reruns the layers of a natively looped backbone in a loop, allowing the model to "think" longer on hard questions by adding steps without generating tokens. On-demand depth saves 15% to 24% of computation at a negligible accuracy cost, and at low latency, LEV outperforms dense models and cascades of similar speed.
Paper Primer
Standard decision models act as a "System 1," returning a probability in a single forward pass. Because their depth is fixed at training time, they cannot adjust their compute budget to match the difficulty of the input, unlike generative models that can "think" via chain-of-thought but pay a heavy decoding penalty for every question.
LEV is a "System 1.5" model: it treats depth as a runtime setting by rerunning a natively looped backbone and reading the options after every loop step. A lightweight stopper predicts whether the current answer agrees with the deepest possible readout, allowing the model to halt early on easy questions without needing labels for the stopping threshold.
On-demand depth provides accuracy comparable to the deepest readout while significantly reducing compute.
On held-out panels, the stopper maintains accuracy within one point of the deepest pass while using 76% to 85% of the total computation.
LEV outperforms dense models and cascades at low-latency thresholds.
At latencies below 50ms, the 1.4B and 3B models are 5.4 to 8.8 points more accurate than dense models and cascades of equivalent speed.
Why use a natively looped backbone instead of just looping a standard model?
Loops added to standard pretrained models often stay unused because the model was not trained to read intermediate states. LEV uses backbones pretrained as loops, ensuring every step is trained to be a functional decision model.
How does the stopper decide when to stop without seeing the correct answer?
The stopper predicts whether the current step's answer agrees with the model's own deepest answer. This consistency target allows the stopper to be fitted on unlabelled traffic rather than requiring ground-truth labels.
LEV demonstrates that depth can be a dynamic, per-question setting rather than a fixed architectural constraint, enabling efficient, adaptive inference for discriminative tasks.
Introduction
We introduce LEV, a looped decision model that adaptively allocates compute to harder questions.
Decision models output a probability for each answer option in a single forward pass, so they compute the same amount of work for every question regardless of difficulty. This fixed‑depth behavior mirrors the fast, intuitive System 1 in dual‑process theory, unlike generative models that reason step‑by‑step (System 2) and incur token‑level cost for every extra thought. Existing decision models such as Jev and Kev therefore waste compute on easy items and cannot allocate more effort to hard ones.
A decision model maps an input and a set of answer options to a probability distribution over the options in one forward pass, without generating intermediate text.
LEV addresses this inefficiency by looping the decoder stack R times, re‑reading the answer options after each loop step and sharing a single readout across depths, so any checkpoint can produce an answer. Because the loop is trained from the start, the model can resume from its hidden state without re‑encoding the input, enabling adaptive, on‑demand depth that spends more computation only where it pays off. Empirically, a 2.6 B LEV improves from 63.2% at depth 1 to 76.2% at depth 4 on JevBench, and a lightweight stopper that predicts agreement with the deepest answer saves 15%–24% of compute while losing less than one accuracy point.
Our contributions are threefold: (1) we propose LEV, a looped decision model that enables adaptive compute; (2) we introduce an on‑demand depth stopper that requires no labeled data; and (3) we release LevBench, a frozen benchmark for evaluating depth‑aware decision models.
LEV enables adaptive compute for decision models, letting a single checkpoint dynamically choose depth to balance speed and accuracy.
Background and Motivation
We expose why fixed‑depth decision models waste compute and motivate adaptive looping.
Decision models evaluate all answer options in one forward pass, but they allocate the same amount of compute to every question. When the depth is fixed, easy questions waste resources while hard questions suffer from insufficient processing. This mismatch motivates a mechanism that can adaptively allocate more compute only where it is needed.
Our empirical study on the public JevBench reveals two key patterns. First, the larger Kev‑4B model improves accuracy on only the standard and hard tiers, gaining 28 correct answers that the smaller Kev‑0.8B never gets; the confidence scores of Kev‑0.8B reliably flag those 42 items where the models disagree, so routing the least‑confident half to Kev‑4B recovers almost the full gain. Second, existing augmentation strategies either double latency (cascades read every item twice) or incur a universal slowdown (reasoning text adds three‑to‑four‑fold latency even on easy items), and naïve retrofitted loops remain closed, barely moving accuracy.
Because they spend the same amount of computation on every question, they over‑invest on easy items that are already solved and under‑invest on hard items that need more processing. Adaptive depth can reallocate compute where it yields the biggest accuracy gains.
**Figure 2.** Evidence for depth chosen per question. (a) Outcome of each public JevBench item under Kev-0.8B and Kev-4B. (b) Gain over Kev-0.8B when Kev-4B answers a random half, the least confident half, or all items. (c) Single read against generated reasoning on the same base model, with latencies from published results on different endpoints. (d) Accuracy change from loops retrofitted onto Qwen3.5-0.8B, each row changing one setting of the default.
The LEV Architecture
LEV loops a shared decoder to adaptively allocate compute per question.
The fixed‑depth cost of decision models motivates reusing a single decoder stack multiple times, enabling early exit once the answer stabilises.
Instead of building a deep stack once, we run the same decoder stack $R$ times and read out after each pass, stopping when the answer matches the deepest reading – like rereading a paragraph until you are sure you understood it.
Step 1: $h^{(1)}=\operatorname{Norm}(h^{(0)}+[0.1,0.1])=[0.3,0.6]$.
Step 2: $h^{(2)}=\operatorname{Norm}(h^{(1)}+[0.1,0.1])=[0.4,0.7]$.
Step 3: $h^{(3)}=\operatorname{Norm}(h^{(2)}+[0.1,0.1])=[0.5,0.8]$.
After each $h^{(t)}$, the pointer head produces logits $z^{(t)}$ and a distribution $p^{(t)}$; suppose $p^{(1)}$ predicts option A with 0.55, $p^{(2)}$ still A with 0.58, and $p^{(3)}$ switches to option B with 0.62.
The stopper sees that $p^{(2)}$ already matches the deepest answer $p^{(3)}$ (both would select B after calibration), so it would exit after step 2, saving the cost of step 3.
Early exit is possible because the model’s confidence stabilises; the loop lets the same parameters refine the answer without re‑encoding the input.
How is LEV different from simply running the decoder stack repeatedly on the same input?
LEV shares the exact same weights across all $R$ passes and reads out after each pass, using a learned stopper to decide when to stop. Naïve repeated inference would recompute the entire stack each time without any early‑exit signal, incurring $R$ × the full cost and never producing intermediate predictions.
Compute the initial hidden state $h^{(0)}$ from the token embeddings.
For $t=1$ to $R$: apply the shared decoder stack $F_{\theta}$, normalise to obtain $h^{(t)}$.
Apply the pointer head to $h^{(t)}$ to obtain logits $z^{(t)}$ and probabilities $p^{(t)}$.
Feed $p^{(t)}$ (and $p^{(t-1)}$) into the stopper logistic regression (Eq 4) to predict $\hat{y}_{t}$.
If $\hat{y}_{t}\ge\tau$, stop and output the current answer; otherwise continue to the next step.
**Figure 3.** LEV with $R$ loop steps, drawn for $R = 4$. (a) A request is packed into one sequence, and the same decoder stack $F_\theta$ runs $R$ times. One shared pointer head reads the `</opt>` and `<decide>` positions after every step. (b) After step $k$, a stopper predicts whether the answer already agrees with the deepest one. (c) Each question exits at its own depth, and the state runs as deep as its deepest question.
Experimental Setup
How the models are built, trained, and evaluated for the looped decoder trick.
Fixed‑depth decision models allocate the same amount of compute to every query, which wastes resources on easy questions and leaves hard ones under‑served.
Instead of stacking a deeper static decoder, the model re‑executes the same decoder stack R times, letting later passes refine the answer while earlier passes handle easy cases.
How does looping the same decoder differ from simply adding more layers?
Adding layers creates new parameters for each depth, increasing memory and training cost. Looping reuses the exact same parameters R times, so the model’s size stays fixed while only the compute time grows linearly with R. Moreover, because each loop is supervised (when a loss is applied after every step), the intermediate states are encouraged to be meaningful, enabling early termination.
Pass 1: the decoder produces hidden vectors h₁ = [0.3, 0.7] for token 1, h₂ = [0.4, 0.6] for token 2, h₃ = [0.5, 0.5] for token 3, h₄ = [0.2, 0.8] for token 4.
Pass 2: the same decoder receives the hidden vectors from Pass 1 as input and updates them to h₁′ = [0.35, 0.65], h₂′ = [0.45, 0.55], h₃′ = [0.55, 0.45], h₄′ = [0.25, 0.75].
The final output is taken from Pass 2; the intermediate Pass 1 output could already answer an easy query (e.g., token 4’s answer exceeds a confidence threshold after Pass 1).
Looping lets the model allocate extra compute only where needed: easy tokens converge after the first pass, while harder tokens benefit from the second refinement, all with a constant parameter budget.
**Table 5.** Evaluation panels and the suites they come from. Suite names are those of the Kev repository. Questions are the scored questions of each panel.
Main Results
LEVs extra loops boost accuracy, while on‑demand depth trims compute with minimal loss.
Extra loop steps increase JevBench accuracy up to 13.0 points for LEV‑2.6B.
Table 1 reports a 13.0‑point gain when adding the second loop step compared with the depth‑one baseline.
**Figure 5** Where depth helps. (a) JevBench accuracy on the standard and hard tiers, with the Qwen3-1.7B control dashed. (b) LevBench questions that LEV-2.6B revises between consecutive depths, or gets right at some shallower depth but wrong at depth four. (c) Gain of LEV-2.6B from depth one to four per Kev-Breadth area. (d) LEV-2.6B on the LevBench tiers, with the cost of on-demand depth in parentheses.
Motivation and Summary
We restate the LEV premise and detail the motivation experiments.
Routing from Kev‑0.8B to Kev‑4B orders items by the top‑probability confidence of the smaller model. Accuracy on JevBench rises as more items are sent to the larger model, surpassing random routing once roughly a seventh of the items are transferred. The curve flattens in the final fifth because the larger model adds no new correct answers when the smaller model is already confident.
**Figure 12.** Accuracy on the 231 public JevBench items when the items on which Kev-0.8B is least confident are answered by Kev-4B, against random routing.
Generated reasoning compares two server configurations that both run DiffusionGemma 26B. The djev server uses a single read path, while OpenJev adds a quantised thinking path that generates additional tokens. The thinking mode incurs more token generation and occasional failures, leading to lower accuracy on hard items despite higher computational effort.
Retrofitted loops reuse layers 8–15 of the 24‑layer Qwen3.5‑0.8B‑Base stack by running them twice with per‑channel ReZero‑style gates. Training a LoRA adapter with rank 16 for two epochs yields gates of mean magnitude 0.006, so the second pass contributes less than one percent of the residual update, and removing the gate harms accuracy.
The LEV decision model leverages loop steps instead of token steps, allowing a single model to trade latency for accuracy at serving time. Stopping based on unlabelled requests saves 15–24 % compute for under one accuracy point, and larger looped backbones may amplify gains on hard questions.
Our experiments cover looped backbones up to 3 B parameters; scaling to larger models remains untested. Cost accounting counts loop steps, but real‑world serving may batch different depths, affecting wall‑clock savings. Latency measurements lack optimisations such as fused kernels or state caching, which could further narrow the gap between looped and dense models.
Benchmark Details
Details of the JevBench benchmark, training data, evaluation protocol, and model results.
This appendix records the JevBench benchmark used to evaluate our models, describing its data composition, evaluation protocol, and the resulting performance tables.
JevBench is a benchmark suite of 231 questions split into easy, standard, and hard tiers, designed to test model accuracy and latency under realistic request conditions.
**Table 6.** Records that the larger training sets add to Train-31k, by part of the pool.
The table compares the performance of the evaluator against the official JevBench harness on the released Kev checkpoints.
**Table 8.** Accuracy and latency on the public JevBench items per tier (48 easy, 72 standard, 111 hard), with the JevBench Intelligence score. Bold marks the best value among the models trained on identical data. $^\dagger$Measured through the production API with the network included, and not comparable to the local measurements.
On-Demand Performance
On‑demand depth performance, calibration, and paired comparisons across panels.
This appendix reports the quantitative impact of on‑demand depth: accuracy, compute cost, calibration, and statistical significance across development and test panels.
On‑demand depth reaches 85.9 % accuracy on the tuning panels while using only 0.34 of the token‑weighted compute of the deepest readout (77.4 %).
Table 10 (development panels) shows the on‑demand accuracy of 85.9 % at cost 0.34 versus the deepest readout accuracy of 77.4 %.
LEV‑2.6B outperforms LEV‑1.4B by 8.2 points at depth four (95 % CI [+3.5, +13.0]), and by 12.6 points on the hard tier (95 % CI [+4.5, +21.6]).
C.2 Paired comparisons on JevBench report these gains with McNemar p < 0.001.
Temperature scaling reduces hard‑tier expected calibration error (ECE) for LEV‑1.4B at depth four from 0.255 to 0.231.
C.3 Calibration, Table 9, shows the post‑scaling ECE value.
**Table 2.** On-demand depth on the two held-out panels, pooled over their questions. Cost is the token-weighted share of the deepest pass. The last two columns are paired differences in accuracy points with 95% CIs. Tables 10 and 11 give every panel.
**Figure 13** Paired item-level differences in accuracy on the public JevBench items (231 in all, 111 hard), in points with 95% CIs. Open markers are the hard items.
Stopping Rules
We evaluate how each stopping rule changes accuracy at a fixed compute budget.
Recall that LEV loops the decoder stack so that harder queries can use more steps, while easier ones stop early.
A stopping rule decides, after each decoder pass, whether the current answer is good enough to halt further computation.
How does the learned stopper differ from the classic top‑probability early‑exit rule?
The stopper combines eight engineered features (e.g., margin, entropy, change of top‑probability) and learns a logistic‑regression boundary, whereas the top‑probability rule looks only at a single scalar after temperature scaling. This richer signal lets the stopper stop more often without sacrificing accuracy.
Both the stopper and the top‑probability rule are calibrated on a held‑out “calibration” partition; the stopper’s threshold is 0.915 for the two Ouro models and 0.8075 for LEV‑3B, while the top‑probability thresholds are 0.81 (LEV‑2.6B), 0.76 (LEV‑1.4B), and 0.69 (LEV‑3B).
Stopper improves accuracy by +0.5 points over the fixed‑depth frontier at the same cost for LEV‑2.6B.
Table 16 shows a +0.5 point gain for the stopper variant on LEV‑2.6B.
Top‑probability rule yields a modest +0.4 point gain for LEV‑2.6B.
Table 16 reports a +0.4 point improvement for the top‑probability rule on LEV‑2.6B.
On the smallest model (LEV‑1.4B) the top‑probability rule actually loses ‑0.5 points relative to the frontier.
Table 16 lists a ‑0.5 point difference for the top‑probability rule on LEV‑1.4B.
Shallow confidence examines the depth‑one readout: the top‑probability rule stops at depth 1 when the scaled top probability exceeds its threshold.
Depth‑one top‑probability stops ≈ 90 % of questions on LEV‑2.6B, achieving an AUROC of 0.95 for separating right from wrong answers.
Table 14 reports a top‑probability AUROC of 0.95 for LEV‑2.6B on the “Kev‑Seen” panel.
Early‑exit rules from prior work (margin, entropy, patience) are compared against the learned stopper and the top‑probability baseline.
Margin‑based early exit gains +0.1 point over the frontier for LEV‑2.6B, matching the top‑probability rule’s improvement.
Table 16 shows a +0.1 point difference for the margin rule on LEV‑2.6B.
**Figure 6** Accuracy against cost. Gray points and line are the fixed depths and their frontier, and the blue line moves the threshold of the stopper to the chosen dot. The dashed line and the star repeat both on the locked test partitions.
**Figure 8** Accuracy of each stopping rule minus the fixed-depth frontier at the same cost on the pooled held-out panels, with 95% CIs. Gray rows are early-exit rules, and patience needs at least three steps. Light blue rows change one part of the stopper or set $a$ single top-probability threshold the way the stopper's is set, and the change features alone never stop LEV-3B.
Dense Model Controls
Dense models and cascades benchmark on‑demand LEV against fixed‑depth baselines.
Dense models and cascades serve as strong reference points for evaluating the on‑demand depth capability of LEV.
**Figure 10** Accuracy against mean latency on the Lev-Bench test partition. Lines join the fixed depths of one LEV checkpoint, and stars mark its on-demand depth. The dashed line is the best dense model or cascade, and each number is the gap of on-demand LEV to it.
The top‑probability rule of LEV‑2.6B reaches 75.8% at a cost of 0.69, just 0.2 pts below the frontier; the calibration‑only stopper hits 75.6% at 0.61 (+0.6 pts), and the full stopper attains 76.7% at 0.64 (+1.4 pts).
When compared to cascades, the same rule gains 3.8 pts on Kev‑Seen, 2.5 pts on Kev‑Transfer, and 2.2 pts on Kev‑Skills, showing that on‑demand LEV consistently outperforms fixed‑depth cascades across diverse test sets.
Table 17 details the accuracy advantage of on‑demand LEV over the best dense option: LEV‑1.4B is +7.0 pts, LEV‑2.6B +6.2 pts, and LEV‑3B +8.8 pts on the LevBench test partition, with similar margins on development splits.
When latency is held constant, on‑demand LEV models still trail the best dense option in FLOP‑matched settings: LEV‑1.4B is –6.0 pts and LEV‑3B –0.7 pts, highlighting the trade‑off between compute efficiency and accuracy.
Training Details
This appendix lists the training hyper‑parameters, loop variants, and evaluation panels used for LEV and the dense controls.
The following tables and descriptions capture all hyper‑parameter choices and evaluation splits used in the experiments. They are needed to reproduce the reported results and to understand how LEV differs from the dense baseline.
**Table 4.** Training settings of LEV and the dense controls. The step weights apply only to LEV, and the last block only to the full-weight runs.
Training proceeds for a single epoch; each optimiser step processes $64$ records, and micro‑batches of roughly $8$ examples are formed by grouping records of similar length. The backbone tokenizers treat the delimiters of Section 4.1 as special tokens, and any user‑supplied text that matches them is rewritten to prevent boundary forging.
Evaluation uses four panels: three tuning panels (Held‑out, Kev‑Transfer, Kev‑Skills) and LevBench, each drawn from distinct source suites. The locked test set combines LevBench (8,740 questions) and Kev‑Breadth (3,031 questions) to assess generalisation.
Training Configurations
Training data variants, dense baselines, joint training, and loss‑recipe effects for LEV models.
This appendix reports the training configurations used for the looped decision model (LEV) and the resulting performance and cost numbers.
**Table 18.** Table 19 per panel, with the deepest JevBench accuracy of each checkpoint. Gray rows (full) train all weights on Train-31k.
**Table 19.** Training data and full-weight fine-tuning, on the pooled held-out panels. Cost and the difference to the frontier are for on-demand depth. Gray rows (full) train all weights on Train-31k. Part of LevBench is in distribution for Train-60k and Train-120k (Section B.4).
**Table 20.** Accuracy of the deepest readout of the looped models and of dense controls trained on Train-31k, Train-60k, and Train-120k (columns). Part of LevBench is in distribution for Train-60k and Train-120k.
**Table 21.** Accuracy of the jointly trained LEV-2.6B and of a model trained for each depth alone (Single), with the paired difference in points. On JevBench the single-depth models reach 60.6%, 72.3%, and 77.9%, against 63.2%, 71.0%, and 76.2% for joint training.
**Table 22.** Recipe variants on the pooled held-out panels. Cost and the difference to the frontier are for on-demand depth, and cost is a share of each model's own deepest pass.
All cost figures count floating‑point operations as two FLOPs per parameter, token, and loop step, providing a hardware‑agnostic measure of compute expense.