U-Space: Uncovering When and Why Uncertainty Arises in Language Models
Tobias Braun, Nils Loose, Alexander Herzog, Virginia Ceccatelli, Marcus Rohrbach, Thomas Eisenbarth, Lorenzo Cavallaro
U-SPACE maps language model uncertainty into an interpretable subspace, enabling training-free, length-robust risk estimation.
How can we decompose language model uncertainty into interpretable, distinct sources like ambiguity or incompleteness, rather than treating it as a single scalar confidence score?
Language models often present incorrect reasoning with authoritative confidence, and existing uncertainty metrics are frequently confounded by generation length or require costly repeated sampling. The authors introduce U-SPACE, a low-dimensional subspace constructed by contrasting semantic anchors of doubt and certainty within the model's residual stream. By projecting hidden states onto this basis, the U-LENS provides a token-level map of uncertainty that combines verbalizable semantic signals with distributional entropy. This approach outperforms established baselines across reasoning benchmarks, maintaining predictive power even when controlling for the length of the generated reasoning trace.
Paper Primer
The U-LENS operates as a linear readout: it maps intermediate residual states into a contrastive basis of uncertainty categories—ambiguity, incompleteness, conflicting evidence, and general uncertainty—without requiring task-specific training or correctness labels.
U-LENS provides superior uncertainty quantification compared to established baselines, particularly under length-controlled evaluation.
Across three reasoning models and four benchmarks, U-LENS consistently leads in AUROC, AUPRC, and AURC metrics when controlling for output token count. It improves AUROC by up to 4.7 points over the strongest baseline on the Magistral 1.1 model.
The method is causal: steering the model's residual states along the U-SPACE basis directions can induce hesitation and reduce confidence, confirming that these directions represent internal states that actively shape model behavior.
Why is generation length a problem for existing uncertainty estimators?
Incorrect reasoning traces are often systematically longer than correct ones, allowing simple estimators to achieve high performance by using length as a proxy for error rather than capturing genuine predictive uncertainty.
What is the advantage of combining verbalizable uncertainty with predictive entropy?
Verbalizable signals (second-order uncertainty) and predictive entropy (first-order uncertainty) capture distinct information; their combination allows the model to modulate its confidence based on both distributional probability and semantic context.
Introduction
We expose why scalar confidence fails and introduce a low‑dimensional U‑SPACE to reveal interpretable uncertainty sources.
Language models now influence high‑stakes decisions, so knowing when a single answer can be trusted is critical. Existing uncertainty estimators often require repeated generations or extra components and collapse all uncertainty into a single confidence score, obscuring where doubt originates. Moreover, generation length correlates with errors, making it hard to tell whether a predictor’s usefulness comes from genuine uncertainty information or merely from token count.
Instead of a single confidence number, we need a signal that tells *why* a model is uncertain, pinpointing the source of doubt during generation.
This figure illustrates a trajectory within a tetrahedral space, representing the evolution of a concept or state. A circular inset highlights a specific region labeled "love," showing a path terminating near a vertex. The main tetrahedral structure connects four colored vertices, with paths colored in gradients of gold, teal, and maroon tracing movements through the interior volume.
Shifting from scalar confidence to interpretable uncertainty sources makes uncertainty quantifiable and actionable.
Background and Related Work
Survey of uncertainty quantification and representation probing methods relevant to U‑LENS.
Predictive uncertainty quantifies how much confidence to place in a single model prediction, a problem that becomes subtle for autoregressive language generation because the model emits a full sequence probability.
It measures how reliably a model’s output reflects the true answer, distinguishing between confident correct predictions and over‑confident errors.
Aggregates token‑level probabilities or entropies across a generated response to obtain a single uncertainty score.
Prompts the model to emit a confidence token (e.g., “True”) or a verbal probability statement, treating the model as a black box.
Detects unsupported statements by checking for inconsistency across multiple sampled generations.
Clusters generated responses by semantic similarity before measuring dispersion, addressing the many‑to‑one mapping problem.
Measures agreement between semantic units across sampled long‑form responses to localize uncertainty within a reasoning trace.
Isolates uncertainty about a specific claim by conditioning the probability on that claim, separating it from lexical variation.
Ranks token‑level predictions by their confidence, allowing downstream filtering of reasoning traces.
Uses Bayesian inference over low‑rank weight perturbations to estimate token‑level uncertainty.
Applies low‑rank weight perturbations to obtain Bayesian token‑level uncertainty, similar to DeepConf but with a different perturbation scheme.
Shows that response length correlates with confidence scores, prompting length‑controlled benchmarks and detrending techniques.
Trains linear probes on hidden states to predict factual correctness or detect false statements.
Classifies features of attribution graphs to guide model interventions.
Applies the model’s unembedding matrix directly to intermediate residual states to read out token probabilities.
Learns a layer‑specific affine transformation to improve the linear readout of hidden states.
Computes a Jacobian‑based linear map $\bar{J}_{\ell}= \mathbb{E}_{x,t,t'\ge t}\frac{\partial h_{L,t'}}{\partial h_{\ell,t}}$ to project intermediate states into the final‑layer space without learning a decoder.
The U-SPACE and U-LENS
We define the U‑SPACE subspace and the J‑LENS readout, and they yield token‑level and trace‑level uncertainty estimates.
The method builds a dedicated low‑dimensional subspace, U‑SPACE, that isolates distinct uncertainty sources, and then reads it with a linear probe, the J‑LENS, to obtain interpretable token‑level maps and a scalar trace‑level score.
U‑SPACE is a small orthogonal basis inside the model’s residual stream that points toward the four semantic uncertainty categories, letting us separate “what kind of uncertainty” from the rest of the representation.
How does U‑SPACE differ from a generic PCA subspace of the residual stream?
PCA finds directions of maximal variance regardless of semantics, while U‑SPACE is built from curated uncertainty and certainty anchors and explicitly orthogonalizes the positive‑negative difference, guaranteeing that each axis aligns with a predefined uncertainty category.
Gather positive anchor set $A^{+}_{c}$ and negative anchor set $A^{-}_{c}$ for each category $c$.
Pull each anchor $a$ into the residual stream using $g_{\ell,a} = (W_{U}\,\text{diag}(\gamma)\,\bar{J}_{\ell})^{\top} e_{a}$.
Normalize each $g_{\ell,a}$ and compute the mean vectors $\mu^{+}_{\ell,c}$ and $\mu^{-}_{\ell,c}$.
Subtract the means, renormalize to obtain bipolar directions $b_{\ell,c}$.
Stack the four $b_{\ell,c}$ into $B_{\ell}$ and orthonormalize to get $U_{\ell}$.
Pull each anchor: $g_{\ell,e_{1}}=e_{1}$, $g_{\ell,e_{2}}=e_{2}$, $g_{\ell,e_{3}}=e_{3}$, $g_{\ell,e_{4}}=e_{4}$.
Normalize (already unit length) and compute means: $\mu^{+}_{\ell,\text{amb}}=e_{1}$, $\mu^{-}_{\ell,\text{amb}}=e_{2}$, $\mu^{+}_{\ell,\text{inc}}=e_{3}$, $\mu^{-}_{\ell,\text{inc}}=e_{4}$.
Subtract and renormalize: $b_{\ell,\text{amb}}=\text{norm}(e_{1}-e_{2})$, $b_{\ell,\text{inc}}=\text{norm}(e_{3}-e_{4})$.
Form $B_{\ell}=[b_{\ell,\text{amb}},b_{\ell,\text{inc}}]$ and orthonormalize (they are already orthogonal), yielding $U_{\ell}=B_{\ell}$.
Even with a trivial Jacobian, the subtraction step removes the shared “certainty” component, leaving pure uncertainty directions that are mutually orthogonal.
J‑LENS (the Jacobian Lens) linearly projects a token’s hidden state onto the U‑SPACE basis, turning the high‑dimensional representation into a concise vector of category alignments.
Why not simply use the unembedding matrix $W_{U}$ as a linear readout instead of the J‑LENS projection?
Using $W_{U}$ would map hidden states directly to vocabulary logits, mixing uncertainty directions with all other semantic information. The J‑LENS first pulls the anchor directions through the Jacobian and RMSNorm scaling, then projects onto the orthogonal U‑SPACE basis, isolating only the uncertainty‑relevant components.
**Figure 2.** **U-SPACE construction and U-LENS readout.** (a) Uncertainty and certainty anchors are pulled from vocabulary space into the residual stream through the J-LENS. Differences between their category centroids are orthogonalized to form the U-SPACE basis $U_{\ell}$. (b) The U-LENS projects each reasoning-token state onto this basis, yielding a token-by-category uncertainty map. Smooth rectification maps these coordinates into the positive cone. (c) At the end-of-thinking marker, the norm of this rectified alignment provides a second-order verbalizable signal. Multiplying it by the mean predictive entropy over the reasoning trace yields the final uncertainty score $S(r)$.
Experimental Results and Analysis
U‑LENS outperforms baselines on both raw and length‑controlled uncertainty metrics.
Recall that the paper maps three uncertainty types—ambiguity, incompleteness, and conflicting evidence—into a low‑dimensional U‑SPACE, enabling a scalar readout (U‑LENS) that captures more than raw confidence.
U‑LENS attains the highest raw AUROC (71.1 %) among all evaluated estimators, beating the next best method (TokUR, 69.6 %) by 1.5 points.
Table 1 reports AUROC scores for 13 methods across three models; U‑LENS leads with 71.1 %.
When controlling for generation length, U‑LENS remains the top performer, achieving 68.4 % AUROC on Gemma 4—2.2 points above the strongest baseline (Feature‑Gaps, 66.2 %).
Table 2 shows length‑controlled AUROC; U‑LENS leads across all three models.
**Figure 3.** One answer’s path through the simplex. Qwen3.5 answers “Was Hamlet left-handed, and did he love Ophelia?”. Each answer token is shaded by the category it is read as: No, William Shakespeare never specifies Hamlet’s handedness in the play. Yes, Hamlet did genuinely love Ophelia, though his madness and his father’s death caused him to cruelly reject her. Panels from left to right draw the path up to the magnified token (specifies, play, love, her).
**Figure b.** Targeted Reasoning Intervention
**Figure 13:** Steering strength, ambiguity. The prompt, the unsteered answer, and the continuation after adding the ambiguity direction to the prompt states at $\lambda = 0.04, 0.08, and 0.09$.
Contrastive Activation Differences
Contrastive prompts reveal how the U‑SPACE captures distinct uncertainty types and transfers to emotion readout.
To test whether the U‑SPACE truly reflects semantic uncertainty, we build minimal prompt pairs that differ only in the presence of ambiguity, incompleteness, conflicting evidence, or generic uncertainty, while deliberately omitting any explicit uncertainty vocabulary.
Decoded tokens from the difference‑in‑means direction align with the intended uncertainty category.
For ambiguity the top tokens are “ambiguous” and “ambiguity”; for conflicting evidence they are “controversial” and “contentious”; for incompleteness they are “cannot”, “unable to”, “impossible”; for general uncertainty they are “dubious”, “suspicious”, “questionable”.
**Figure 6.** **Difference-in-means readout.** Representative content-matched prompt pairs for each uncertainty category and top-15 tokens after decoding the difference-in-means direction through the J-LENS on Qwen3.5. Highlighted tokens reflect manually matched tokens to the specific content for clarity. Non-English tokens were translated. The acronyms behind the tokens reflect the original language of the token.
We next ask whether the orthonormal basis construction used for uncertainty also works for unrelated properties, specifically emotion readout on the GoEmotions benchmark.
The emotion basis built with the same U‑SPACE pipeline attains above‑70 % AUROC on all four emotion axes.
Table 5 shows AUROC scores of 71 % (anger), 73 % (sadness), 79.8 % ± 4.6 (fear), and 75 % (negative affect).
Specificity and Transfer
We evaluate transfer of constructed axes to emotion specificity and reward‑hacking detection.
To verify that the low‑dimensional construction transfers beyond the original task, we probe two distinct scenarios: emotion‑specific readout on negative‑only comments and detection of reward‑hacking responses.
Emotion‑specific axes achieve higher AUROC than the generic negative‑affect axis on negative‑only comments.
Table 6 shows each own axis reaches ≥ 67 % AUROC, with the sadness axis improving by +24.0 ± 5.9 over the generic axis.
**Results.** The emotion basis reads all four axes above 70% AUROC (cf. Table 5). The lexical baseline reaches only 51.3%–66.7%, while performance remains essentially unchanged on the no-anchor subset, indicating that the basis does not simply perform keyword matching. Twenty random orthonormal bases score at chance.
Semantic‑anchor axes separate reward‑hacking from genuine responses with AUROC up to 96.9 %.
Table 7 reports AUROC 90.2–96.9 for the semantic axis versus 59.3–60.7 for the length‑only baseline.
**Table 7.** Differentiating reward-hacking with an adapted U-SPACE construction. Results for differentiating reward-hacking responses at the end-of-turn token, read at the pre-registered depth of two-thirds (layers 40, 42, and 27 for Gemma 4, Qwen3.5, and Magistral 1.1, respectively). AUROC evaluates alignment with the constructed semantic-anchor axis, whereas length-only AUROC uses response token count as the sole score.
Overall, the transfer experiments demonstrate that a basis built without task‑specific supervision can still provide meaningful, category‑specific judgments, though it remains blind to subtle, non‑obvious cheating strategies.
Extended Results
Extended experiments expose how uncertainty estimators react to trace length, J‑LENS ablation, and model‑specific performance.
We evaluate a suite of uncertainty estimators—including Mean NLL, Self‑Certainty, DeepConf, TokUR, Semantic Entropy, SentenceSAR, and supervised probes—across three language models (Gemma 4, Qwen3.5, Magistral 1.1) and report both length‑matched and unadjusted metrics.
Multi‑pass methods require an extra forward pass; for example, P(True) asks a self‑evaluation prompt and scores “True” versus “False”. Table 10 shows the resulting accuracies for each benchmark.
**Figure 7.** The reward-hacking contrast, decoded. Top, one of the 100 matched pairs, with the sentence addressed to the judge tinted. Below, per model, the fifteen highest-scoring vocabulary items for the hacked-minus-genuine direction over the 70 construction pairs at the end-of-turn token.
These opposite trajectories stem from different aggregation rules rather than contradictory notions of uncertainty; MSP, which depends on the first scored token, stays nearly constant, and the Acone signal changes only modestly before saturating.
TokUR attains the highest AUROC on Gemma‑4, reaching 75.1 %.
Table 12 reports TokUR’s AUROC as 75.1 % ± 0.4 for Gemma 4.
U‑LENS achieves the top AUROC on Qwen3.5, with 73.2 %.
Table 12 lists U‑LENS AUROC for Qwen3.5 as 73.2 % ± 0.5.
Generation‑length alone is a strong predictor on Gemma‑4, yielding 70.1 % AUROC.
Table 12 shows the “Generation length” row for Gemma 4 with AUROC 70.1 % ± 0.5.
**Figure.** At every budget, we verify that the answer remains the model's greedy continuation. The prediction and its correctness are therefore fixed, so systematic score changes reveal sensitivity to the inserted reasoning tokens.
Table 10 records the baseline accuracies for the three benchmarks; Gemma 4 attains 89.0 % on MMLU‑Pro, 80.7 % on Omni‑MATH, and 63.1 % on SuperGPQA, establishing a performance ceiling for uncertainty estimation.
Token-Level Readout
Token-level readout reveals distinct performance patterns across supervised methods.
This appendix reports token‑level readout metrics for the supervised baselines examined in Section F.4. Each block lists fourteen measurements (mean ± standard error) for a distinct experimental condition.
Across the first experimental condition the supervised baselines achieve a modest average score.
Mean of the fourteen measurements is 14.6 ± 0.3.
The second condition yields substantially higher average performance.
Mean of the fourteen measurements is 68.9 ± 1.2.
In the third condition the average score rises to the high‑30s.
Mean of the fourteen measurements is 37.2 ± 0.9.
The fourth condition returns to low‑teens average performance.
Mean of the fourteen measurements is 13.4 ± 0.4.
The fifth condition again shows strong average scores in the low‑60s.
Mean of the fourteen measurements is 62.6 ± 0.5.
The sixth condition produces mid‑40s average scores.
Mean of the fourteen measurements is 46.3 ± 1.1.
The seventh condition yields an average in the high‑20s.
Mean of the fourteen measurements is 27.3 ± 0.2.
These patterns illustrate that supervised methods transfer unevenly across token‑level contexts, excelling in some regimes while struggling in others.
Causal Intervention and Baselines
Assess transfer robustness, runtime costs, and causal steering limits of supervised uncertainty probes.
We examine how well supervised uncertainty probes transfer across benchmarks, how much extra compute they require, and how far causal steering can push a model along single‑category directions.
Feature‑Gaps measures how far a model’s predictions deviate from a simple linear fit to the training data, exposing systematic gaps that correlate with uncertainty types.
For causal intervention we steer Qwen3.5 along the four simplex directions (ambiguity, incompleteness, conflicting evidence, general uncertainty) by adding the corresponding component of $s_t$ at every block whose relative depth lies in $[0.50,0.85]$.
**Figure 14.** **Steering strength, incompleteness**. The prompt, the unsteered answer, and the continuation after adding the **incompleteness** direction to the prompt states at $\lambda = 0.04, 0.08$, and $0.09$.
**Figure 16.** **Steering strength, general uncertainty.** The prompt, the unsteered answer, and the continuation after adding the general uncertainty direction to the prompt states at $\lambda = 0.03, 0.10$, and $0.17$.
Jacobian Analysis
Key details on Jacobian construction, readout depth, and related appendix contents.
Appendices C–H enumerate the auxiliary experiments that support the main analysis: C shows contrastive activation differences as evidence that categories belong to the model; D extends the construction to emotion and reward‑hacking poles; E details checkpoints, datasets, prompts, generation settings, scoring protocols, and baseline implementations; F reports trace‑length analyses, per‑model unadjusted results, cross‑benchmark transfer, and runtime costs; G provides token‑by‑token readout on single answers; and H describes the steering protocol and the impact of injecting or ablating a direction.
The mean Jacobian $\bar{J}_\ell$ averages the effect of an intermediate residual representation on current and future final‑layer representations, computed over neutral prompts $x\sim R$ and source positions $ t $ with $t' \ge t$; this yields a matrix that, when composed with the unembedding $ W_U $, defines a residual‑stream direction $g_{\ell,a}$ for each anchor token $ a $.
For Gemma 4 and Qwen 3.5 we use publicly released lenses fitted on WikiText; for Magistral 1.1 we fit a lens on 100 WikiText‑103 prompts of length 128, estimating the lens by back‑propagating from the target residual stream without updating any parameters, which distinguishes this procedure from probe training.
The readout layer $\ell^\ast$ is fixed at two‑thirds of each model’s depth (layers 40/60, 42/64, 27/40) following prior work that found the J‑LENS reads best after the first third; sweeping the R‑LENS (which modifies the backward pass) alongside the J‑LENS shows strongly correlated A‑cone curves with no early‑layer improvement.
Constructing the $U$‑SPACE uses two anchor sets: uncertainty anchors $ A^{+} $ (61 seed cues from Chen et al. 2018) and certainty anchors $ A^{-} $ (348 high‑certainty words from Rocklage et al. 2023); subtracting their embeddings removes shared components and isolates the direction that separates doubt from confidence.
**Figure 5.** Readout depth and lens ablation. (a) Readout depth ablation across all model layers. (b) Lens comparison between J-LENS and R-Lens fitted on the identical data by Blank et al. [2026]. Both panels are read from the seed 42 rather than averaged over the three generation seeds.
Anchor Word Analysis
Anchor words reveal how uncertainty and certainty manifest in the U‑SPACE.
The appendix lists the “anchor words” that populate the two poles of the $U$‑SPACE: positive anchors ($A^+$) signal uncertainty, while negative anchors ($A^-$) signal certainty. For each of the four uncertainty categories—Ambiguity, Incompleteness, Conflicting evidence, and General uncertainty—the table shows ten representative words per side, together with the token count $n$ for Gemma 4.
**Table 3.** Anchor words of the U-SPACE. Positive (uncertainty) anchors are from Chen et al. [2018] and negative (certainty) anchors are from Rocklage et al. [2023]. Shown are the ten words per side whose pulled-back directions align best with their side's centroid at the readout block, averaged over the three models. $n$ is the number of words per anchor set that form a single token for Gemma 4 (the text gives the unfiltered counts).
To assign a word to a category, each word and a one‑line definition of the category are embedded with a sentence encoder independent of the evaluated models. The similarity of a word to each definition is z‑scored across the four categories, and the word is placed in the category with the highest z‑score.
**Table 4.** Anchor words as a keyword score. Each trace's thinking block is scored by the number of distinct words of $\mathcal{A}^+$ it contains, or by that number minus the number of distinct words of $\mathcal{A}^-$. AUROC $\times 100$, unadjusted and length-matched.
The lexical baseline shows that anchor tokens are rare (present in 23 %–34 % of Gemma 4 traces and only 4 % of TriviaQA traces), yet $U$‑LENS performance is essentially unchanged on cue‑free traces (68.6 % AUROC versus 68.4 % on the full set). This indicates that $U$‑LENS does not rely on the presence of these cue words, whereas the simple keyword score does.
Experimental Details
Details on models, benchmarks, prompts, generation, and scoring procedures.
We use the model checkpoints published on Hugging Face, pinned to the revisions Qwen3.5‑27B6, Gemma‑4‑31B‑it7, and Magistral‑Small‑2507. The lenses and their revisions are given in Appendix A. Layer indices follow the paper’s convention: Gemma‑4 reads the hidden state at layer 40 of 60, Qwen3.5 at layer 42 of 64, and Magistral 1.1 at layer 27 of 40.
We evaluate on four benchmarks—MMLU‑Pro, Omni‑MATH, SuperGPQA, and TriviaQA—covering multi‑domain reasoning, olympiad mathematics, graduate‑level expertise, and factual QA. For Omni‑MATH we use the rule‑verifiable subset with difficulty ≥ 4.25; for SuperGPQA we use the hard subset. From each pool we draw 1,000 items by proportional stratified sampling with seed 42. Table 8 lists the sources, pools, and stratification schemes.
All prompts are zero‑shot and placed entirely in the user turn, with no system prompt beyond the model’s chat template. Prompt formats follow each benchmark’s official instruction.
- Multiple‑choice (MMLU‑Pro): “Think step by step and then finish your answer with ‘the answer is (X)’ where X is the correct letter choice”.
- Omni‑MATH: each problem ends with “Please reason step by step, and put your final answer within \boxed{}”.
- TriviaQA: “Answer the following question in a single brief but complete sentence”.
We generate three responses per item using sampling seeds 41, 42, and 43 and the model‑specific sampling parameters from Table 9. The generation budget is 65,536 tokens for Gemma‑4 and Qwen3.5, and 32,768 tokens for Magistral 1.1 (its context window is 40,960 tokens). A response is scored when it completes within the budget, and answers are extracted from the text following the reasoning block using each benchmark’s standard rule. Table 10 reports the number of scored responses and the corresponding accuracies (mean ± std over the three seeds).
Scoring uses the recorded response. Single‑pass baselines read the next‑token distributions from a teacher‑forced pass over the entire generation. The U‑LENS reads its entropy factor over the thinking block (Section 3.2) and its cone alignment at the end‑of‑thinking marker. For length‑controlled metrics we split scored responses into ten quantile bins of generated‑token count and compute the metric within each bin.