Beneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs

Seogyeong Jeong, Jaehui Hwang, Dongyoon Han, Geonmo Gu, Alice Oh, Taekyung Kim

Reasoning operations in LLMs form distinct, separable geometric structures in hidden representations that persist across models and tasks.

Large language models generate reasoning traces as sequences of tokens, but it remains unclear whether these traces are internally organized by the functional reasoning operations they express. The authors map reasoning traces to a taxonomy of eight operations—such as decomposition, deduction, and arithmetic—and use supervised probes to test if these operations occupy distinct, separable regions in the model's hidden representation space. Reasoning operations are indeed linearly separable in middle layers, with representations that are context-dependent and distributed across spans rather than tied to specific lexical cues.

Paper Primer

The paper investigates whether the functional roles of reasoning steps—like breaking a problem into sub-tasks or performing a calculation—are encoded as distinct geometric structures in an LLM's hidden states. By applying a Polya-inspired taxonomy to reasoning traces, the authors demonstrate that these operations are not just textual labels but are represented as consistent, separable directions in the model's internal geometry.

Reasoning operations are linearly separable in hidden representations, with separability peaking in middle layers.

Probes trained on held-out spans achieve high macro AUROC across multiple models (Qwen2.5-7B, Qwen3-8B, Gemma4-31B), outperforming text-only baselines. Hidden-state probes outperform the strongest text-only (bag-of-words/TF-IDF) baselines by up to 0.097 in AUROC and 0.193 in AUPRC.

The method hinges on the "operation-span" approach: segmenting reasoning traces into functional chunks and fitting supervised linear probes to identify operation-specific directions. This reveals that identical surface tokens acquire different representations depending on the operation of their surrounding chunk, and that preceding reasoning context causally shapes the representation of subsequent operations.

Operation-level geometry persists even when the model makes factual errors.

Probes trained on correct traces remain highly predictive when applied to spans containing factual errors, showing only a modest attenuation in separability. Macro AUROC for factual-error spans remains high at 0.955 (vs. 0.971 for non-error spans).

Why does this study use a functional taxonomy rather than just analyzing the raw tokens?

The authors argue that reasoning traces are not homogeneous sequences but sequences of functional chunks; a functional taxonomy allows them to test if the model's internal geometry reflects the *role* of a reasoning step (e.g., planning vs. execution) rather than just the surface words used to express it.

Does this structure simply emerge because the model is following a fixed, predictable sequence of steps?

No; the authors control for positional and lexical confounds. They show that position-only classifiers perform significantly worse than hidden-state probes, and that the operation-aligned structure persists even when controlling for lexical similarity and digit density.

Introduction

We expose the hidden‑space gap between textual reasoning steps and model representations.

Reasoning in large language models proceeds through distinct functional steps such as problem formulation, goal decomposition, and deduction. Although these steps are explicitly written in the model’s output, we do not know how they are arranged in the model’s hidden representation space. This section frames the gap our work aims to fill.

A reasoning operation is a logical step expressed in text that guides the model toward a solution, e.g., formulating the problem or deducing a sub‑result.

**Figure 1.** Overview of reasoning-vector analysis. Given an input problem and the model-generated reasoning trace, we probe hidden representations with reasoning operation vectors corresponding to different reasoning operations. The left panel shows token-level reasoning operation-alignment scores over the generated text, while the right panel projects span representations onto Decomposition and Recall reasoning-vector axes. Regions with strong reasoning operation-specific signals appear along the corresponding directions.

We find that hidden representations separate reasoning operations, with the strongest separability in middle layers. Operation signals spread across spans, causing identical tokens to be encoded differently depending on their surrounding operation. Attention‑masking experiments confirm that preceding context causally shapes subsequent operation representations, and the geometry degrades when the model makes factual errors.

Preliminary

Defines a taxonomy of reasoning operations and frames the separability question for LLM hidden states.

Reasoning LLMs produce a trace $r = (t_1,\dots,t_N)$ before emitting the final answer $y$, with the joint probability factorized as $p(r, y \mid x) = \prod_{i=1}^{N} p(t_i \mid x, t_{<i}) \cdot p(y \mid x, r)$. For each token position $i$ we extract hidden representations $h_i^{(\ell)} \in \mathbb{R}^d$ from layer $\ell$. The central question is whether the geometry of these representations encodes the functional role of each reasoning step.

We view the trace as a sequence of contiguous chunks, each chunk performing a distinct reasoning operation that aligns with Pólya’s four‑stage problem‑solving framework.

**Table 1.** Overview of main reasoning operations for our analyses. These eight frequent operation types are used as span labels in our representation analysis.

We next ask whether hidden representations of different operations are geometrically separable, and whether any observed structure can be reduced to lexical or positional artifacts. Subsequent subsections probe layer‑wise evolution, span‑level distribution, and the effect of attention masking, culminating in an analysis of how factual errors perturb the operation geometry.

Operation Separability

We examine whether hidden states truly encode reasoning operations beyond surface cues.

Do hidden representations truly capture the reasoning operation a token participates in, or do they merely reflect surface lexical patterns? The paper first asks whether operation‑level separability exists in held‑out hidden states, then probes whether this signal survives a battery of lexical controls.

Separability means that a token’s hidden vector can be distinguished from others solely by the reasoning operation it belongs to, regardless of the surrounding words.

To rule out the possibility that lexical cues drive this separability, the authors introduce four controls: (1) text‑only classifiers, (2) lexically matched span comparisons, (3) competing‑operation vocabulary subsets, and (4) digit/formula density restrictions.

**Figure 3.** Visualization of reasoning-operation clusters in the learned LDA space. For each target operation, held-out span representations from Qwen3-8B are projected onto a two-dimensional plane whose x-axis is the corresponding reasoning operation vector $d_c$ and whose y-axis is the first principal component of the residual representations. Points are colored by their annotated reasoning operation labels.

**Figure 5.** Identical tokens acquire reasoning-dependent representations. For pairs of reasoning operations, we collect common surface tokens that appear in both operation contexts and visualize their layer-wise scores along the two corresponding reasoning operation vectors in Qwen3-8B. Each point is an occurrence of a shared token and is colored by its annotated reasoning label.

**Table 2.** Macro AUROC/AUPRC of position-only logistic regression, the stronger of bag-of-words and TF-IDF logistic regression, and mean-pooled hidden-state probes. Position and lexical content are informative, but hidden-state probes achieve higher aggregate performance across all three models.

Further robustness checks show that supervised LDA sharpening is not essential (PCA alone yields AUROC ≈ 0.94), that operation separability persists across relative trace‑position bins, and that the phenomenon generalizes to Llama‑3‑8B and transfers to GPQA‑Diamond and MATH‑500 without retraining.

Within-Span Structure

We examine whether reasoning signals are confined to a few cue tokens or spread across whole spans.

Section 3.2 showed that reasoning operations become most separable in middle layers, but it left open whether that separability stems from a handful of cue tokens or from information distributed across the whole span.

To test the cue‑local hypothesis, we compute for each token $t$ in a span $s_i$ its alignment $a(l, y_i) d(l) y_i$ and then measure the variance of these token‑level scores within the span, denoted $Var(l)$. High $Var(l)$ indicates that only a few tokens dominate the signal, whereas low $Var(l)$ suggests a more uniform distribution.

Across layers we observe that $Var(l)$ is relatively high in early layers, drops sharply in the middle layers, and rises modestly again near the final layers, implying that middle‑layer separability is not driven by isolated cue tokens but by a broadly distributed operation‑aligned representation.

**Figure 4.** Quantitative analysis on intra-span LDA score variances. We investigate the LDA score variances within each span across layers using Qwen3-8B. Early-layer separability is often sparse and token-local, while mid-late-layer separability is distributed across the tokens. We further generalize the results for Qwen2.5-7B and Gemma4-31B in the Appendix.

Next we ask whether lexical identity alone can explain the alignment signal. For each pair of reasoning operations we intersect the ten most frequent token identities, project each shared token’s hidden state at every layer into the probe space, and compute its alignment with both operation vectors.

Masking the 30 tokens immediately preceding a reasoning chunk reduces the target operation‑alignment score across all operation categories, showing that the onset of a new chunk’s representation depends on information from the preceding context rather than solely on the chunk’s own tokens.

**Figure 6.** Preceding context shapes operation representations. Masking attention to preceding context reduces target reasoning operation-alignment score, indicating that subsequent reasoning operations depend on prior reasoning context.

Masking the entire preceding annotated chunk produces a qualitatively similar reduction in alignment scores, though the magnitude differs because the masked span length and eligible examples differ; a random between‑chunk control further validates that the observed effect is not an artifact of sample selection.

Generalization and Robustness

We examine whether operation‑level geometry survives factual errors, projection choices, span representations, and lexical or positional confounds.

Operation‑level geometry is a promising signal, but we must know whether it survives real‑world perturbations. The authors therefore probe robustness across factual errors, projection methods, span representations, positional cues, and lexical confounds. This section enumerates a battery of controls to answer that question.

In the factual‑error analysis (Section C.3), probes trained on correct traces are applied to spans that contain explicit errors. Even in error‑laden spans, operation identity is still detectable: macro $AUROC$/$AUPRC$ reach 0.920/0.759 (middle‑token) and 0.955/0.877 (mean‑pooled). However, Deduction and Arithmetic Computation suffer significant drops, Recall is unchanged, and Algebraic Manipulation shows a mixed pattern.

**Table D.** Transfer of reasoning-operation probes trained exclusively on correct reasoning traces to factual-error and operation-matched non-error spans from incorrect Qwen3-8B reasoning traces. Results are averaged across layers 10–30. The error and non-error groups contain equal numbers of spans for each operation. Reported $p$-values are from one-sided paired Wilcoxon signed-rank tests of whether separability is higher for non-error spans than for factual-error spans. Values smaller than 0.001 are reported as $< 0.001$.

To test whether the supervised $LDA$ projection drives the observed separability, the authors repeat the analysis using only $PCA$ (Section D.1). With 8, 32, and 128 principal components, macro $AUROC$ stays above 0.84, and gains plateau after 32 components. Thus $LDA$ sharpens the signal but is not essential for the core effect.

**Table B.** Llama-3-8B replication. M and Mid denote mean-pooled and middle-token span representations.

Span‑representation choice is another potential confound (Section D.2). The authors evaluate middle‑token, mean‑pooled, first‑token, and last‑token representations. All choices preserve the qualitative ordering of operations, with mean‑pooled often yielding stronger early‑layer separability because it aggregates token‑level cues.

Position information alone can predict some operations (Section D.3.1). By encoding each span as a 50‑dimensional binary position vector, a logistic‑regression classifier attains above‑chance $AUROC$ for Extraction and Final Answer, yet the hidden‑state probe consistently outperforms this baseline.

**Table F.** Train + Test occurrence counts for each reasoning operation and model.

When spans are stratified by start position (Section D.3.2), operation labels remain distinguishable within each interval, indicating that position contributes but does not fully account for the hidden‑state geometry.

Lexical cues are examined next (Section D.4). A bag‑of‑words / $TF\!-\!IDF$ classifier (Section D.4.1) is informative, yet the hidden‑state probe surpasses it for all models except Decomposition, where stereotyped wording gives the text baseline a slight edge.

**Table I.** Macro AUROC/AUPRC of the stronger text-only baseline among TF-IDF and BoW and the mean-pooled hidden-state probe.

Lexically matched pair analysis (Section D.4.2) directly tests whether operation identity survives when lexical similarity is high. Positive median $Δ$ values for all eight operations demonstrate that the probe captures operation semantics beyond surface word overlap.

**Table J.** Lexically matched pair analysis. Positive effects indicate that the frozen probe is more strongly aligned with operation identity than with broad lexical similarity. Confidence intervals are computed across anchor-level effects.

Competing‑operation vocabulary subsets (Section D.4.3) further control for cue words. Probes remain predictive when spans contain top‑TF‑IDF words from rival operations, indicating that a few salient tokens cannot override the overall operation signal.

Digit and formula density (Section D.4.4) could trivially explain Arithmetic Computation separability. Restricting evaluation to spans with 50 %–75 % numeric tokens still yields above‑chance $AUROC$, showing that density alone is insufficient.

Statistical reliability is assessed via 5 000 stratified bootstrap replicates (Section D.5). All 48 $AUROC$ confidence intervals stay well above the 0.5 chance line, confirming that the operation‑level separability is stable under resampling.

Taxonomy and Annotation

We define a taxonomy of reasoning operations and annotate spans to enable separability analysis.

The paper lacks a systematic way to refer to the diverse reasoning steps LLMs generate, which hampers any analysis of whether different operations occupy distinct representation subspaces. Without a shared taxonomy and reliable span annotations, claims about operation‑level separability remain ambiguous.

To populate the taxonomy, annotators were given a prompt that listed the eight canonical operations, span‑selection rules, and a JSON output format. Each reasoning trace was segmented into non‑overlapping spans, and annotators selected the operation that best described the functional role of each span.

Human validation sampled up to 12 spans per label (84 spans total) and assigned each to three annotators, yielding a majority‑vote label and a Fleiss’ kappa of 0.666. This validation confirms that the taxonomy can be applied consistently by humans.

**Table L.** Evaluation on spans with 50–75% digit or mathematical-token density. The theoretical chance AUROC is 0.500 for all rows.

**Table M.** Bootstrap confidence intervals for reasoning-operation separability. We report layer-averaged AUROC and AUPRC with 95% confidence intervals for mean-pooled and middle-token representations. Intervals are computed from 5,000 stratified bootstrap replicates.

**Table N.** Human validation of GPT-5 operation-span annotations. Each span was independently labeled by three of seven human annotators.

**Figure.** Probing results for different reasoning steps across layers for Qwen models. The left column shows AUROC and the right column shows AUPRC. Panels (a) and (b) correspond to Qwen3-8B, (c) and (d) to Qwen2.5-7B, and (e) and (f) to Qwen2.5-14B.

**Figure F.** Quantitative analyses on separability of reasoning operations in representation spaces at position mean. AUROC and AUPRC of one-vs-rest operation classifiers are shown across layers for Qwen3-8B, Qwen2.5-7B, and Gemma4-31B. Solid lines denote true reasoning-operation labels, while dashed lines denote random-label and random-position baselines. Across models, performance remains above the baselines and typically peaks in middle layers.

**Figure G.** Quantitative analyses on separability of reasoning operations in representation spaces at position first. AUROC and AUPRC of one-vs-rest operation classifiers are shown across layers for Qwen3-8B, Qwen2.5-7B, and Gemma4-31B. Solid lines denote true reasoning-operation labels, while dashed lines denote random-label and random-position baselines. Across models, performance remains above the baselines and typically peaks in middle layers.

**Figure H.** Quantitative analyses on separability of reasoning operations in representation spaces at position last. AUROC and AUPRC of one-vs-rest operation classifiers are shown across layers for Qwen3-8B, Qwen2.5-7B, and Gemma4-31B. Solid lines denote true reasoning-operation labels, while dashed lines denote random-label and random-position baselines. Across models, performance remains above the baselines and typically peaks in middle layers.

**Table O.** Reasoning operation schema (Stages 1–2).

Experimental Setup

Describes the data, models, annotation pipeline, and hidden‑state probing used to evaluate reasoning operations.

The experimental setup must provide fine‑grained reasoning traces and a way to test whether individual layers encode the eight operation types.

We need two complementary corpora—one heavy on computation, one heavy on theorem knowledge—and a shared taxonomy so that spans from any model can be compared.

Each correct model response is broken into contiguous token spans, each tagged with one of the eight operation types, providing the ground truth for probing.

How does this span‑level labeling differ from token‑level classification used in prior probing studies?

Token‑level probes treat each token independently, ignoring the logical grouping of tokens that together implement a reasoning step. Span‑level labeling respects the contiguous operation boundary, allowing the probe to capture the collective semantics of the whole step rather than a noisy mixture of unrelated tokens.

We treat the hidden representation of the middle token of each span as a feature vector, reduce its dimensionality, and learn a linear separator that points toward the operation’s prototype direction.

Compute the “not‑E” mean: $\mu_{\neg E} = (\mu_A + \mu_R)/2 = (-0.5,0.5)$.

Prototype direction for Extraction: $d_E = \mu_E - \mu_{\neg E} = (1.5,-0.5)$.

Alignment of a test span $z_{test}= (0.8,0.2)$ with Extraction: $a = (0.8,0.2) \cdot (1.5,-0.5) = 1.2 - 0.1 = 1.1$.

Higher $a$ indicates the test span is more likely to be Extraction than the other classes.

The dot‑product score directly reflects how close a span’s representation lies to the operation’s prototype direction, making AUROC a natural measure of separability.

**Figure 2.** Layer-wise separability of reasoning operations across language models in middle token position. (A) Peak-layer one-vs-rest AUROC for each reasoning operation in each model, with 95% confidence intervals. (B) AUROC averaged across reasoning operations at different model depths (embedding, early, middle, and late layers), along with a random baseline. Across models and reasoning operations, separability remains high and is strongest in the middle layers.

With this pipeline in place, the subsequent sections evaluate how separability varies across layers and model families.

Erroneous Execution Experiments

Operation geometry survives factual errors but its separability drops modestly.

Operation-level geometry remains detectable under factual errors but its separability drops modestly.

Probes achieve AUROC 0.955 on error spans versus 0.971 on correct spans (mean‑pooled), and 0.920 versus 0.937 (middle‑token). -0.016 AUROC (mean‑pooled) and -0.017 AUROC (middle‑token)

Attenuation varies by operation: Deduction and Arithmetic Computation exhibit the clearest AUROC drops, while Recall shows virtually no change.

**Figure A.** Layer-by-token heatmaps of reasoning-operation alignment scores. We visualize token-level alignment scores with the corresponding reasoning-operation vector across layers for representative reasoning traces. Each heatmap shows how strongly each generated token aligns with a target reasoning operation vector at each layer. The examples illustrate that operation-specific scores are often localized or weak in early layers, become more coherent over contiguous token spans in middle layers, and may become less sharply localized in later layers. This supports the view that reasoning-operation representations emerge through contextual processing rather than being attached only to isolated cue tokens.

**Figure B.** Additional layer-by-token heatmaps of reasoning-operation alignment scores. We provide additional representative examples for other reasoning-operation types. Consistent with Figure A, the alignment scores become more structured across contiguous generated tokens in middle layers, while early-layer scores are more sparse and token-local. These qualitative patterns are consistent with the span-level separability and intra-span variance analyses.

**Figure C.** Quantitative analysis on intra-span LDA score variances across additional models. We report the intra-span LDA score variances across layers for Qwen2.5-7B and Gemma4-31B. Consistent with Qwen3-8B, variance is high in early layers and decreases toward middle layers, indicating a shift from token-local cues to operation signals distributed across the span.

**Table A.** Exploratory onset-to-onset temporal-ordering analysis. For each operation, $\delta_{onset} = t_{late} - t_{early}$, where the two onsets are detected using the same causal three-token rolling-mean procedure and regime-specific thresholds. Negative values indicate that the sustained late-layer signal is detected before the sustained early-layer signal under this criterion. The fraction column reports $Pr(\delta_{onset} < 0)$. The reported $p$-values are from uncorrected one-sided Wilcoxon signed-rank tests for a negative location shift. A significant $p$-value does not necessarily indicate that negative onset differences occur in a majority of examples.

**Figure D.** Additional attention-masking intervention results. We report context-intervention results beyond the pre-token setting shown in the main text. In the pre-chunk setting, we mask attention from the first token of a target reasoning operation chunk to the entire immediately preceding annotated reasoning operation chunk. In the random-control setting, we mask a randomly selected preceding block with a comparable length. Across settings, blocking preceding context generally reduces the target reasoning operation-alignment score, indicating that subsequent reasoning-operation representations depend on prior context. The reduction is stronger for structured preceding-context masking than for the random control, supporting the view that relevant preceding reasoning context contributes to the formation of the next operation representation.

Even when reasoning steps err, LLMs preserve a recognizable operation geometry, though with reduced separability.

Appendices and References

Defines the taxonomy of reasoning operations used throughout the paper and documents the annotation protocol.

The paper needs a unified way to describe the fine‑grained steps that solvers perform, because without a shared taxonomy it is impossible to compare human reasoning, model traces, and evaluation metrics.

The authors break problem solving into four stages—Planning, Carrying Out the Plan, Looking Back, and Final Answer—each decomposed into hierarchical levels of concrete operations.

**Table P.** Reasoning operation schema (Stage 3).

**Table Q.** Reasoning operation schema (Stage 4).

**Table R.** Summary of the human annotation procedure and interface. The instructions were originally provided to annotators in Korean.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers