Thinking Inertia: LLMs Keep Thinking When Told Not To
Dianqiao Lei, Kevin Qinghong Lin, Pan Lu, Philip Torr, James Zou
LLMs exhibit "Thinking Inertia," where explicit reasoning persists even when models are instructed to answer directly.
Do LLMs actually stop "thinking" when prompted to provide only a final answer, or do they continue to generate hidden reasoning?
Large Language Models (LLMs) are increasingly marketed with "no-thinking" modes, yet users have no reliable way to verify if these models are actually suppressing their internal reasoning traces. The authors decompose model responses into pre-answer text and final answers, evaluating them across three levels of strictness: empty-thinking compliance, question-pre-answer relevance, and explicit inference detection. They find that explicit reasoning persists even under strict controls, a phenomenon they term "Thinking Inertia," which becomes more prevalent as the answer space opens from Boolean to open-ended tasks.
Paper Primer
The paper challenges the assumption that "no-thinking" settings or simple instructions effectively stop an LLM from generating reasoning traces. By analyzing six models across six intervention modes, the authors demonstrate that visible explicit inference is not just a function of model settings, but is structurally tied to the complexity of the answer space.
The core mechanism is a three-level evaluation framework: Empty-Thinking Rate (ETR) for strict answer-only compliance, Question-Pre-answer Relevance (QRel) for semantic alignment, and Explicit Inference Rate (EIR) for visible reasoning. This framework acts as a filter: it separates simple answer-only output from relevant but non-inferential text, and finally from explicit, step-by-step reasoning.
Explicit reasoning is highly resistant to suppression, especially in open-ended tasks.
In open-ended tasks, even under the strictest "no-thinking" instructions, models continue to produce explicit reasoning in over 60% of outputs. The Explicit Inference Rate (EIR) follows a consistent "staircase" pattern: lowest on Boolean tasks, intermediate on multiple-choice, and highest on open-ended tasks.
There is a fundamental trade-off between answer-only compliance and task accuracy.
Forcing models to suppress reasoning in open-ended tasks reduces the Explicit Inference Rate but results in a ~15 percentage point drop in accuracy. This trade-off is largely absent in Boolean and multiple-choice tasks, where accuracy remains stable despite successful reasoning suppression.
Why is "no-thinking" not just a simple toggle in modern LLMs?
The authors show that "no-thinking" is a capability dependent on the task's answer space rather than a simple switch. Models exhibit "Thinking Inertia," where they default to reasoning when the answer space is open-ended because the task requires more intermediate computation to reach a correct conclusion.
How does this paper differ from previous work on "efficient reasoning"?
Previous work focuses on optimizing or compressing reasoning traces to save compute. This paper focuses on the behavioral reality of the output: it asks whether a model actually stops "thinking" in its visible response when requested, regardless of what happens in its hidden states.
Researchers and developers should stop treating "no-thinking" as a reliable model setting. Achieving concise, direct answers requires task-aware routing rather than universal suppression, as forcing direct answers on complex tasks currently sacrifices accuracy.
Introduction: The Persistence of Thinking
Introducing the experimental gap: why no‑thinking prompts fail to suppress model reasoning.
LLMs often continue internal reasoning even when prompted to give only a final answer, exposing a gap between instruction‑level no‑thinking controls and actual model output.
Even when a model is instructed to suppress reasoning, it tends to retain and emit explicit inference traces, especially as the answer space widens.
The model first emits the pre‑answer sentence “The acceleration due to gravity on Earth is approximately 9.8 m/s².”
It then follows with the final answer “9.8 m/s²”.
This illustrates that even strict no‑thinking prompts leave a visible reasoning trace, revealing the gap the paper aims to measure.
**Figure 1.** No-thinking controls do not guarantee response-level no-thinking. (Left) On a representative MMLU question, DeepSeek-V4-Flash and Qwen3-235B-A22B are asked for only the final option, yet expose question-relevant pre-answer text. Appendix Figure E.1 confirms the instruction's clear answer-only reading. (Right) Under native no-thinking, green ETR and orange visible-length bars across the six benchmarks show Boolean tasks closer to answer-only than multiple-choice and open-ended tasks.
We measure no‑thinking by normalizing any response into a pre‑answer trace $T$ and a final answer $A$, then evaluating three levels: Empty‑Thinking Rate (ETR) for strict answer‑only compliance, Question–Pre‑answer Relevance (QRel) for similarity between the question and $T$, and Explicit Inference Rate (EIR) for visible logical steps.
A Response-Level Evaluation Framework
We detail the experimental protocol for measuring no‑thinking and introduce response‑level metrics.
To quantify “no‑thinking” we design a controlled experiment that varies the intervention applied to the prompt while keeping model architecture and data fixed.
Instead of looking only at the final answer, we split the model’s output into any pre‑answer text $T$ and the answer $A$, then evaluate each part separately.
How does response‑level evaluation differ from traditional surface proxies?
Surface proxies only inspect the final token stream (e.g., length or presence of special tags). Response‑level evaluation also looks at the intermediate text $T$, measuring its relevance and explicit inference, thus exposing reasoning that proxies cannot see.
Three common shortcuts—prompt‑side control, special reasoning tokens, and short generation length—are used to force a model into a “no‑thinking” mode.
Experimental Results and Analysis
We assess how thinking controls shape pre‑answer content, accuracy, and metric trade‑offs across models and tasks.
Switching from THINK‑ON (M1) to THINK‑OFF (M2) cuts explicit inference (EIR) by up to 85 % on Open‑ended tasks.
EIR drops from roughly 99 % to 14 % across the six evaluated models.
Think‑on (M1) lets the model emit any pre‑answer text; Think‑off (M2) tries to suppress it; M3‑M6 add increasingly strong prompt constraints.
How does M3 differ from a standard step‑by‑step prompt?
In M3 the step‑by‑step instruction is applied while the native thinking switch remains off; a standard step‑by‑step prompt assumes the model’s internal reasoning is fully enabled, so M3 isolates the effect of the external prompt from the internal switch.
**Table 3.** LLM baselines for evaluation.
**Figure 4.** Think-on (M1), Think-off (M2), and Step-by-step (M3) across six models and three answer-space families. Rows show modes; the columns pair ETR bars with EIR points (left) and QRel. bars with visible pre-answer length for the four models with complete length traces (right). Gray dashed segments mark six-model means for ETR and QRel.; black triangles connected by dark lines mark the mean EIR; colored marks follow the model legend.
**Figure 5.** On Open-ended tasks, strict answer-only prompting exposes a trade-off between answer-only compliance and task accuracy. Columns show Bool, MCQ, and Open-ended answer spaces; rows show Acc., ETR, QRel., and EIR. M2 is the native no-thinking baseline; M4 and M5 add increasingly strong natural-language constraints, while M6 forms a separate structural prefix-forcing branch. Colored lines show individual models; the black line and gray band show the mean and one standard deviation. For ETR, these summaries include only models with a disabled-thinking condition.
**Figure 6.** Changing only the answer space reshapes no-thinking behavior on the same questions. We rewrite matched numeric-answer items from MMLU and MMLU-Pro for Qwen3-32B under M2 from MCQ into Boolean or open-ended forms. The four panels report Acc., ETR, QRel., and EIR on the same answer-space variants; red values give the change from the MCQ form, computed before rounding.
Native no‑think modes (M2) often fail to suppress reasoning, and performance is highly dependent on the answer space.
Related Work
Survey of prior work on selective reasoning and instruction‑following constraints.
Efficient and Adaptive Reasoning research explores how test‑time computation can be allocated selectively, with methods that control reasoning length via reinforcement‑learning budgets, token‑budget prompts, budget forcing, or self‑training for concise traces, and approaches that compress intermediate reasoning by distilling shorter chains or using terse drafts. Empirical work shows that longer traces are not uniformly beneficial—excessive length can hurt efficiency and even accuracy, and the optimal reasoning length varies with task difficulty and model capability. Recent studies train models to decide when extended reasoning is needed and extend selective reasoning to vision‑language models.
The instruction‑following literature treats “no‑think”, “answer‑only”, and prefix‑forcing modes as directives whose compliance depends on models’ ability to obey natural‑language constraints, with instruction tuning, human‑feedback alignment, and synthetic instruction data expanding the instruction distribution. Benchmarks such as IFEval and FollowBench evaluate adherence to output‑format and content constraints, while works on latent‑reasoning (e.g., Coconut) and trace inversion demonstrate that visible reasoning may not reflect underlying computation. Our work frames “produce no visible reasoning” as a verifiable output constraint and evaluates interventions using Explicit Text Ratio, Question Relevance, Explicit Inference Rate, and task accuracy.
Discussion and Conclusion
We outline the limits of evaluating no‑thinking at the response level and discuss future directions.
We argue that “no‑thinking” must be assessed at the response level, using the decomposition $Response = T + A$ to isolate answer‑only compliance, pre‑answer relevance, and explicit inference, while evaluating task accuracy on $A$ alone. Across local Qwen3 models, scaled variants, and frontier APIs, we observe that explicit no‑think controls fail to eliminate visible inference, that “thinking inertia” aligns with answer space, and that stricter answer‑only constraints introduce a compliance‑accuracy trade‑off. Moreover, compressibility is driven by the structural support of the question rather than its difficulty or model capability.
This table outlines the model groups, their respective thinking control mechanisms, and the decoding and evaluation scope used for different modes.
Evaluating no‑thinking as a response‑level property reveals inherent trade‑offs and underscores the need for controllable, auditable stopping mechanisms rather than simple output length restrictions.
Appendix: Experimental Details
Details on dataset splits, model interfaces, and inference settings used in the appendix analyses.
Appendix B.1 lists the split sizes for each benchmark (BoolQ, StrategyQA, MMLU, MMLU‑Pro, GSM8K, MATH) and the sampling strategy applied to the API models.
Table B.2 details the model groups, their thinking‑control interfaces, and the uniform decoding settings used for all modes.
The minimal setting is treated as near‑off rather than a strict off, and provider‑side reasoning signals are consulted only for strict $|T| = 0$ decisions.
Appendix: Evaluation Protocol
Details the evaluation setup, metrics, and aggregation methods used for appendix experiments.
We adopt the response‑level protocol from Section 2.3, decomposing each model output into pre‑answer text $T$ and final answer $A$. All appendix results report the same Accuracy, Explicit Text Ratio (ETR), Question Relevance (QRel), and Explicit Inference Rate (EIR) metrics as in the main paper.
Provider‑specific reasoning signals are consulted only to confirm the answer‑only boundary when they are available, ensuring that the evaluation focuses on the intended response format.
Supplementary Table C.1 outlines the evaluation scope for each appendix result block, linking experimental setups to their corresponding metrics.
Local Qwen models (Qwen3‑4B and Qwen3‑32B) are tested on six benchmarks—BoolQ, StrategyQA, MMLU, MMLU‑Pro, GSM8K, and MATH—across Thinking Modes 1–6. Frontier models (Qwen3‑235B‑A22B, DeepSeek‑V4‑Flash, GPT‑5.4, Gemini‑3‑Flash‑Preview) use each provider’s public thinking‑control interface on the same suite.
Full MMLU domain runs for Qwen3‑4B, Qwen3‑32B, DeepSeek‑V4‑Flash, and Qwen3‑235B‑A22B are grouped by original MMLU super‑categories and, where applicable, by individual subjects.
Additional experiments probe MMLU variants with choice shuffling, numeric labels, and boolean verification, as well as matched MCQ, numeric‑source items, boolean verification, and open numeric generation.
We also evaluate surface and candidate‑visibility interventions, alongside a numeric‑source generation intervention.
Aggregated results are visualized in Figure G.1 and Table H.1 for global local‑model summaries and Qwen scaling evidence, while frontier no‑think and suppression/forcing outcomes appear in Figures H.1, G.2 and Table H.1.
Domain‑level and subject‑level analyses are presented in Figure H.2, Tables I.1 and I.2; question‑interface controls in Tables I.3 and I.4; and a candidate‑removal stress test in Table I.5.
Run aggregation follows the mode numbering from Section 2.4, using item‑weighted averages for grouped Bool, MCQ, and open‑ended rows. Frontier aggregates employ an 8,192‑token rerun with seed 20260502.
QRel is computed against the complete visible pre‑answer text, including returned reasoning fields. GPT‑5.5 evaluates the full corpus, and the resulting EIR appears in global, frontier, scaling, and domain analyses, while question‑structure controls examine relevance and consistency.
**Table.** Result block, Models and task scope, and Appendix evidence supported.
Appendix: Validation of Evaluation
We validate encoder robustness, judge consistency, and human annotation reliability across three evaluation levels.
We validate the three‑level evaluation by probing encoder robustness, judge consistency, and human annotation reliability.
The table compares the performance of two encoders, all-MiniLM-L6-v2 and Qwen3-Embedding-4B, across two evaluation settings (M2: Think-Off and M5: Strict) and three task types (Bool, MCQ, Open).
The table presents performance metrics for three different encoders (all-MiniLM-L6-v2, BGE-base-en-v1.5, and E5-base-v2) across two evaluation settings (M2: Think-Off and M5: Strict) and three answer types (Bool, MCQ, and Open).
**Table D.4.** Paraphrase-only outputs do not inflate EIR. The five-way GPT-5.5 rubric separates semantic paraphrases from explicit inference even when a paraphrase has little lexical overlap with the question. Table D.4 reports the central M2/M5 settings using the same all-output denominator for both categories. Across the complete evaluation, only 355 of 572,688 outputs (0.06%) are labeled paraphrase-only. Because only the explicit-reasoning label contributes to EIR, these paraphrases, together with relevant non-inferential text, are excluded from the visible-reasoning result.
The table presents data comparing two modes (M2 and M5) across three answer spaces (Bool, MCQ, Open) in terms of "Explicit reasoning" and "Paraphrase-only" percentages. The text below the table notes that both judges preserve the relations Bool < MCQ < Open under M2 and M5, and M5 < M2 in every answer space.
**Supplementary Table D.5.** GPT-5.5–Opus 4.8 agreement across the complete evaluation. Each group pools the other experimental dimensions. Percentages are calculated over outputs for which both judges have a usable label.
**Supplementary Table D.7:** Per-annotator agreement with the blinded Level 3 judges on all 600 outputs. Five-way agreement requires the same rubric label; binary agreement asks only whether the judge and annotator agree on whether the output is explicit reasoning.
**Supplementary Table D.8.** Inter-annotator reliability on the 402 non-empty pre-answer texts.
The table presents the performance metrics across different modes (M2, M5) and answer spaces (Bool, MCQ, Open) for five annotators (A1-A5), including the calculated mean for each row.
Appendix: Robustness Audits
Robustness audits quantify rare copying, paraphrase, and answer‑first patterns across models.
We audited 572,688 successful records from six models, applying three detectors to bound possible measurement failures: near‑verbatim copying, semantic paraphrase, and answer‑first ordering.
The copying detector flags a record when its pre‑answer text shares at least 80 % of four‑grams with the question stem and length differs by less than 1.5×, yielding 301 flagged copies (0.053 %).
Under a pessimistic scenario where all flagged copies receive QRel = 1 solely due to copying, the global mean QRel could shift by at most 0.00053, far smaller than the observed differences in the main figures.
Semantic paraphrases are captured by a separate rubric that labels 355 outputs (0.06 %) as paraphrase‑only, which are excluded from the EIR numerator, ensuring that question restatement does not inflate inference rates.
Answer‑first patterns are identified by locating the first boxed answer within three lexical tokens of the start and requiring three tokens after it, finding 149 such cases (0.028 %).
**Table.** Analysis of response patterns across different models.
"Give the final answer directly without any reasoning." means: > Provide only the final result or conclusion, and do not show the step-by-step thinking, explanation, derivation, or analysis that led to it. For example: **Question:** What is 27 × 14? * With reasoning: > 27 × 14 = 27 × (10 + 4) = 270 + 108 = 378. * With "Give the final answer directly without any reasoning": > 378 Another example: **Question:** Which is larger, 0.5 or 3/7? * With reasoning: > 0.5 = 1/2 = 0.5, while 3/7 ≈ 0.4286, so 0.5 is larger. * Direct answer only: > 0.5
Appendix: Case Studies
Five regimes show varied pre‑answer length and relevance while all answers stay correct.
Across the five regimes, pre‑answer length and question relevance vary widely, yet every final answer is correct.
See Table 19, which lists $|T|_w$ from 0 to 144 words and $QRel$ from 0.000 to 0.896 for the five cases.
A regime groups a model‑mode pair by the length of its visible pre‑answer text and the semantic relevance of that text to the question.
Visible pre‑answer $|T|_w$ = 23 words.
Computed $QRel$ = 0.896, the highest among all cases.
Final answer selected = “B”, matching the gold option.
Even a brief pre‑answer can achieve a correct answer when its relevance to the question is high.
The table presents five cases involving different regimes, models, modes, records, $|T|_w$ values, and QRel scores. Below the table, a shared input for Cases 1–4 is provided: "As of 2017, the share of GDP spent on the military by the United States is about: A. 1%, B. 3%, C. 6%, or D. 10%. The gold option is B."
This table presents four cases of model behavior analysis: Native residual information, Re-elicitation, Compression cost, and Candidate visibility. Each row details the record setting (model, dataset, item, and mode), the observed pattern of the model's response, and the resulting interpretation regarding model behavior and interface constraints.
Appendix: Global Results
Global trends of accuracy and ETR across six models and six modes.
All six models lose accuracy when switching from native no‑think (M2) to strict answer‑only (M5).
Figure 8 left heatmap shows a consistent drop in accuracy for every model across the six datasets.
**Supplementary Figure G.1:** Global six-mode profiles for local Qwen models. The plotted values are the corresponding dataset-level aggregates; EIR is averaged over the three answer-space families for each model-mode cell.
**Supplementary Figure G.2:** Dataset-level view of the no-thinking-performance trade-off. The left panel shows the accuracy change from native no-think (M2) to strict answer-only suppression (M5); the right panel shows the resulting M5 ETR. The plotted values are the corresponding dataset-level aggregates.
Appendix: Detailed Tables
Detailed tables and figures expand the compact results, showing scaling and no‑think patterns.
This section provides the full tables and figures that underpin the compact results presented earlier, giving readers the raw numbers behind scaling trends and no‑think behavior.
The “Thinking Inertia Frontier” row summarizes native no‑think behavior, followed by three metric rows: M2 ETR, M2 QRel., and M2 EIR, each reported for Bool, MCQ, and Open answer types.
Model‑level results list Qwen3‑235B, DeepSeek, GPT, and Gemini across the three answer formats, showing a wide spread in Explicit Text Ratio, Question Relevance, and Explicit Inference Rate.
Interpretation of Table H.1: larger models improve accuracy, yet the hierarchical answer‑space structure persists—BoolQ and StrategyQA compress most easily, while GSM8K and MATH remain resistant, and M6 often reverts toward M2 rather than extending M5.
**Supplementary Figure H.1:** Frontier native no-think heatmaps by answer family. The three panels report the same native M2 outputs at successive levels: strict answer-only compliance (ETR), instruction-aware pre-answer relevance (QRel.), and explicit visible inference (EIR). The staircase is visible in all three panels: Bool is easiest to compress, MCQ is intermediate, and Open retains question-relevant and inferential content across providers.
Supplementary Table H.1 (table21) compares Qwen models (4B, 32B, 235B) across three modes (M1 THINK‑ON, M2 THINK‑OFF, M3 COT) and multiple datasets, with blue, orange, and green shading encoding Accuracy, ETR, and QRel. respectively.
Supplementary Table H.2 (table22) reports Level 3 Explicit Inference Rate (EIR) for the two central no‑thinking settings (M2 THINK‑OFF and M5 STRICT), confirming the same Bool < MCQ < Open ordering across all model sizes.
Table H.3 (table23) presents performance percentages for Qwen3‑4B, Qwen3‑32B, and Qwen3‑235B under M2 THINK‑OFF and M5 STRICT, again showing Open tasks dominate while Bool tasks often collapse to near‑zero.
**Supplementary Figure H.2:** MMLU domain heatmaps under the same multiple-choice answer interface. The left panel shows answer-only success under native no-think; the right panel shows the accuracy change caused by strict answer-only suppression. The full area-level control table is given in Table I.1.
Supplementary Table I.1 (table24) provides full MMLU area‑level controls, reporting Accuracy, ETR, and EIR for four models across four domains (Humanities, Social Sciences, STEM, Other) and four modes (M1, M2, M5, M6).
Note: Qwen3‑4B and Qwen3‑32B runs use local native `enable_thinking` controls, whereas Qwen3‑235B is evaluated via a provider‑mediated API, ensuring consistent mode numbering as defined in Section 2.4.
Appendix: MMLU Metrics
This section details MMLU area and subject metrics across models and modes.
Supplementary Table I.1 presents area‑level results for four modes (M1–M6) across four models, reporting Accuracy, Explicit Text Ratio (ETR), and Explicit Inference Rate (EIR) for each MMLU domain.
Supplementary Table I.2 breaks down performance by individual subjects within Humanities, Social Sciences, STEM, and Other, again showing Accuracy and ETR (with darker orange indicating higher ETR) for the same four modes.
**Supplementary Table I.2.** Subject-level MMLU breakdown using the original MMLU supercategories. Each mode cell reports Acc./ETR; darker orange means higher ETR. Mode 1 reports Acc. only.
Appendix: Interpretations and Perturbations
Interpretation of perturbation experiments and their effect on reasoning metrics.
Table I.2 breaks down performance by subject area. Quantitative subjects (e.g., physics, math) show lower Explicit Text Ratio (ETR) and higher accuracy loss when compressed, whereas fact‑heavy subjects retain high ETR with little accuracy drop.
Table I.3 evaluates whether the observed MMLU patterns stem from answer‑label names or option ordering. The modest deltas in Accuracy, ETR, and Question Relevance (QRel) across “choice shuffle” and “numeric labels” variants, together with high consistency scores, suggest that neither label semantics nor option position drive the main effects.
**Supplementary Table I.3:** MMLU surface perturbations with the answer set fixed. Deltas are relative to the base version; darker colored cells indicate larger absolute changes.
Table I.4 rewrites each MMLU item as a yes/no verification problem with a single candidate answer displayed. Under the no‑think regime, ETR increases and QRel decreases, yet the Pos/Neg split shows the model does not simply echo the presented candidate; it still discriminates between correct and incorrect proposals.