Agentic Abstention: Do Agents Know When to Stop Instead of Act?

Agentic Abstention measures when agents should stop acting on infeasible tasks, using CONVOLVE to learn stopping rules.

How can we train LLM agents to recognize when a task is infeasible and stop acting, rather than hallucinating an answer or wasting compute on impossible goals?

LLM agents often continue interacting with environments long after a task has become impossible, wasting compute and potentially causing errors. This "agentic abstention" problem is particularly difficult when infeasibility is only revealed through interaction, rather than being obvious from the initial request. CONVOLVE addresses this by distilling full interaction trajectories into a reusable playbook of stopping rules, which are appended to the agent's context to guide future decisions. On WebShop, this method more than doubles the timely abstention recall for Llama-3.3-70B, raising it from 26.7% to 57.4% without requiring parameter updates.

Paper Primer

Agentic abstention is a sequential decision problem where an agent must choose between answering, abstaining, or continuing to act. Unlike static question-answering, where a model decides once, an agent's context evolves, meaning the realization that a task is unachievable often occurs mid-trajectory.

CONVOLVE functions as a context-engineering loop: a reflection model analyzes past trajectories to identify signals of infeasibility, and a curator model distills these into a structured playbook. The agent then treats this playbook as a dynamic set of "lessons" to consult before taking further actions.

Agentic abstention is a persistent failure mode across diverse environments.

Across web, terminal, and QA benchmarks, most tested models achieve less than 50% average abstention recall, with timely abstention (stopping at the first possible moment) consistently below 40%. Most agents fail to stop immediately, often acting for many turns on tasks that are fundamentally unsolvable.

CONVOLVE significantly improves timely abstention without model retraining.

Using only 20 training trajectories, Llama-3.3-70B improved its timely abstention recall on WebShop from 26.7% to 57.4%. The method is data-efficient and transferable; lessons learned by smaller models (8B) can effectively improve the performance of larger models (70B).

Why is this problem harder than standard LLM abstention?

Standard LLM abstention is a single-turn decision. Agentic abstention is a sequential process where the agent has a rich action space (search, click, terminal commands) and the environment state changes, meaning the agent must continuously re-evaluate if the goal remains reachable.

Does model scale or reasoning capability solve this automatically?

No. The authors find that larger models and increased reasoning effort do not guarantee better timely abstention; while they may improve eventual abstention, they often fail to stop at the earliest possible moment.

Reliable agentic systems require "stopping judgment" as much as "task-completion ability." CONVOLVE demonstrates that this judgment can be offloaded to a context-based playbook rather than requiring expensive model fine-tuning.

Introduction: The Problem of Agentic Action

We define Agentic Abstention and expose its challenges for LLM agents.

LLM agents act over many turns, invoking tools and observing environments to fulfill user goals. When a goal is ambiguous or impossible, agents frequently keep issuing tool calls that cannot succeed, wasting time and resources. Recognizing such infeasibility early and stopping is essential.

Agentic Abstention is the ability of a tool‑using agent to decide, at any turn, that further interaction will not help and to stop acting.

How does Agentic Abstention differ from the usual LLM “answer‑or‑abstain” setting?

Standard LLM abstention is a one‑shot choice between answering and refusing. Agentic Abstention adds a third option—continue interacting—and requires the model to decide at each turn whether more evidence could change the outcome, making it a multi‑step control problem.

To study this problem we built a benchmark of over 28 000 instructions spanning web‑shopping, terminal interaction, and question answering. We adapted 16 datasets from AbstentionBench and added two new abstention families: Request‑based Abstention, where the instruction is deliberately ambiguous, and Environment‑based Abstention, where the environment is altered so the original request becomes impossible (e.g., removing all red shirts).

Evaluating 13 LLM‑as‑Agent systems and two scaffolds reveals a consistent difficulty: models often recognize infeasibility only after many wasted steps. Timely abstention recall (AbsRec@1) stays below 30 % on web tasks and under 25 % on terminal tasks, while overall recall can be higher but still far from perfect. Moreover, larger or more capable models do not guarantee better timely abstention; reasoning improves it in some cases but can also reduce overall recall.

We introduce CONVOLVE, a context‑engineering technique that distills full interaction trajectories into reusable stopping rules and appends them to the agent’s prompt. On the WebShop benchmark, CONVOLVE raises timely abstention recall for Llama‑3.3‑70B from 26.7 % to 57.4 % without any parameter updates.

The cost of unconstrained agentic action is high; early, informed abstention can cut waste dramatically.

Defining Agentic Abstention

We build three balanced abstention datasets covering web, terminal, and QA scenarios.

Evaluating whether agents can recognise infeasibility is hard because real‑world applications span many environments—web interfaces, command‑line terminals, and open‑ended QA. Existing benchmarks focus on solvable tasks, leaving a gap in systematic measurement of abstention behaviour. This section introduces three curated datasets that fill that gap.

Web‑based decision‑making uses the WebShop simulator. We keep the original 500 solvable instructions and add 500 rewrites that force abstention, split evenly between request‑based (subjective preference, underspecified intent, false premise) and environment‑based “Missing Target” cases where the catalog is altered.

Terminal‑based evaluation adapts Terminal‑Bench 2.0. After preserving the 89 original solvable tasks, we generate 167 request‑based rewrites (false premise or underspecified intent) and 21 environment‑based “Missing Prerequisite” variants that delete required files or services.

Interactive QA draws from AbstentionBench. We filter 16 source datasets to 27 073 samples, covering five abstention categories and converting each into a sequential decision problem with a limited‑search action space.

AbstentionBench aggregates diverse QA tasks and annotates each with a binary “should‑abstain” label, enabling systematic measurement of an agent’s ability to refuse answering when uncertain.

**Figure 1.** This is an Environment-based Abstention example in a web shopping scenario, where the agent only discovers that the task is infeasible after interacting with the environment. We show three trajectories, (i) Timely success: where the agent abstains in the earliest possible step where is has enough information to do so, (ii) Delayed success: where the agent eventually abstains correctly following a few steps of unnecessary tool calls, and (iii) Failure to abstain: where the agent issues unnecessary tool calls for the remaining turns and does not abstain within the 10-turn budget.

**Figure 2.** (a) Each adapted task in TerminalBench 2.0 consists of four core components: (1) a containerized environment initialized with the relevant packages and files, (2) an instruction describing the task to be completed, (3) a set of tests for verifying completion, and (4) a manually written reference solution. For our abstention setting, we rewrite the original instruction to construct abstention-warranted variants. (b) Examples of rewritten instructions under two abstention scenarios: False Premise or Contradiction and Underspecified Intent.

Taxonomy of Infeasible Tasks

Defines the two abstention modes and how they let agents stop futile tasks.

LLM agents often waste interaction steps by persisting on tasks that cannot be completed, draining resources and confusing users.

When a user’s request contains contradictory premises, subjective preferences, or vague intent, the agent can infer that a reliable answer is impossible and should abstain.

The agent issues a search query for “red unicorn”.

The search returns zero results.

After the first failed attempt, the agent checks the request type and classifies it as “Missing Target”.

The agent emits an abstention response: “I cannot find a red unicorn in the catalog.”

By recognizing the missing target early, the agent avoids further futile queries and saves interaction budget.

How is Request‑based Abstention different from simply saying “I don’t know”?

It is a purposeful decision based on detecting that the task itself is unsolvable, not a lack of factual knowledge. The agent reasons about the request’s feasibility before replying.

When the execution environment lacks a required resource—such as a file, permission, or external service—the agent discovers the missing prerequisite after limited probing and then abstains.

The agent runs “ls” to list files in the project directory.

It observes that “Makefile” is absent.

Recognizing the missing prerequisite, the agent aborts the compile request.

It replies: “Cannot compile because the required Makefile is missing.”

Detecting the missing prerequisite early stops a cascade of failed commands and informs the user promptly.

Why not keep trying different commands until the task succeeds?

Because the missing prerequisite (e.g., the Makefile) is a static deficiency that cannot be resolved by repeated attempts; continued retries only waste time and may produce misleading error output.

Measuring Abstention Performance

Formal episode‑level metrics quantify both correctness and timeliness of agent abstention.

Evaluating abstention demands more than a binary success flag; we must know if the agent abstains when it should and how early it does so.

We need a family of episode‑level scores that reward correct abstention and penalize both missed and premature stops, while also measuring how quickly the agent reacts once abstention becomes justified.

Episode 1: $w_1=1$, $\tau_1=1$ → delay $d_1=0$, counts for AbsRec@1, AbsRec@2, … and contributes $S_1=1$, $L_1=1$, $P_1=1$ to SPL.

Episode 2: $w_2=2$, $\tau_2=\infty$ → no abstention, $d_2=\infty$, contributes $0$ to all AbsRec@K and $S_2=0$ to SPL.

Episode 3: $w_3=3$, $\tau_3=4$ → delay $d_3=1$, counts for AbsRec@2…AbsRec@10 but not for AbsRec@1; $S_3=1$, $L_3=3$, $P_3=4$, SPL contribution $3/\max(4,3)=3/4=0.75$.

Compute metrics: AbsRec@1 = 1/3 ≈ 33 %; AbsRec@2 = 2/3 ≈ 67 %; AbsRec@10 = 2/3 ≈ 67 %; SPL = (1·1/1 + 0 + 1·3/4)/3 = (1 + 0 + 0.75)/3 ≈ 0.58.

Even a single late abstention (episode 3) boosts overall recall but reduces SPL, illustrating how SPL captures timeliness beyond the binary AbsRec@K ladder.

How does AbsRec@K differ from a standard recall metric that simply counts “did the agent ever abstain?”

Standard recall treats any eventual abstention as a success, ignoring whether the decision was made after sufficient evidence. AbsRec@K adds a deadline $K$, rewarding only those abstentions that occur within $K$ steps after the earliest justified moment $w_i$, thus explicitly measuring timeliness.

The CONVOLVE Framework

CONVOLVE equips agents with a learnable playbook that signals when to stop interacting.

Agents often keep acting after a task becomes impossible, wasting interaction steps and resources. CONVOLVE addresses this by turning the agent’s own rollouts into a reusable “playbook” that tells future episodes when to quit.

CONVOLVE treats each episode as a short notebook entry: after the episode finishes, a reflection model writes a concise rule into a shared playbook, and that playbook is appended to the agent’s prompt for the next episode.

How does CONVOLVE differ from simply fine‑tuning the model on abstention examples?

Fine‑tuning changes the model’s internal weights and requires retraining for every new domain. CONVOLVE leaves the base model untouched; it instead writes explicit, human‑readable rules into a playbook that the model reads at inference time, making the abstention behavior modular, reversible, and cheap to update.

Trajectory $\tau^{(1)} = (x^{(1)}, o^{(1)}_1, a^{(1)}_1, o^{(1)}_2, a^{(1)}_2, o^{(1)}_3, \text{abstain})$ is recorded.

The reflection model produces $y^{(1)} = $ “abstain after observing $o^{(1)}_2$” because the task became infeasible at step 2.

Update $c^{(1)} = U(c^{(0)}, \tau^{(1)}, y^{(1)})$ adds a rule “if observation $o_2$ then abstain” to the “late‑abstain” section.

In a second episode the agent sees the same observation $o_2$; the injected playbook matches the rule and triggers abstention at step 2 instead of step 3.

The example shows that a single concise rule can shift the abstention point earlier without altering the underlying model parameters.

Run the agent on a training example and record the full trajectory $\tau^{(k)}$.

Feed $\tau^{(k)}$ to the reflection model $\phi$ to obtain feedback $y^{(k)}$.

Pass $c^{(k)}$, $\tau^{(k)}$, and $y^{(k)}$ to the deterministic updater $U$ to produce $c^{(k+1)}$.

Append the updated playbook $c^{(k+1)}$ to the system prompt for the next rollout.

Repeat for all training examples (up to two reflection/update rounds per example).

We instantiate CONVOLVE on the abstention‑only slice of WebShop, which contains 500 examples split into four scenarios: Subjective Preference, Underspecified Intent, False Premise/Contradiction, and Missing Target. After stratified sampling we keep 101 examples for held‑out evaluation, use 20 for the CONVOLVE run, and 8 for validation.

**Table 1.** CONVOLVE improves abstention on WebShop. Trained variants use 20 trajectories.

For evaluation we run the same 70B model on the 101‑example held‑out set, once with the CONVOLVE playbook injected and once without any playbook (baseline). All other settings—provider, temperature, token limit, interaction limit, timeout, and retry policy—remain identical.

Experimental Results

We show how CONVOLVE boosts timely abstention and analyze influencing factors.

The CONVOLVE framework trains agents to recognize infeasible tasks early, reducing wasted interaction.

CONVOLVE more than doubles timely abstention recall for Llama‑3.3‑70B.

Timely recall rises from 26.7 % to 55.3 % on the Web scenario after training on 20 trajectories.

Across Web, Terminal, and QA settings, agents often fail to abstain promptly; AbsRec@1 typically lies between 0 % and 0.3 %.

Model scale mainly lifts overall recall, while reasoning mode influences timely recall; scaffolding choices dominate Terminal performance.

**Figure 3.** Abstention is hard for agents, especially timely abstention. Abstention Recall increases with larger K, but early abstention (e.g., AbsRec@1) remains low across settings and systems. This suggests that agents often abstain only after unnecessary interaction, rather than when abstention first becomes warranted.

**Figure 4.** AbsRec@K across Abstention Categories in Web, Terminal, and QA scenarios. From top to bottom, the rows show Web, Terminal, and QA results. Missing Target in Web, Underspecified Intent in Terminal, and False Premise and Underspecified Intent in QA are the most difficult cases across models. Performance for the same model varies significantly across abstention categories.

**Figure 5.** More reasoning leads to some improvement in AbsRec@1 while overall abstention recall (AbsRec@10) decreases on both Web and Terminal scenarios.

**Figure 6.** Cumulative over-abstention rate by turn in Web and Terminal scenarios on solvable instances.

**Figure 7.** Scaling improves overall recall but not timely recall.

Robustness and Scaling

Additional ablations show CONVOLVE consistently boosts abstention across both benchmarks.

We evaluate CONVOLVE on the additional AbstentionBench and TerminalBench suites, using the same stratified splits described in §G.1.

CONVOLVE lifts AbsRec@1 on TerminalBench for Llama‑3.3‑70B from 41.6 % to 66.2 %.

Table 3 shows the base model at 41.6 % and the CONVOLVE‑augmented Llama‑3.3‑70B + 70B at 66.2 %.

CONVOLVE raises SPL on TerminalBench for Llama‑3.3‑70B from 23.6 to 46.2.

Table 3 reports SPL = 23.6 for the base model and SPL = 46.2 for the CONVOLVE‑enhanced Llama‑3.3‑70B + 70B.

**Table 1.** Performance comparison of base models, base models with in-context learning, and base models with CONVOLVE across AbstentionBench and TerminalBench datasets.

Qualitative Analysis of Abstention

Qualitative examples illustrate where abstention fails and what the failures imply.

This section presents three representative failure modes that expose limits of the abstention mechanism.

The ablations reveal three distinct failure modes where the agent does not abstain.

FalseQA premise correction, ALCUNA question‑answering miss, and TerminalBench grounding loss.

The model identifies the false premise that CO₂ consists solely of oxygen.

It corrects the premise and answers that CO₂ contains one carbon atom and two oxygen atoms.

This shows the agent can recognize and rectify false premises, so abstention is unnecessary.

The model detects the role confusion between lawyer and judge.

It answers that a lawyer represents a client while a judge decides the verdict.

The correction avoids abstention because the task remains feasible after premise fixing.

Turn 1: Retrieval returns unrelated passages about American bison hunting, a Saskatchewan town, and European bison herd dynamics.

Turn 2: The agent answers “herd” despite lacking a grounded source for Bison bonison.

The agent proceeds with an ungrounded answer instead of abstaining, exposing a gap in detecting infeasible tasks.

Step 1: The agent searches for tokenizer metadata to determine vocabulary size.

Step 2: It fails to locate any tokenizer files or libraries.

Step 3: Rather than abstaining, it defaults to a guessed vocabulary size n = 100 256 and proceeds with optimization.

This illustrates that missing environmental grounding leads the agent to fabricate defaults instead of abstaining.

Experimental Configuration

Describes the models, scaffolds, scenarios, and metrics used to evaluate agentic abstention.

We evaluate a representative set of recent open‑weight and API‑based LLMs, including OpenAI GPT‑5.4‑mini, Grok 4.1 Fast, Llama 3.3 {8B, 70B} Instruct, GPT‑OSS 120B, Minimax M2.5, Qwen 3 {8B, 14B, 32B, 235B A22B {Instruct, Thinking}}, Gemma 4 31B it, and GLM 5.1.

For the agentic scaffolds we use Codex CLI and Terminus 2, treating Terminus 2 as a neutral testbed for model comparison.

Scenario‑specific coverage limits exhaustive model–scaffold pairing. The WebShop scenario serves as the main cross‑model setting, evaluating eight LLM‑as‑agent systems (GPT‑5.4‑mini, Grok 4.1 Fast, Llama‑3.3‑70B‑Instruct, GPT‑OSS‑120B, MiniMax‑M2.5, Qwen‑3‑235B‑A22B‑Instruct, Gemma‑4‑31B‑it, GLM‑5.1).

In the QA scenario we evaluate five models (GPT‑5.4‑mini, Llama‑3.3‑70B‑Instruct, Qwen‑3‑235B‑A22B‑Instruct, GLM‑5.1, Gemma‑4‑31B‑it) under a retrieval setup, while Terminal‑Bench fixes the base model to GPT‑5.4‑mini and compares Terminus 2 with Codex CLI across reasoning‑effort settings.

Secondary analyses include scaling experiments with Qwen‑3 models at 8B, 14B, 32B, and 235B on Web tasks; reasoning experiments contrasting Qwen‑3‑235B‑Instruct and Thinking on Web tasks and varying reasoning effort for GPT‑5.4‑mini on Terminal tasks; and reporting over‑abstention for solvable terminal tasks.

Proprietary models are accessed through their first‑party APIs, whereas open‑weight models are accessed via Together AI and OpenRouter.

Evaluation metrics focus on capturing timely abstention behavior.

Timely Recall is a special case: for request‑based abstention $w_i=1$, so Timely Recall equals AbsRec@1; for environment‑based abstention $w_i=2$, Timely Recall equals AbsRec@2.

We also report SPL (Success weighted by normalized inverse path length), adapted from embodied navigation to penalize delayed abstention decisions.

Over‑abstention rate quantifies the proportion of solvable instances on which the agent incorrectly abstains.

Related Work and Broader Impacts

We discuss societal implications and situate our work among prior LLM abstention research.

LLM agents operating with tools or browsers can fail not only by giving wrong answers but also by taking unnecessary actions, consuming resources, or expressing misplaced confidence when a task is underspecified or infeasible.

Encouraging evaluation of appropriate non‑completion helps agents recognize missing information, unavailable resources, contradictory instructions, or infeasible goals and communicate this clearly, reducing hallucinations and wasted tool use.

However, excessive abstention harms usefulness, frustrates users, and may shift responsibility back to them for tasks the agent could have solved.

Our benchmark and CONVOLVE method are intended as research tools; real‑world deployments will need to combine abstention with clarification, escalation, human oversight, or domain‑specific safeguards, especially in high‑stakes settings.

In the broader literature, LLM abstention is defined as a model’s decision to refrain from answering under uncertainty, with prior work focusing on unanswerable queries, knowledge limits, and safety‑aligned prompts.

Recent analyses (Wen et al., Kirichenko et al.) examine abstention from query, model, and human‑value angles and organize datasets by scenario, yet they remain limited to single‑turn interactions.

Other studies address related capabilities: uncertainty quantification for agents, tool‑use efficiency (SMART), over‑search harms, theoretical tool‑use boundaries, and selective quitting for safety, all of which inform but differ from our focus on abstention as a distinct decision.

Context‑engineering approaches such as Reflexion, TextGrad, GEPA, Dynamic Cheatsheet, and ACE adapt prompts via feedback; our CONVOLVE framework repurposes this idea to curate execution trajectories into reusable stopping rules.

**Figure 8.** (a) Length distributions show similar token counts between original and rewritten WebShop instructions. (b) t-SNE visualization of instruction embeddings reveals substantial semantic overlap, demonstrating that original and rewritten WebShop instructions are semantically indistinguishable. The similarity breakdown across the three abstention scenarios is shown in Figure 9. (c) Word clouds of original and rewritten WebShop instructions.

**Figure 9.** t-SNE visualizations of original and rewritten WebShop instructions by abstention category: Subjective Preference, Underspecified Intent, and False Premise or Contradiction.

Conclusion and Limitations

We recap the abstention challenge and note CONVOLVE’s limits for reliable agents.

Agentic Abstention asks LLM agents to recognize when a task cannot be completed and to stop interacting rather than persisting.

Across web browsing, command‑line terminals, and interactive question‑answering, agents repeatedly face the decision of whether to continue or to abstain.

In practice they tend to abstain too late—or not at all—expending many wasteful actions, especially when a task looks feasible at first but proves unresolvable after a few interactions.

Our method CONVOLVE mitigates this by distilling full interaction trajectories into compact stopping rules that can be injected as context, eliminating the need to fine‑tune model parameters.

The results indicate that a truly reliable agent must combine stronger task‑completion capabilities with sharper judgment about when further action ceases to be useful.

Dataset Construction Details

Details of how WebShop, Terminal‑Bench, and AbstentionBench data are adapted for abstention evaluation.

We evaluate agents in three interactive settings—web environments, terminal environments, and question answering—using WebShop, Terminal‑Bench 2.0, and the 16 datasets aggregated in AbstentionBench.

WebShop presents textual observations of webpages and a set of clickable actions; its test set contains 12,087 human‑written instructions.

From the first 500 test instructions we create 500 solvable tasks plus 249 request‑based and 251 environment‑based abstention instances, yielding a balanced 1 : 1 split of feasible and infeasible tasks.

**Table 2.** Examples of original and rewritten WebShop instructions for the three abstention categories. We use GPT-5.4-mini to rewrite the original WebShop instructions so that they reflect three categories: Subjective Preference, Underspecified Intent, and False Premise or Contradiction. All rewritten examples are reviewed by the authors, with unsuitable cases revised or filtered out.

Terminal‑Bench 2.0 comprises 89 professionally verified, high‑skill tasks that require long, interdependent action sequences.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers