Agent-Reflex: Teaching Language Agents to Act Reflectively
Guanting Dong, Xinyu Yang, Jiajie Jin, Yuyang Hu, Xiaoshuai Song, Yifei Chen, Xiaoxi Li, Jiejun Tan, Yutao Zhu, Tat-Seng Chua, Ji-Rong Wen, Zhicheng Dou
Agent-Reflex internalizes failure-recovery rules as reusable meta-skills to improve agent robustness in noisy, real-world environments.
How can we train language agents to proactively recover from noisy tool feedback and user interventions without relying on expensive inference-time prompting?
Agents often fail in real-world deployments because they are trained in idealized, noise-free environments that lack realistic tool errors and user corrections. Agent-Reflex addresses this by mining "reflection meta-skills" from success-failure contrasts in noisy environments and internalizing them into the agent's policy through supervised fine-tuning and on-policy distillation. This approach achieves a 58.6 overall score across five long-horizon benchmarks, significantly outperforming existing reflection and environment-scaling methods.
Paper Primer
The paper identifies that current agent training relies on stable, noise-free feedback, leaving agents unable to handle common production issues like service timeouts or user-driven corrections. Agent-Reflex treats reflection as a learnable meta-skill: it mines recovery rules from cross-model success/failure pairs and internalizes them into the policy parameters.
The framework operates in three stages: Environment Reflection Skill Discovery (RSD) mines executable recovery rules from noisy tool-return failures; Reflective Supervised Fine-Tuning (ReSFT) trains the agent on these rules to handle local repairs and user corrections; and On-Policy Reflective Policy Optimization (ORPO) uses reflection-guided distillation to provide dense, token-level supervision during reinforcement learning. ORPO is like a coach watching a student: it compares the student's action to a "privileged" version that has access to the recovery skill, then rewards the student more heavily when their choice aligns with the expert's corrective path.
Agent-Reflex significantly improves performance on long-horizon tool-use benchmarks compared to existing reflection and environment-scaling baselines.
Evaluation on five long-horizon benchmarks (e.g., VitaBench, ACEBench) using a Qwen3-8B backbone. 58.6 overall score, surpassing strong baselines like NoisyAgent-8B and EnvScaler-8B.
The framework enhances agent robustness to interaction noise.
Performance on AgentNoiseBench, which injects perturbations into tool and user feedback. 29.1 score, outperforming the next best 8B method (25.9).
Why is this approach better than simply prompting an agent to "reflect" at inference time?
Inference-time reflection relies on the same policy that failed to generate its own critique, often leading it to repeat the same flawed plan. Agent-Reflex internalizes verified recovery rules into the model parameters, allowing the agent to act proactively without needing external prompts or auxiliary reflectors.
What is the scope of this method — does it require specific environment types?
The method requires database-backed executable environments with verifiable outcomes to mine reflection skills. It is designed for long-horizon tool-use workflows where interaction noise and user intervention are common.
Researchers can now move beyond per-episode hindsight rewrites by mining reusable, environment-indexed reflection skills that generalize across tasks and improve robustness to real-world noise.
The Challenge of Noisy Tool Use
We expose the gap between clean training data and noisy real‑world feedback, motivating Agent‑Reflex.
Large language models now act as long‑horizon agents, but real deployments expose them to unreliable tool services (MCP) and ad‑hoc user corrections. Existing training pipelines assume stable, noise‑free feedback, leaving agents ill‑prepared for failure detection and recovery. Agent‑Reflex addresses this mismatch by internalizing reflective recovery skills that anticipate and correct noisy interactions.
Agents must anticipate noisy tool feedback and user‑driven corrections because their training data is clean, yet deployments are messy.
**Figure 1.** From idealized interaction to reflective recovery. (a) Prior environments assume clean tool feedback, whereas (b) real ones return noisy observations and user corrections at arbitrary turns. (c) Success rate of GPT-5.2 and Claude-4.5-Sonnet under no, user, and tool noise. (d) Share of the eight tool-error families mined from 24,510 real error returns.
The core gap is the mismatch between clean training data and noisy real‑world tool feedback.
Prior Approaches to Agent Reflection
Surveys prior reflection approaches and environment synthesis, then outlines the skill‑discovery pipeline.
Prior work on agent reflection and on scaling environments for tool‑use is reviewed before the paper’s own skill‑discovery pipeline is introduced.
Appends a verbal self‑critique to the model’s output at inference time, using the same policy that generated the answer as the source of the critique.
Similar to Reflexion, Self‑Refine lets the model generate a refinement of its own answer, again without external verification.
Retrieves cross‑task insights at test time, using a separate memory of prior experiences to guide the current answer.
Learns a per‑trajectory repair signal during training, moving the corrective feedback into the loss.
Incorporates reflective signals into reinforcement learning by rewarding trajectories that include successful self‑corrections.
Extends ReflectRL to large‑scale models, learning repair policies from diverse trajectories.
Exposes the policy to perturbed interactions using only a scalar return, encouraging robustness to noisy tool feedback.
Synthesizes executable tools with verifiable tasks, enabling large‑scale generation of training environments.
Creates a diverse set of executable environments by scaling existing tool specifications across many topics.
Builds database‑backed services over hundreds of real‑world themes and verifies outcomes through state transitions.
Internalizes environment dynamics through world rehearsal, allowing agents to practice in a simulated version of the target environment.
The agent generates a short textual critique of its own answer immediately after producing it, using the same policy that generated the answer as the source of feedback.
The model produces a revised version of its answer after an initial generation, using a second forward pass that conditions on the original output.
**Table 5.** LLM agent pool for RSD trajectory sampling. The pool only supplies behavioral contrasts for skill mining; all training trajectories come from Qwen3-8B.
Reflective Supervised Fine-Tuning
ReSFT turns reflection skills into supervised training data for the policy.
Policies trained on clean corpora stumble when tool feedback is noisy or users intervene. The missing piece is a way to teach the policy how to recover from its own mistakes without exposing privileged correction signals at test time. ReSFT supplies exactly that by turning explicit reflection skills into supervised examples.
ReSFT converts each reflection skill into a training pair (input $x$, target $y$) so that a frozen behavior policy $\pi_{0}$ learns to repair its own errors as if it had never seen the skill.
SR: Store the pair $(h_{t^{*}}, y_{\text{SR}})$ because the continuation succeeds; discard $\kappa$ from the stored input.
KR: Reset to the initial state $s_{0}$, prepend $\kappa$ to the original prompt $q$, sample $y_{\text{KR}}=$ “start from revised plan”, and keep $(q, y_{\text{KR}})$ after removing $\kappa$.
UR: A user simulator converts $\kappa$ to a natural‑language correction $u_{\kappa}=$ “please redo the last step”. After restoring $s_{t^{*}}$, we store $(h_{t^{*}} \oplus u_{\kappa}, y_{\text{UR}})$ with $y_{\text{UR}}=$ “executed corrected step”.
Merge: $\mathcal{D}_{\text{ReSFT}}$ now contains the three reflective pairs plus any correct trajectories from $\pi_{0}$ (e.g., a fully successful run). The loss $\mathcal{L}_{\text{ReSFT}}$ is computed over all four examples.
By training on these mixed examples, the policy learns to recover from its own mistakes without ever seeing the privileged skill token during inference.
How does ReSFT differ from ordinary supervised fine‑tuning of a language model?
Standard fine‑tuning only sees correct input–output pairs, so the model never experiences its own error states. ReSFT deliberately injects error‑state contexts (restored $s_{t^{*}}$ or $s_{0}$) and pairs them with successful recoveries, while hiding the privileged skill $\kappa$ in SR and KR. This forces the policy to learn a repair behavior that it can invoke at test time without any extra tokens.
On-Policy Reflective Policy Optimization
ORPO injects reflective guidance into on‑policy RL using mixed clean‑noisy rollouts.
When agents act in environments that occasionally return noisy tool feedback, standard on‑policy RL lacks a signal that tells the policy which token decisions caused the failure. ORPO addresses this gap by turning the agent’s own on‑policy rollouts into a dense, reflection‑guided supervision signal.
ORPO compares each token’s likelihood under the current policy to that under the behavior policy, then scales the RL advantage by a confidence‑weighted, group‑normalized signal that reflects noisy outcomes.
Compute the group mean µ_q = (8+6+10+9)/4 = 8.25.
Compute the variance $\sigma_{q}$² = [(8‑8.25)² + (6‑8.25)² + (10‑8.25)² + (9‑8.25)²]/4 = 2.1875.
Normalize each return: $A_{q,1}$ = (8‑8.25)/√2.1875 ≈ -0.17, $A_{q,2}$ ≈ -1.52, $A_{q,3}$ ≈ 1.18, $A_{q,4}$ ≈ 0.52 ($\epsilon$ omitted for clarity).
Even though the noisy trajectories have lower raw returns, the group‑wide normalization keeps them in the same advantage distribution, allowing the policy to learn from their failures without treating them as outliers.
Sample a task q from the RL task pool $Q_{RL}$.
Generate G on‑policy trajectories, mixing a fraction $\nu$ of noisy rollouts with clean ones.
Obtain returns $R_{q,i}$ for each trajectory and compute group‑normalized advantages $A_{q,i}$.
For every token in every trajectory, compute the log‑probability difference and form the importance weight $\rho_{q,i,t,k}$.
Form the combined loss $L_{\text{ORPO}$} = $L_{\text{RL}$} + \lambda\,\sum_{i,t,k} \rho_{q,i,t,k}\,$A_{q,i}$ ($\lambda$ is a scalar weighting hyper‑parameter).
Back‑propagate the loss and update the policy parameters $\theta$.
**Figure 3. ORPO.** A mixed rollout group of mostly clean and a few perturbed trajectories shares one group-normalized advantage. Re-scoring the same tokens with and without the skill card of their environment yields the detached, confidence-weighted OPD signal.
How does ORPO differ from standard on‑policy methods like PPO?
Standard on‑policy algorithms treat each trajectory’s return independently and use a clipped surrogate objective. ORPO, by contrast, mixes clean and noisy rollouts, normalizes returns across the whole mixed group, and applies a token‑level importance weight that reflects the confidence‑adjusted advantage. This yields dense, reflection‑aware supervision rather than a single scalar surrogate.
Reflection Skill Distillation
RSD turns sparse outcome signals into dense reflective supervision.
Reflection Distillation solves the bottleneck that outcome‑only supervision leaves the policy blind to *how* to recover from tool failures. By converting those sparse signals into token‑level guidance, the agent learns to anticipate and correct mistakes as it acts.
RSD mines failure‑recovery trajectories, builds a library of reflection meta‑skills, and distills them into the policy so that every token receives a reflective supervision signal.
Family A: failed tokens $(t_1, t_2, t_3)$ → skill prompt “repair‑A”.
Family B: failed tokens $(s_1, s_2, s_3)$ → skill prompt “repair‑B”.
Verifier checks that “repair‑A” and “repair‑B” each raise the task success probability above 0.9.
Training pairs become $(t_1\!arrow\!\text{repair-A}_1,\; t_2\!arrow\!\text{repair-A}_2,\; …)$ and similarly for family B.
During fine‑tuning the policy receives a loss on every token of the failed rollout, guided by the corresponding repair token.
Dense token‑level loss forces the model to learn *where* in a trajectory a reflection should be triggered, not just *that* a reflection is needed.
How does RSD differ from ordinary supervised fine‑tuning on the same data?
Ordinary fine‑tuning treats the failed rollout as a single example and learns only from the final outcome. RSD, by contrast, aligns each failed token with a verified corrective continuation, providing a per‑token teaching signal that explicitly encodes the reflective decision.
**Figure 2.** RSD and ReSFT. Stage 1 perturbs tool returns of 557 MCP-anchored executable environments with mined failure families, contrasting success/failure rollouts into an offline skill library. Stage 2 samples local-repair (SR), global-recovery (KR), and user-correction (UR) continuations from that library, keeping only verifier-validated ones with skill prompts stripped.
Experimental Results
Key quantitative gains of Agent‑Reflex across tool‑use and noisy interaction benchmarks.
Agent‑Reflex internalizes reflective recovery skills so agents can act proactively when tool feedback or user instructions are noisy. We evaluate on two benchmark groups: long‑horizon tool‑use suites and the noisy interaction suite AgentNoiseBench, comparing against reflection, environment‑scaling, and RL baselines.
Agent‑Reflex attains the highest overall score on AgentNoiseBench, reaching 29.1 % versus 25.9 % for the strongest baseline.
Table 2 reports the per‑model scores; Agent‑Reflex leads the 8 B family.
On the long‑horizon suite, Agent‑Reflex exceeds Reflexion‑8B and ExpeL‑8B on VitaBench (17.5 % vs 6.8 %), surpasses NoisyAgent‑8B on ACEBench (78.0 % vs 52.0 %), and beats EnvScaler‑8B on $\tau$²‑Bench (62.1 % vs 54.6 %). These gains show that reflective supervision adds reusable recovery knowledge beyond single‑trajectory cues and complements environment‑level meta‑skills.
When trained with On‑Policy Reflective Policy Optimization (ORPO), the agent maintains a stable advantage over GRPO, DAPO, REINFORCE++, SEED, and NoisyAgent RL across 100 RL steps. ORPO’s confidence‑weighted advantage isolates positively‑shifted tokens, preventing neutral tokens from diluting the update and yielding a consistently rising reward curve.
Ablation (Table 3) shows every component matters: removing the Skill‑Recovery (SR) module drops performance most dramatically (45.9 → 41.3 on $\tau$²‑Bench, 12.3 → 8.9 on VitaBench). The remaining modules (KR, UR, OPD, weighting) each contribute incremental gains, confirming the complementary nature of the full system.
AgentNoiseBench is a diagnostic suite that injects realistic tool‑side and user‑side noise into benchmark tasks, measuring how well an agent recovers from perturbed interactions.
How does AgentNoiseBench differ from a standard robustness benchmark that only perturbs inputs?
AgentNoiseBench injects structured, task‑specific failures (e.g., HTTP 401, ambiguous user requests) and evaluates the agent’s ability to recover via learned reflective skills, rather than merely measuring degradation under random input noise.
**Figure 5.** Per-axis Stability-Gated Success on AgentNoiseBench’s native taxonomy, normalized by the three-model mean on each axis (10 = that average). (a) Seven tool-side axes. (b) Six user-side. * marks a lead clearing a 95% paired-bootstrap interval against both baselines, † the backbone alone.
**Figure 4.** Comparison of agent policy optimization algorithms. All runs start from Qwen3-8B and train for 100 RL steps on the same task pool under matched budgets. (a) Final accuracy on held-out benchmarks. (b) Mean training reward on the same tasks.
This table lists various AI model providers, their corresponding model families, and the specific checkpoints used for trajectory sampling.
Supplementary Analysis and Details
Ablation studies quantify the impact of removing Reflective Recovery components.
The appendix enumerates related‑work surveys (A.1–A.3), dataset descriptions (B.1–B.4), baseline families (C.1–C.4), implementation details (D.1–D.5), and supplemental experiments (E.1–E.3), followed by limitations, broader impacts, and ethics.
Agent‑Reflex derailed only 28.5 % of rollouts, whereas the EnvScaler‑8B baseline derailed 44.8 %.
Stability‑Gated Success on AgentNoiseBench shows a lower derailed‑trajectory rate for Agent‑Reflex.
The backbone model derailed 56.1 % of rollouts, substantially higher than Agent‑Reflex.
Same metric as above, comparing the unmodified 8B backbone.
On the misleading‑lure tool‑side axis, Agent‑Reflex outperforms baselines by 13.9 points.
Figure 5 reports the margin after normalizing by three‑model means.
On the incomplete‑information tool‑side axis, the margin is 10.9 points.
Same analysis as above.
For hard‑failure template A, the margin is 12.3 points.
Figure 5 aggregates four hard‑failure templates; this is the lowest among them.
For hard‑failure template B, the margin is 13.5 points.
Figure 5 shows the highest hard‑failure margin.
On the cognitive‑overload user‑side axis, the margin is 12.9 points.
Figure 5 highlights this as the only user‑side axis beating the EnvScaler‑8B baseline.
We varied the perturbation‑fraction $\nu$ in ORPO between 0.1 and 0.4; performance remained stable with no consistent advantage, so we fix $\nu$ = 0.25 for all experiments.