SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation

Yibo Peng, Long Lian, David Wagner, Sizhe Chen

SecOPD uses token-level distillation from clean inputs to train LLMs that ignore malicious instructions in untrusted data.

How can we defend LLMs against adaptive prompt injections by replacing coarse sequence-level feedback with fine-grained token-level signals?

AI agents are highly vulnerable to prompt injections, where malicious instructions hidden in external data override the agent's original task. Existing defenses rely on sequence-level feedback, which fails to distinguish between benign and malicious tokens, leaving models susceptible to adaptive attacks. Secure On-Policy Distillation (SecOPD) solves this by providing token-level supervision: the model generates a response to an attacked input, while a frozen teacher scores that same response against a clean, injection-free version of the input. This fine-grained signal allows the model to suppress injection-following tokens while preserving task-relevant ones, reducing the attack success rate against state-of-the-art adaptive attacks by an order of magnitude compared to prior methods.

Paper Primer

SecOPD treats prompt-injection defense as a token-level credit assignment problem. By comparing the student's log-probabilities on an attacked input against a teacher's log-probabilities on the corresponding clean input, the method forces the model to align its output with the benign task regardless of the injection.

The core mechanism is a token-level distillation signal: the model is penalized when its output on an attacked input deviates from what the teacher would have produced on a clean input. This allows the model to "ignore" instructions in untrusted data while maintaining utility on the original user task.

SecOPD achieves an order-of-magnitude improvement in robustness against state-of-the-art adaptive prompt injections.

Against the PISmith adaptive attack, SecOPD achieves a 9.0% attack success rate (ASR) on the Qwen3.6-27B model, compared to 94.0% for the prior state-of-the-art, Meta-SecAlign.

Security gains generalize to unseen agentic tool-calling domains.

On the AgentDojo benchmark, SecOPD achieves a 4.7% ASR, outperforming Meta-SecAlign (5.5%) while maintaining utility comparable to the undefended model.

Why does token-level feedback outperform sequence-level methods like DPO or GRPO?

Sequence-level methods assign a single reward to the entire response, making it impossible to penalize only the malicious tokens while rewarding the benign ones. SecOPD provides a signal for every token, allowing the model to learn exactly which spans follow the injection and which follow the trusted task.

Does this defense require a complex security judge at test time?

No. The teacher model is only used during training to provide feedback based on clean inputs. At test time, the defended model is a standard, standalone LLM that requires no additional infrastructure or external judges.

The Prompt Injection Problem

LLMs fail against adaptive prompt injections because coarse sequence‑level feedback hides insecure tokens.

Prompt injection is identified as the number‑one threat to AI agents, and adaptive attacks now achieve almost 100 % Attack Success Rate (ASR). Existing defensive fine‑tuning recipes—such as DPO or GRPO—use sequence‑level feedback, assigning a single reward to an entire response. This coarse signal prevents the model from learning which individual tokens are insecure, leaving it vulnerable to adaptive prompt injections.

An attacker embeds a malicious instruction inside untrusted data so the LLM treats it as part of its prompt and obeys the attacker’s command.

When training provides a single reward for the whole output, the model cannot learn which tokens stem from the trusted prompt and which stem from the injection.

**Figure 1.** Motivation for using token-level feedback. When the LLM answers both the trusted prompt and the injection, Sequence-level feedback (a reward) treats the entire response as one unit, which obscures which tokens should be encouraged or discouraged. In contrast, token-level feedback rates each token, assigning high advantage to tokens consistent with the trusted prompt and low advantage to tokens that follow the injection. This finer-grained training signal enables more precise learning of the prompt-injection security policy.

Sequence‑level feedback cannot isolate malicious tokens, so it leaves LLMs exposed to adaptive prompt injections.

Existing Defensive Strategies

We situate SecOPD among prior system and model defenses and related learning paradigms.

Prompt‑injection defenses fall into two broad families: system‑level wrappers that sit around a fixed LLM, and model‑level modifications that change the LLM’s behavior.

Techniques that keep the underlying LLM unchanged and add external components—detectors, prompt shields, input filters, sanitizers, or action‑restriction layers—to block malicious instructions before they reach the model.

Approaches that augment the LLM itself, typically by introducing a special message type that marks untrusted data and fine‑tuning the model to ignore any instructions embedded in that segment.

Meta‑SecAlign builds a preference dataset that pairs a clean response with an insecure response to the same injected prompt, then uses Direct Preference Optimization (DPO) to push the model toward the clean side.

GRPO supplies token‑level supervision by scoring each generated token against the clean‑input reference, encouraging the model to stay on the safe trajectory token by token.

A training paradigm where a model learns to reproduce the behavior induced by an auxiliary system prompt, effectively distilling contextual cues into the model’s parameters.

Provides token‑level supervision on trajectories generated by a student model, using the teacher’s outputs as targets for each step.

Attacks that iteratively optimize a malicious prompt against the defended model’s current behavior, often using reinforcement learning to maximize Attack Success Rate (ASR).

Empirically, SecOPD outperforms the Meta‑SecAlign baseline on the PISmith adaptive attack, achieving a lower Attack Success Rate while preserving downstream utility.

Formalizing Defensive Fine-Tuning

We formalize prompt‑injection defenses and introduce token‑level OPD feedback as the core mechanism.

Sequence‑level feedback (e.g., DPO or GRPO) assigns a single preference or reward to an entire response, so it cannot pinpoint which token(s) betray the injected goal $g$.

OPD is like a teacher grading each word of a student's essay rather than the whole essay — the teacher supplies a token‑level signal that tells the student exactly where it deviated.

How does OPD differ from standard DPO, which also uses KL‑based objectives?

Standard DPO treats the KL term as a single scalar attached to the whole response, so it cannot tell which token caused a violation. OPD computes a separate KL divergence $D_t$ for every token $t$, giving a precise per‑token advantage that can suppress the injected command without harming the rest of the output.

Compute $\Delta D_1 = \log0.6 - \log0.9 \approx -0.405$.

Compute $\Delta D_2 = \log0.5 - \log0.8 \approx -0.470$.

Compute $\Delta D_3 = \log0.7 - \log0.1 \approx 1.945$.

Form token‑level advantages $A_t = -\,\operatorname{sg}[\Delta D_t]$, yielding $A_1 = 0.405$, $A_2 = 0.470$, $A_3 = -1.945$ (the negative sign flips the third token’s signal).

The large negative advantage on “delete” flags the injected command at token 3, while the positive advantages on the first two tokens reinforce benign generation.

The SecOPD Mechanism

SecOPD delivers token‑level feedback by comparing student rollouts on attacked inputs with teacher scores on clean inputs.

Sequence‑level feedback treats an entire response as a single scalar, which masks the fact that only the injected suffix is harmful. The remedy is to assign credit or blame to each token individually, letting the model keep the trusted portion while suppressing the malicious continuation.

For every token generated from an attacked prompt we ask a clean‑prompt teacher how likely that token should be; the signed difference becomes a per‑token training signal.

$\Delta$₁ = (‑0.4) − (‑0.2) = ‑0.2 → A₁ = −sg(‑0.2) = +1 (teacher prefers token t₁).

$\Delta$₂ = (‑0.8) − (‑1.5) = +0.7 → A₂ = −sg(+0.7) = ‑1 (student is already better; we penalize to match clean behavior).

$\Delta$₃ = (‑0.6) − (‑0.3) = ‑0.3 → A₃ = −sg(‑0.3) = +1 (teacher prefers token t₃).

Tokens that the clean teacher rates higher receive a positive boost, while tokens that the teacher rates lower are suppressed, enabling the model to keep trusted content and discard injected instructions.

How does SecOPD differ from standard on‑policy distillation (OPD) that uses sequence‑level returns?

Standard OPD treats the whole sampled trajectory as a single return, so a malicious suffix drags down the entire sequence. SecOPD replaces that scalar with per‑token advantages derived from a clean‑prompt teacher, allowing the model to reinforce good tokens and punish only the injected ones.

**Figure 2.** Overview of SecOPD training. The LLM first generates a rollout from an attacked input containing a trusted user prompt and untrusted external data. The same rollout tokens are then scored by the initialization LLM given the corresponding clean input, where the injected instruction is absent. These token-level scores provide the training signal: tokens aligned with the trusted task are encouraged, while tokens following the injected instruction are suppressed. The optimized LLM is the student, and the initialization LLM is kept fixed as the teacher.

Sample a clean example (I, c) from the dataset D and an injection goal g from the pool G.

Render the clean prompt $p_c$ ← R(I, c) and the attacked prompt $p_a$ ← R(I, A(c, g)).

Roll out the current student policy $\pi$_$\theta$ on $p_a$ to obtain a token sequence z.

For each position t, build the student context hᵃ_t = $p_a$ ‖ $z_{<t}$ and the teacher context hᶜ_t = $p_c$ ‖ $z_{<t}$.

Compute log‑probabilities ℓ_{$\theta$,t} and ℓ_{teacher,t}, form $\Delta_{t}$ and the stopped‑gradient advantage $A_t$ = −sg($\Delta_{t}$).

Apply a standard policy‑gradient update to $\pi$_$\theta$ using the sampled tokens, their log‑probabilities, and the advantages {$A_t$}.

By grounding feedback at the token level, SecOPD preserves the intended task output while actively suppressing any injected instruction, all without external supervision.

Experimental Setup

SecOPD is fine‑tuned on a 19 K instruction dataset while evaluating four models.

SecOPD is fine‑tuned on a 19 K instruction‑following dataset.

The dataset is assembled from Cleaned‑Alpaca following Chen et al. (2025c).

We evaluate the four Qwen3.6‑27B variants using Tinker for defense training and lm‑eval for benchmark execution. Meta‑SecAlign serves as the strongest prior model‑level defense, while GRPO provides a sequence‑level on‑policy baseline that mixes injected and benign samples with a binary reward and KL divergence regularization.

Security is measured by Attack Success Rate (ASR) on two suites: SEP (9.1 K instruction‑following samples) and AgentDojo (949 user‑task/injection pairs). For SEP we run static attacks and the adaptive PISmith attacker, reporting pass@10 ASR on 1 024 test examples after filtering candidates with witness matching and double‑checking each with six LLM calls.

Utility is assessed with AlpacaEval2 (both directly and via the SEP judge) and four non‑judge benchmarks: MMLU‑Pro, GPQA‑Diamond, GSM8K, and Minerva‑Math, covering reasoning, knowledge, and math capabilities.

Security and Utility Performance

SecOPD dramatically lowers adaptive attack success while keeping utility high.

Recall that SecOPD supplies token‑level feedback during defensive fine‑tuning, which lets the model suppress injected tokens while preserving the rest of the prompt.

SecOPD cuts the strongest adaptive attack success rate (PISmith) by 85 percentage points, from 94 % down to 9 %.

Table 1 shows PISmith ASR = 94.0 % for the undefended model, 94.0 % for Meta‑SecAlign, 61.2 % for GRPO, and 9.0 % for SecOPD.

**Table 1.** Security results measured by attack success rate (ASR, ↓). PISmith is the strongest adaptive attack in our evaluation; AgentDojo measures whether robustness transfers to tool-use tasks.

**Table 2.** Utility scores ($\uparrow$). SecOPD preserves most general capability while providing substantially stronger adaptive robustness than Meta-SecAlign and substantially better utility than GRPO.

**Table 5.** Manual audit of the 46 Minerva-Math examples answered correctly by the undefended model but incorrectly by SecOPD.

The table presents an evaluation of different attack methods, comparing automated evaluation (Eval.+) with human evaluation (Human+). It includes columns for the attack type, sample size ($n$), the number of positive evaluations for both automated and human metrics, false positives (FP), false negatives (FN), and the percentage of agreement between the two evaluation methods.

The case study (5.5) illustrates that SecOPD suppresses the injected “winter” token while still answering the trusted question, unlike Meta‑SecAlign and GRPO which obey the malicious instruction.

Benchmark Details: SEP

Details of the SEP benchmark and its static attack families.

This appendix formalizes the SEP benchmark used throughout the paper and enumerates the static and adaptive attack families evaluated.

The SEP benchmark tests whether instruction‑tuned LLMs can keep trusted instructions separate from injected malicious prompts.

Static SEP evaluation uses six fixed attack families—straightforward, straightforward‑before, ignore, ignore‑before, completion, and completion‑ignore—varying attack form and injection position.

The combined ASR metric counts an example as compromised if any of the six variants succeeds, providing an overall robustness score.

**Table 7.** Full SEP static attack breakdown over six non-adaptive attack families.

Beyond static attacks, we evaluate two adaptive settings. The basic adaptive scenario adapts completion‑style attacks to Qwen’s chat format by substituting delimiter‑like tokens drawn from nearby embedding vectors.

Benchmark Details: Utility and AgentDojo

Appendix B details utility benchmarks, evaluation protocols, and the AgentDojo benchmark results.

The appendix reports the suite of utility benchmarks used to assess model performance on instruction following and reasoning tasks.

AgentDojo measures both benign utility and robustness against adaptive prompt injections across several domains.

**Table 6.** Full per-domain AgentDojo evaluation. Utility measures benign user-task success without injection, AdvUtility measures user-task success under injection, and ASR measures attacker-task success under injection.

AlpacaEval2, SEP Utility, MMLU‑Pro, GPQA‑Diamond, GSM8K, and Minerva‑Math together cover instruction following, factual reasoning, scientific QA, and multi‑step mathematics.

SEP witness words are filtered with a two‑stage evaluation to avoid over‑counting false positives caused by substring matches or quoted injections.

The attack‑success judge runs three sampling calls per model at temperature 1.0, aggregates six total calls, and requires unanimous YES to label an attack successful.

The system prompt asks the judge to reason step‑by‑step about the injection goal, while the per‑sample prompt supplies the injection and model response.

A manual audit of 300 SEP outputs confirmed all 33 automated positives and found no false negatives, matching the automated evaluation (Table 3).

Training and Attack Details

Additional experimental details and auxiliary tables for the study.

This appendix supplies the auxiliary tables and low‑level experimental settings referenced in the main text.

**Table 4.** Example close-delimiter substitutes for the undefended Qwen3.6-27B model.

**Table 8.** Main SecOPD training configuration.

For each target defense we train a dedicated PISmith attacker, initializing from QWEN3‑4B‑INSTRUCT‑2507 and fine‑tuning on 100 Dolly Closed‑QA examples drawn from PIArena. Training uses TRL GRPO; during evaluation the attacker is frozen and generates ten candidate injections per SEP test example, with pass@10 ASR reported on 1 024 SEP instances.

The GRPO baseline fine‑tunes the same undefended Qwen3.6‑27B model using LoRA rank 128. A Gemini judge supplies a sequence‑level security reward: injected examples that achieve the attacker’s goal receive negative reward, while secure responses receive positive reward; benign examples receive zero reward and serve only for KL regularization toward the original model.

AgentDojo v1.2.1 is evaluated across four domains (Workspace, Banking, Travel, Slack) covering 949 user‑task–injection‑task pairs. The benchmark supplies a static “`important_instructions`” attack and the “`repeat_user_prompt`” defense setting. Utility, AdvUtility, and ASR are reported, where Utility measures clean‑task success, AdvUtility measures task success under injection, and ASR measures attacker‑task success.

Table 5 reports a manual audit of 46 Minerva‑Math examples that the undefended model answered correctly but SecOPD answered incorrectly; evaluations were run with chain‑of‑thought (“thinking”) enabled, using the input delimiter format for tool contexts, and with tensor‑parallel size 2.

The final GRPO and SecOPD evaluations use a maximum context length of 32 768 tokens, whereas the original undefended and Meta‑SecAlign evaluations used 16 384 tokens.

Conclusion

We summarize SecOPD’s impact, limits, ethics, and acknowledgments.

SecOPD delivers token‑level training signals that reduce the Attack Success Rate by roughly ten‑fold compared with the previous state‑of‑the‑art, directly refuting the claim that LLMs cannot be secured.

Our approach is scoped to indirect prompt injections—situations where the user’s prompt is benign but the surrounding environment data is malicious. It does not cover jailbreaks (Zou et al., 2023) or direct prompt injections (Mu et al., 2025).

We also assume a clear signal that marks which input fragments are trusted; models that must infer trust from context remain vulnerable, and SecOPD’s reasoning traces deliberately omit any mention of an injection, which could affect downstream detection methods.

The strongest adaptive attack (PISmith) still reaches 9.0 % ASR, so future, more sophisticated attacks may erode the gains we report.

Ethically, the attacks used in our experiments are confined to controlled benchmarks and we provide no instructions for real‑world exploitation. While improving robustness reduces risk, no single defense is sufficient; practitioners should combine input filtering, action constraints, monitoring, and least‑privilege tool access.

We thank the KACST‑UCB Joint Center on Cybersecurity, the Noyce Foundation, and the Tinker Research Grant for support, and acknowledge Zhanhao Hu, Xiwen Min, Ding Zhong, Uriah (Yu‑Lin) Tsai, Muxi Lyu, and Yiwei Hou for valuable discussions and feedback.

Author Contributions and AI Usage

Additional experiments detail AI‑assistant usage, error analysis, and extended evaluation tables.

We employed AI assistants to aid coding, debugging, and language editing, but every scientific claim, experimental result, and final manuscript sentence was manually reviewed and verified by the authors.

In the Minerva‑Math utility error analysis we examined all 46 examples that the undefended model answered correctly but SecOPD answered incorrectly, finding only four genuine reasoning mistakes.

The remaining 42 cases stem from final‑answer organization, extraction mismatches, or premature termination of a derivation rather than systematic reasoning degradation.

Representative failures include a task‑switching error where the model starts an integer‑equation problem but switches to a square‑root problem, an extraction error where a correct probability 1/16 is marked wrong by the verifier, and an incomplete geometry derivation that halts early.

Table 6 reports full per‑domain AgentDojo results: Utility, AdvUtility, and ASR for the undefended model, Meta‑SecAlign, GRPO, and SecOPD (ours), showing SecOPD’s strong utility under injection while keeping attacker success low.

Table 7 presents the SEP static attack breakdown across six non‑adaptive attack families, where SecOPD reduces attack success to near‑zero for most families compared with the undefended baseline.

Table 8 lists the main SecOPD training configuration: a frozen QWEN3.6‑27B initialization model, 128 trainable LoRA parameters (rank 1 × 10⁻⁴), learning rate 1.0, sampling temperature 16K, and Alpaca‑style maximum generation length.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers