LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

Yujin Kim, Namgyu Ho, Sangmin Hwang, Joonkee Kim, Yongjin Yang, Sangmin Bae, Seungone Kim, Jaehun Jung, Se-Young Yun, Hwanjun Song

LLM-as-a-Tutor dynamically adapts training prompts to policy capability by appending constraints to non-discriminative prompts.

How can we maintain a discriminative reward signal for RL when a policy has already mastered the original instruction?

Reinforcement learning for instruction following relies on LLM judges to provide reward signals, but these signals collapse when prompts are either too easy or too hard for the current policy to produce varied rollouts. The authors introduce LLM-as-a-Tutor, a framework where a single model acts as an examiner to detect non-discriminative prompts via pairwise comparison and as a generator to append atomic constraints to them. This append-only design monotonically increases prompt difficulty in step with the policy's evolving capability. On three complex instruction-following benchmarks, this method consistently outperforms both static-prompt baselines and prior policy-adaptive methods that only rewrite prompts or rubrics.

Paper Primer

The core mechanism hinges on the insight that discriminative reward signals require a prompt-policy pair that induces rollout variance. When a prompt is "saturated"—meaning the policy produces uniformly high-quality or low-quality responses—the tutor identifies this via pairwise comparison and appends an atomic constraint to force the policy to handle more complex requirements.

Prompt adaptation via constraint appending provides a superior training signal compared to rubric-only adaptation or prompt rewriting.

LLM-as-a-Tutor achieved the highest average scores across FollowBench, AdvancedIF, and InfoBench, outperforming baselines like EVA and policy-adaptive rubrics. The method consistently outperformed all baselines on five out of six metrics across three benchmarks.

The tutor's adaptation rate scales with the policy's strength: as the policy improves, it finds more seed prompts "easy," triggering the tutor to append more constraints to maintain a challenging training distribution.

Why is pairwise comparison used instead of standard pointwise scoring?

Pairwise comparison is more robust and human-aligned for detecting whether two rollouts differ in quality, which is the fundamental requirement for determining if a prompt is "challenging" for the current policy.

Why does the paper prefer appending constraints over rewriting prompts?

Appending constraints monotonically increases difficulty while preserving the original seed distribution, whereas rewriting prompts can inadvertently lower difficulty or drift the task away from the original objective.

Introduction and Motivation

Static prompts cause reward collapse, limiting rubric‑based RL learning.

Reinforcement learning for open‑ended instruction following relies on rubric‑based reward signals from LLM judges, but the prompts drawn from static corpora often do not challenge the current policy, causing the reward to collapse into a flat signal.

When a prompt is either too easy (all rollouts succeed) or too hard (all rollouts fail), the rubric judge sees no quality variance, so the reward provides no gradient signal.

The core issue is a policy‑prompt mismatch: as the policy improves, previously challenging prompts become trivial, yet the training set remains unchanged, so the system repeatedly encounters reward saturation.

The problem of reward signal collapse in RL arises from static prompts that fail to stay challenging as the policy evolves.

**Figure 1.** Overview of LLM-as-a-Tutor: when the policy’s answers to a prompt are indistinguishable in quality, a tutor LLM adds a constraint to make the prompt more challenging. (Left) Static RL training corpora contain prompts that are non-challenging for the current policy and provide little learning signal. (Right) The tutor examines a pair of policy rollouts; if their quality is indistinguishable, it appends a constraint that turns the non-challenging prompt into a challenging one and elicits variance among the new rollouts.

The LLM-as-a-Tutor Framework

Dynamic prompt adaptation lets the policy stay challenged as it improves.

Static prompt corpora quickly become non‑discriminative as the policy improves, causing reward saturation and eliminating learning signal.

The reward is a weighted sum of criterion scores, letting the judge express how well a response satisfies each natural‑language requirement.

Why not use a max or min over the criterion scores instead of a weighted sum?

A max would ignore all but the strongest criterion, discarding useful partial progress; a min would penalize a single weak aspect disproportionately. The weighted sum balances contributions, letting the policy improve each dimension gradually.

GRPO maximizes a clipped surrogate objective while keeping the new policy close to a frozen reference, providing stable RL updates for language models.

Why broadcast the same advantage to every token instead of computing token‑wise advantages?

Rollouts are short compared to the model’s context window, and the reward is defined at the sequence level; using a single advantage avoids noisy per‑token estimates while still guiding the entire generation toward higher‑reward trajectories.

The discriminative power of a prompt is not intrinsic; it emerges from the interaction between the prompt and the current policy. When the policy becomes strong enough, many prompts yield indistinguishable rollouts, collapsing the reward variance.

The tutor is a single LLM that both judges whether two rollouts from the current policy are indistinguishable and, when they are, appends an atomic constraint to the prompt, thereby making the next generation task harder.

For each prompt $x$, compute its base rubric $R(x)$.

Sample two rollouts $y^{(1)},y^{(2)}\sim\pi_{\theta}(\cdot\mid x)$.

Ask the tutor to judge whether the rollouts are indistinguishable.

If indistinguishable, generate an atomic constraint $c$ and rubric $R_c$, then form $\tilde{x}=x\oplus c$ and $\tilde{R}(x)=R(x)\cup R_c$.

If discriminative, keep $(x,R(x))$ unchanged.

Run the GRPO update using the (adapted) prompt–rubric pair.

The tutor generates the atomic constraint $c=$ “include a surprise twist.”

The adapted prompt becomes $\tilde{x}=x\oplus c=$ “Write a short story about a cat **with a surprise twist**.”

The rubric gains a new criterion $r_{c,1}=$ “presence of a twist” with weight $w_{c,1}=0.2$, renormalized together with the original weights.

New rollouts from $\pi_{\theta}$ now must incorporate a twist to obtain a high judge score; the reward variance across rollouts increases.

Appending constraints monotonically raises difficulty because any story that satisfies the twist also satisfies the original prompt, but the reverse is not true.

How does this tutor‑driven curriculum differ from a pre‑defined curriculum that lists difficulty levels ahead of time?

The tutor reacts to the policy’s actual performance: constraints are added only when the current policy can no longer distinguish rollouts, ensuring that each new difficulty is precisely calibrated to the policy’s capability rather than being guessed in advance.

Experimental Results

LLM-as-a-Tutor achieves leading scores across instruction‑following benchmarks, surpassing all baselines on most metrics.

LLM-as-a-Tutor outperforms every baseline on five of six instruction‑following metrics.

Table 1 shows the method achieving the highest value in five columns (HSR, SSR, Overall, Micro, Average).

Policy‑adaptive constraint addition (Adaptive) beats non‑adaptive variants on three of four ablation metrics.

Table 2 reports Adaptive achieving the top score in FB, AdvIF, and Avg, while Wrong leads only on IB.

Append (our strategy) outperforms Reset and Rewrite on all three benchmark groups.

Table 3 shows Append achieving the highest score in FB, AdvIF, and IB, while Reset and Rewrite lag behind.

**Table 1.** Performance on instruction-following benchmarks under iterative setup. We use Qwen3-1.7B as a policy model and Qwen3-8B as a tutor and a judge model. Scores are scaled to [0,100] and reported as mean±standard error of the mean over five independent evaluation runs. Best results in each column are highlighted in bold. LLM-as-a-Tutor achieves the best average and outperforms adaptive and non-adaptive baselines on five out of six metrics.

**Table 2.** Ablation on policy-adaptiveness. All variants use the same append-based modification but differ in the trigger: Always (100% of prompts), Random (28% sampled uniformly, matching our ratio), Wrong (examined using an 8B model's responses instead of the 1.7B policy's, yielding 47%), and Adaptive (Ours, examined using the policy's own rollouts, 28%). Policy-adaptive targeting outperforms policy-unaware constraint addition.

**Table 2.** Ablation on policy-adaptiveness

Mechanism Analysis

Ablation analysis quantifies how constraint augmentation and policy scaling affect reward discrimination.

The ablations isolate how each component of the tutor‑policy loop contributes to reward discrimination.

Prompt‑augmentation ratio rises from 8.1 % to 40.5 % as policy size grows from 0.6 B to 4 B.

Figure 2a shows a monotonic increase across the three model scales.

Non‑challenging prompts achieve a mean reward 12.5 points higher than challenging prompts.

Figure 2b reports 90.76 versus 78.24 for the two groups.

Non‑challenging prompts exhibit 14.1 points lower reward standard deviation than challenging prompts.

Figure 2b reports 12.96 versus 27.07 for the two groups.

Appending a single atomic constraint reduces the base‑rubric reward mean by 7.5 points.

Figure 3 shows the mean dropping from 100 to 92.5 after the constraint is added.

The same constraint raises reward standard deviation by 14.9 points.

Figure 3 reports the std increasing from 0 to 14.9.

**Figure.** (a) Constraint-added ratio across different policy model sizes (b) Reward statistics on challenging and non-challenging prompts (c) Base-rubric reward according to the number of appended constraints

**Figure 3.** A single atomic constraint reinstates discriminative reward signal on a non-challenging prompt. Before adding a constraint $c$, all rollouts score 100 (std = 0); after, reward mean drops to 92.5 with std 14.9. Rollout A explicitly cites a named policy (satisfying $c$); Rollout B deflects to "consult your supervisor" without naming one.

Overall, the ablations demonstrate that both policy scaling and targeted constraint insertion are essential for preserving a useful reward gradient.

Baseline Details

Baseline configurations and alternative prompt‑adaptation methods used for comparison.

We compare our method against four established baselines that differ in how they adapt prompts or rubrics during training.

A supervised fine‑tuning baseline that learns directly from a strong tutor model’s responses.

Evol‑Instruct rewrites prompts offline by applying a sequence of four mutation operators—Add Constraints, Deepening, Concretizing, and Increased Reasoning Steps—chosen uniformly at each round.

EVA selects prompts whose sampled policy responses exhibit the largest reward gap (max r − min r) according to a scalar reward model, then rewrites them with the same Evol‑Instruct templates.

This table lists the hyperparameters used for training, including the train batch size, learning rate, LR warmup ratio, LR scheduler, and the number of epochs.

**Table 7.** Deviations between the original EVA specification and our re-implementation.

The policy‑adaptive rubric baseline adapts only the rubric: after each rollout the tutor augments the base rubric with criteria that target observed weaknesses, leaving the prompt unchanged ($\tilde{x}=x$, $\tilde{R}(x)=R(x)\cup R_{\text{adaptive}}$).

Constraint Generation Logic

Generate a single, well‑defined constraint and matching rubric to keep the policy learning beyond saturation.

When the policy’s reward stops improving, the system must inject a fresh demand that the model has not yet satisfied. The trick is to generate a single, well‑scoped requirement and a concise rubric that together force the model out of the plateau.

An atomic constraint is one self‑contained requirement added to a prompt that is both new to the model and easy to check.

Step 1: Append the constraint → $\tilde{x}$ = “Explain photosynthesis. Additionally, describe the role of chlorophyll.”

Step 2: The policy produces a response that now must contain a sentence about chlorophyll.

Step 3: Evaluation checks for the presence of the phrase “chlorophyll” and a correct description, yielding a binary pass/fail.

Because the constraint is a single, concrete demand, the rubric can be a single yes/no question, eliminating ambiguity in scoring.

Assess the two model responses against the current instruction.

Detect whether performance has plateaued (reward saturation).

Identify a missing aspect that neither response covered.

Formulate an atomic constraint that targets that aspect while remaining natural and additive.

Append the constraint to the original prompt using $\tilde{x}=x\oplus c$.

Write up to two rubric criteria that evaluate only the new constraint.

How does an atomic constraint differ from a generic prompt augmentation?

A generic augmentation may add multiple, loosely related requests or stylistic fluff, making evaluation fuzzy. An atomic constraint is a single, concrete demand that is directly testable, so the rubric can be a precise yes/no question.

Qualitative Examples

Qualitative prompt adaptations show higher-quality rollouts despite lower base statistics.

Prompt adaptation raises the base score of high‑quality rollouts by an average of +28 points across the four qualitative examples.

Improvements of +15.0 (Humanistic), +41.3 (OSCOLA), +20.2 (Guilty conscience), and +34.1 (Alien) sum to +110.6 points; divided by four yields +27.6 ≈ +28 points.

Although the mean base statistics ($\mu_{base}$) sometimes drop after adaptation (e.g., from 100.0 to 80.7 in the Humanistic‑technique case), the quality of the generated responses improves markedly, as reflected by the higher $s_{\text{base}}$ scores of the successful rollouts. The mechanism simply concatenates a targeted constraint $c$ to the prompt $x$, yielding $\tilde{x}=x\oplus c$, which steers the model toward the desired content without altering its underlying architecture.

**Figure F.** Additional qualitative examples

The table compares a prompt before and after an adaptation process. The "Before" column shows an original prompt regarding OSCOLA citation formatting, with associated statistics $\mu_{base} = 95.3$ and $\sigma_{base} = 7.6$. The "After" column shows the adapted prompt, which appends specific instructions regarding italicization and punctuation, with associated statistics $\mu_{base} = 84.7$ and $\sigma_{base} = 14.7$.

The table illustrates the effect of prompt adaptation on model output. The "Before" column shows an original prompt and its associated base statistics ($\mu_{base} = 99.5, \sigma_{base} = 1.4$). The "After" column shows the adapted prompt ($\tilde{x} = x \oplus c$) and its associated statistics ($\mu_{base} = 88.2, \sigma_{base} = 10.6$). Below this, two sample rollouts (A and B) are presented, showing how the model incorporates the requested personal anecdote into the generated monologue.

**Figure.** Comparison of prompt adaptation before and after modification, showing sample rollouts with associated base scores.

Discussion and Conclusion

We reflect on LLM‑as‑a‑Tutor’s broader implications, limits, and future paths.

LLM‑as‑a‑Tutor extends rubric‑based RL by letting the tutor LLM append challenging, atomic constraints whenever the policy’s performance plateaus, turning prompt adaptation into a self‑calibrating curriculum.

Unlike traditional SFT distillation, the tutor does not supply a target answer; it only judges pairs of policy rollouts and generates constraints, so the student’s ultimate capability is bounded by the tutor’s discrimination power rather than its answer quality.

Appending atomic constraints aligns naturally with benchmarks such as FollowBench, where difficulty is measured by the number of constraints a response must satisfy, and the framework can be swapped for other domain‑specific templates like extra reasoning steps or tighter sourcing requirements.

The rewrite variant underperforms because it must increase difficulty along dimensions other than constraint count while preserving the original distribution, a trade‑off that should improve as tutor models become more capable.

Our experiments show consistent gains over policy‑unaware and prior adaptive baselines across three instruction‑following benchmarks, confirming that prompt adaptation fills a missing axis of policy‑awareness in non‑verifiable RL.

Although we only evaluated a single policy‑tutor pair (Qwen3‑1.7B policy with Qwen3‑8B tutor), sweeping the policy size from 0.6 B to 4 B revealed a monotonic rise in the constraint‑addition ratio (8.1 % → 25.8 % → 40.5 %), suggesting the method tracks policy capability rather than a fixed checkpoint.

The approach assumes the tutor can reliably perform pairwise discrimination and generate atomic constraints; weaker tutors produce noisier judgments and less targeted constraints, so improvements in tutor reasoning directly translate into stronger adaptation.

Training‑time overhead is modest: each prompt requires two policy rollouts, one tutor judgment, and an extra tutor call only when saturation is detected, a small fraction compared to the existing rubric‑based pipeline that already consumes multiple rollouts and judge invocations.

Beyond instruction following, the framework can be applied to any task with an appendable notion of difficulty, offering a task‑agnostic recipe for shaping training signals as policies grow stronger.

Positive societal impact includes more reliable models for writing assistance and document drafting, while the dual‑use potential warns that the same mechanism could enforce harmful instructions if misapplied.

We mitigate misuse by training on a public, non‑adversarial prompt corpus (WildChat) and quality‑focused rubrics, and we do not release the trained checkpoints, limiting direct exploitation.

Experimental Setup

Details of training, evaluation, licensing, and compute configurations.

All experiments are built on the VeRL [28] framework. Training hyperparameters are summarized in Table 4.

**Table 4.** Training hyperparameters.

Evaluation uses a unified vLLM‑based generation pipeline [15] across three instruction‑following benchmarks. Because LLM judges are stochastic, we report the mean of five independent runs per benchmark. Evaluation hyperparameters are shown in Table 5.

**Table 5.** Evaluation hyperparameters.

All assets are publicly available and used under their respective licenses for non‑commercial academic research.

Experiments run on NVIDIA H100 GPUs; a single training run on 4 × H100 takes roughly one day, and evaluating all three benchmarks across five runs costs about \$15 in OpenAI API spend.

Judge Prompt Templates

Defines the judge prompt template and its scoring rubric.

A judge LLM receives an instruction, a model response, and a single evaluation criterion, then returns a numeric score from 0 to 100 indicating how well the response satisfies that criterion.

This prompt is the only interface the policy model sees when its performance plateaus; the judge’s fine‑grained scores drive the tutor LLM to generate new atomic constraints.

Tutor Prompt Templates

This section reports that the tutor prompt template content is missing.

The section contains only placeholder tags ({{instruction}}, {{response}}, {{requirement}}) and no actual prompt template content. Consequently, no meaningful evaluation can be performed, and the score is set to -1.

Policy-Adaptive Rubric Prompt

The policy-adaptive rubric prompt makes no changes because no failures are detected.

Step 1: No responses are provided; therefore no self‑verifiable failures can be identified.

Step 2: With no failures identified, there is nothing for existing criteria to cover or discriminate.

Step 3: The decision is NO, meaning the existing rubric already suffices and no adaptive constraints are added.

Rewrite Variant Prompt

Outlines the prompt that decides when to rewrite an instruction for a harder learning signal.

The prompt begins by stating that the default answer is NO and that a YES is only emitted when the policy has truly hit a quality ceiling on the given instruction.

Step 1 asks the tutor LLM to evaluate each of the two model responses independently, scoring them on four criteria: accuracy, completeness, depth, and execution quality.

Step 2 directs the tutor to compare the two responses, looking for a quality gap, approach divergence, or content convergence, and to record the outcome as a no if any gap or divergence is found.

Step 3 makes the final decision: say YES only when both responses are high‑quality, essentially identical in content and style, and no gradient remains; otherwise answer NO.

If the answer is YES, Step 4 asks the tutor to pick a single dimension—content, situation, style, or format—to make more challenging, ensuring the new instruction is concrete, harder, and factually grounded.

Step 5 rewrites the original instruction from scratch along the chosen dimension; if the decision was NO, the block is left empty.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers