DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training

Shubham Gandhi, Saurabh Goyal, Kiran Kate, Yara Rizk

DRACO improves long-horizon agent training by dynamically generating rubrics and redistributing rewards to specific steps without ground-truth verifiers.

How can we train long-horizon agents in outcome-blind settings by dynamically generating rubrics and attributing credit to specific reasoning steps?

Long-horizon agents often fail because they lack programmatic verifiers to check success, and assigning a single scalar reward to an entire trajectory wastes information by failing to distinguish between correct and incorrect steps. DRACO solves this by generating dynamic, per-trajectory rubrics that adapt to the agent's specific rollout, then redistributing the trajectory-level reward to individual steps using a closed-form credit assignment rule. This approach achieves significant gains on benchmarks like AppWorld, outperforming verifier-trained baselines while relying solely on process-based rubric signals.

Paper Primer

The core mechanism hinges on two components: a frozen judge that proposes and scores criteria based on the specific trajectory, and a credit rule that reallocates the trajectory's total advantage to individual steps based on their rubric-derived quality. The credit rule is a closed-form redistribution: it concentrates the reward on steps cited by the judge as "good" (for winners) or "bad" (for losers) while conserving the total trajectory-level push.

DRACO achieves superior performance on AppWorld without access to ground-truth verifiers.

On AppWorldTN, DRACO improves Task Goal Completion (TGC) by 15.9 points over the base model and outperforms GRPO trained with sparse ground-truth rewards by 5.3 points.

The method is robust to judge choice, allowing for cost-effective self-evaluation.

Replacing the frontier judge with a self-judge variant reduces evaluation costs by 5.1x while maintaining high performance, surpassing the outcome-aware reference on AppWorldTN.

Why is this approach necessary compared to standard reinforcement learning from verifiable rewards (RLVR)?

RLVR assumes a programmatic verifier exists, which is often unavailable for complex, open-ended agent tasks. DRACO operates in an "outcome-blind" regime, using process-based rubrics to provide a learning signal where no ground-truth oracle exists.

Does the credit redistribution change the overall trajectory reward?

No. The redistribution is mathematically designed to conserve the total trajectory-level push (total-push conservation), ensuring that credit is only moved to the steps responsible for the rubric verdicts rather than inflating or deflating the overall influence.

The Challenge of Outcome-Blind Training

Why training long‑horizon agents without a verifier is fundamentally hard.

Many long‑horizon agent tasks lack a programmatic verifier that can tell whether a rollout succeeded. In such outcome‑blind settings the only feedback comes from multi‑criteria rubrics that score an entire trajectory, leaving a coarse signal for dozens of steps. This mismatch makes credit assignment difficult: a single scalar reward is both noisy and uninformative for the individual decisions that produced it.

Training signals are derived solely from process‑based rubrics, with no external ground‑truth outcome available at any point during learning.

DRACO tackles this gap by (1) generating dynamic rubrics that adapt to each rollout’s actual behavior and (2) redistributing the rubric verdict over the responsible steps, yielding per‑step advantages that can be used within GRPO.

The core challenge is bridging the divide between verifiable rewards and long‑horizon tasks that lack such signals.

Prior Approaches to Agent Training

We position prior long‑horizon methods against DRACO across six key criteria.

GRPO treats every step in a trajectory as equally important, aggregating rewards uniformly across the whole episode.

**Table 1.** Comparison with the closest prior work. $\checkmark$ yes, $\times$ no, $\text{\textcircled{d}}$ partial, – N/A. *Outcome-Blind*: the training reward consults no ground-truth outcome ($\text{\textcircled{d}}$: judge plus verifier). *Rubric Reward*: reward derived from written criteria ($\text{\textcircled{d}}$: a criteria-guided judge emitting one verdict rather than per-criterion scores). *Dynamic Rubrics*: criteria that change during training, driven by the training algorithm rather than generated per instance by a fixed procedure. *Traj.-Level Scoring*: the rubric is scored once on the finished rollout, not at each position. *Step Attribution*: differentiated credit reaches the policy update at step granularity. *No Learned Attribution*: attribution requires no trained module.

The DRACO Framework

DRACO replaces a single trajectory‑wide reward with rubric‑driven step credit.

Without a verifier, long‑horizon agents receive only process‑based feedback, which makes it hard to tell which reasoning steps deserve credit.

Instead of a single static checklist, the judge builds a fresh rubric for each task and each sampled rollout, then scores the rollout against that rubric.

How does a dynamic rubric differ from a static, hand‑crafted checklist?

Static checklists are fixed before any rollout and cannot capture task‑specific sub‑goals or unexpected failure modes. Dynamic rubrics are generated per‑task and per‑rollout, so they adapt to the actual behavior of the policy and only keep criteria that a rollout actually violates.

After the judge produces a trajectory‑level reward $R_i$, DRACO redistributes the resulting advantage $A_i$ onto the individual reasoning steps that the rubric cites.

Compute $p_j$ and $f_j$: step 1 → $p_1=1$, $f_1=1$; step 2 → $p_2=1$, $f_2=1$; step 3 → $p_3=1$, $f_3=0$.

Quality scores: $Q_1 = 0.5$, $Q_2 = 0.5$, $Q_3 = 1.0$.

Suppose trajectory 1 received $A_1 = +0.8$ (winner) and trajectory 2 received $A_2 = -0.4$ (loser).

Weights: for $A_1\ge0$, $\text{weight}_1=0.5$, $\text{weight}_2=0.5$, $\text{weight}_3=1.0$; for $A_2<0$, $\text{weight}_1=1-0.5=0.5$, $\text{weight}_2=1-0.5=0.5$, $\text{weight}_3=1-1.0=0.0$.

Normalize weights for trajectory 1: $\sum_k \text{weight}_k = 2.0$, so $a_1 = 0.8 \cdot \frac{12 \times 0.5}{2.0} \cdot \frac{1}{2}=1.2$, $a_2 = 1.2$, $a_3 = 0.8 \cdot \frac{12 \times 1.0}{2.0} \cdot \frac{1}{2}=2.4$.

Normalize weights for trajectory 2: $\sum_k \text{weight}_k = 1.0$, so $a_1 = -0.4 \cdot \frac{12 \times 0.5}{1.0} \cdot \frac{1}{2}= -1.2$, $a_2 = -1.2$, $a_3 = 0$ (weight zero).

Credit moves from long, noisy steps to the few steps the rubric actually judges, while preserving the overall sign of the trajectory’s verdict.

Why does the weight formula use $Q_j$ for winners but $1\!-\!Q_j$ for losers?

When the trajectory is a winner ($A_i\ge0$) we want to amplify steps that passed most criteria (high $Q_j$). When it is a loser ($A_i<0$) we instead want to suppress the steps that failed most criteria, which is achieved by giving them larger weight $1\!-\!Q_j$.

DRACO stitches together dynamic rubrics and credit reallocation so that a long‑horizon agent receives fine‑grained, outcome‑blind feedback without any external verifier.

Sample a group of $G$ trajectories for the current task.

Generate a dynamic rubric: propose criteria from the instruction, extend them with rollout‑specific sub‑goals, merge, deduplicate, and drop any criterion that no trajectory fails.

Score each trajectory against the surviving criteria, obtaining passes $p_i$, fails $f_i$, and step citations.

Compute the outcome‑blind reward $R_i$ (Eq. 3) and standardize it to $A_i$ (Eq. 1).

For every step $j$, compute quality $Q_j$ (Eq. 4), weight $\text{weight}_j$ (Eq. 5), and step advantage $a_j$ (Eq. 6).

Update the policy model with the GRPO gradient (Eq. 2) using $a_j$ as the per‑token advantage.

**Figure 1.** Overview of DRACO. Top: for each task, the judge proposes criteria from the instruction and each sampled trajectory; proposals are merged, deduplicated, and scored to produce an outcome-blind reward $R_i$ (Section 3.2). Bottom: within a rollout group, GRPO normalizes rewards into trajectory advantages $A_i$; DRACO then reallocates each $A_i$ across steps according to the judge's per-criterion verdicts, yielding step advantages $a_j$ that concentrate credit on the steps the rubric implicates (Section 3.3).

Empirical Results

DRACO’s full method delivers the strongest success gains across all benchmarks.

DRACO achieves a 15.9 % absolute gain in Task Goal Completion on AppWorldTN over the untrained Qwen3.6‑27B base policy.

Base TGC 69.4 % vs DRACO 85.3 % (Table 2).

**Figure 2.** Performance vs. evaluation cost, one panel per benchmark; up and to the left is better. AppWorld panels plot TGC, $\tau$-bench banking plots success rate; SGC follows the same ordering (Table 2). Cost is the US-dollar total for one pass over the full split at the rates of Section 4.1.3; the $\tau$-bench axis also includes the GPT-5.4 user simulator and judge. Points are Qwen3.6-27B $p^1$ values from Table 2.

**Figure 3.** Rollout termination modes across training steps for DRACO with and without step credit. Bars show the percentage of rollouts per termination mode (normal completion/give-up vs. length, turn, or server errors), independent of task correctness. We interpret broad trends rather than exact error distributions.

**Figure 4.** Overall rubric pass rate across training steps. Each setting is evaluated on its training rubrics (dynamic rubrics vary per step). The dotted line shows DRACO re-scored on the static held-out rubric set. Figure 10 in Appendix A reports per-criterion pass rates.

DRACO consistently outperforms baselines across benchmarks.

Summary and Future Directions

We close with DRACO’s empirical gains and outline next steps.

DRACO consistently shortens episodes on the Qwen3.6‑27B model—turns drop from 18.7 to 14.7 on AppWorldTN and from 22.9 to 20.7 on AppWorldTC—while improving Task‑Goal Completion by 15.9 points on AppWorldTN and matching a verifier‑trained baseline under the same budget.

On the held‑out $\\tau$‑bench, episode turns increase modestly and the performance gains cost more, reflecting that the untrained policy model ends earlier rather than solving tasks faster. Both dynamic rubric settings converge to the same behavior: agents almost always submit an answer before the episode runs out, with failures only due to response‑length or turn‑budget limits. The step‑credit assignment rule, which reallocates reward based on per‑criterion judge attributions while preserving total magnitude, proves essential—ablation studies confirm its importance, and future work should explore how judge quality impacts overall success.

Limitations and Caveats

DRACO learns long‑horizon tasks by auto‑generating rubrics and reallocating credit to reasoning steps.

Rubric‑based rewards provide no external signal to confirm that the criteria faithfully capture the intended outcome. Validating them would require human raters, which the paper defers to future work, and the experiments rely on a single frozen judge without reporting chance‑corrected agreement with humans.

The attribution step also remains opaque: credit can be reallocated to the wrong reasoning steps yet still improve policy performance, and discriminative dropout causes the set of active criteria to vary with each sampled group, a variance the authors do not quantify.

Both benchmarks (AppWorld and $\tau$-bench) are used under their released licenses and consist entirely of synthetic data, so no real users or live services are involved. Because the reward consists of natural‑language criteria, the target is transparent and auditable, but any bias in the judge propagates to the policy, and the same tool‑competent capability can be misused, though the paper’s contribution is limited to the training signal and all evaluation is sandboxed.

Additional Results

Key DRACO gains: +29.5 SGC and \$2.5 cost reduction after 100 outcome‑blind steps.

DRACO raises SGC by 29.5 points after 100 outcome‑blind training steps.

SGC improves from 41.1 to 70.6 on the Qwen3.6‑27B model.

**Table 4.** Task success and consistency with variability, the counterpart of Table 2. Means are identical to Table 2, and best values per column are in bold as there. For trained rows, subscripts give the standard deviation over the three checkpoints each mean averages (90/95/100 steps for Qwen3.6-27B, 65/70/75 for Qwen2.5-32B-Instruct), with three evaluation runs per checkpoint. Base rows are single-checkpoint, so their subscripts instead give the standard deviation over three independent evaluation runs; SALT reports that same quantity, and its published values are quoted. Only $p^1$ admits an across-run spread, since $pass^k$ for $k \ge 2$ is a joint statistic over all three runs. We report $p^1$ and $p^3$ here; $p^2$ is in Table 2. Average columns are row-level summaries and are reproduced without subscripts. - = not evaluated on that split, or not reported by the cited work.

**Figure 5.** **pass**$^k$ vs. **pass**@$k$ over $n=3$ evaluation runs, in %. Solid: **pass**$^k$, all $k$ runs succeed. Dashed: **pass**@$k$, at least one succeeds. The two coincide at $k=1$ by construction. Values and protocol follow Table 2. All panels are Qwen3.6-27B; axes do not start at zero.

**Figure 6.** Frontier models vs. DRACO on `AppWorld_TN`. TGC (left) and SGC (right) against the cost of one pass over the 168 tasks; up and to the left is better. Circles: frontier models, one run each. Star: DRACO on Qwen3.6-27B with the frontier judge. Hollow circle: the same policy untrained. Brackets give parameter counts (total, active) for open-weight models; closed models publish none. Values are in Table 8.

**Figure 7.** Train TGC vs. training step This is TGC on the training tasks, for post-hoc analysis, and it is not part of any reward.

**Table 5.** Average agent turns with variability, the counterpart of Table 3; lower is better. Means are identical to Table 3, and best values per column are in bold as there. For trained rows, subscripts give the standard deviation over the three checkpoints each mean averages, with three evaluation runs per checkpoint. Base rows are single-checkpoint, so their subscripts instead give the standard deviation over three independent evaluation runs. Avg. is a row-level summary across benchmarks and is reproduced without a subscript. — = not evaluated on that split.

**Table 6.** **Task-success by difficulty** on both AppWorld test splits, in %, by the AppWorld metadata label and scored against each stratum's own denominator (counts in the header). Entries are the $p^1$ level of Table 2 and follow its conventions throughout. For trained rows, subscripts give the standard deviation over the three checkpoints each mean averages, with three evaluation runs per checkpoint. Base rows are single-checkpoint, so their subscripts instead give the standard deviation over three independent evaluation runs. $-$ = not evaluated on that split.

Additional Analysis

Ablation experiments reveal how each DRACO component contributes to performance.

This section isolates the effect of each DRACO component by removing it and measuring the resulting performance drop.

Replacing GPT‑5.4 with the policy model (self‑judge) matches GPT‑5.4 on 89.4 % of verdicts, showing only a modest degradation.

Measured over 60,689 verdicts across 100 training steps using replay of the original judge calls.

When the self‑model applies a given rubric, it agrees with GPT‑5.4 on 89.4 % of cases, far above the 72.0 % baseline of a trivial “always‑pass” judge.

Annotation results over 9,590 rollouts (60,689 verdicts) demonstrate this agreement.

Self‑model criterion generation attains a recall of 0.21 and precision of 0.39, indicating it captures only a fraction of GPT‑5.4’s criteria.

Recall and precision are averaged over 11 training steps, with recall ranging 0.16–0.28 and precision declining from 0.44 to 0.33 (p = 0.006).

Precision for self‑model generation is 0.39, consistently higher than recall, reflecting that it writes fewer criteria than GPT‑5.4.

Precision averages 0.39 across 11 steps, decreasing from 0.44 to 0.33.

During union, the self‑model retains 80 % of GPT‑5.4’s criteria (recall = 0.80) while adding many extra checks.

Recall is computed over 175 groups across 11 steps, exceeding the ceiling of 0.55.

Union precision falls to 0.44, below recall, because the self‑model keeps far more non‑essential criteria.

Precision averages 0.44, dropping from 0.57 to 0.36 (p = 0.002) over 11 steps.

Extra criteria introduced by the self‑model reduce the proportion of discriminative signals to 31.3 % of rollouts.

Measured by the share of criteria that at least one rollout fails, across 11 steps.

By contrast, the baseline (GPT‑5.4) retains 46.5 % discriminative signal, highlighting the cost of surplus criteria.

Same measurement as above for the frontier judge.

84.4 % of DRACO’s distinct criteria are task‑specific, covering 57.2 % of all judgments, confirming that dynamic rubrics add fine‑grained supervision.

Aggregated over 6,245 distinct criteria from 96 rollouts of DRACO.

Step credit contributes +3.2 TGC on per‑trajectory rubrics but only +0.8 TGC on static rubrics, showing its dependence on dynamic criteria.

Reported in Section 4.3; the credit rule activates only when rubrics produce mixed pass/fail outcomes.

When rubrics are static, credit is inert on 76.4 % of rollouts, whereas with DRACO it is inert on only 48.1 %.

Inert share computed over training windows; static rubrics quickly become unanimous.

DRACO’s dynamic rubrics keep credit active on 48.1 % of rollouts, preserving gradient signal throughout training.

Same inert‑share analysis as above.

**Figure 8.** **Training-time judge cost**, in USD, for the four outcome-blind settings and the self-judge variant, which scores with the policy model itself, with thinking enabled and $k=3$ scoring calls per trajectory (Section 4.3). All settings use Qwen3.6-27B as the policy model and GPT-5.4 as the judge, except the self-judge. (a) Cost over 100 training steps, split across the five phases that issue judge calls: task-specific and rollout-specific rubric generation, union, scoring, and credit reallocation. A setting that does not run a phase draws no bar. (b) The same cost per training step against `AppWorld_TN` TGC ($p^1$, Table 2); up and to the left is better. Costs use the token rates of Section 4.1.3.

**Figure 9.** The three jobs a judge does, self-model vs. GPT-5.4, measured by replaying the run's judge calls through the policy checkpoint on the archived prompts, with a single call and no thinking (the ablated self-judge of Table 7). (a) Scoring a rubric it is given. How often the self-model returns the same verdict as GPT-5.4, over all 60,689 verdicts, and the two ways it can differ. Each series has its own denominator, named in the legend. Faint lines are per-step values, bold lines smoothed. (b) Writing the criteria and (c) merging them into the set $a$ group is scored on. Here the two models write their own criteria, so we compare the sets themselves: a third model pairs up criteria that mean the same thing, and we then ask how much the two sets overlap. Recall is how much of GPT-5.4's set the self-model also wrote; precision is how much of the self-model's set GPT-5.4 also wrote. Legends give means, and the shading is simply the gap between the two curves. Perfect overlap is not the target, because two judges writing criteria for the same task will not pick the same checks: the dotted lines show GPT-5.4 compared against itself on two rollouts of one task, and are the reference to read the curves against. The self-model scores a given rubric well but too leniently, reproduces only about a fifth of the criteria GPT-5.4 writes, and at the merge keeps most of GPT-5.4's criteria while adding roughly twice as many of its own.

**Figure 10.** Per-rubric pass rate vs. training step, static rubrics with and without step credit. One panel per static criterion; faint lines are raw per-step values, bold lines smoothed. The dotted series re-scores DRACO against the same fixed set, which it never trained on.

Credit Inertia Analysis

Key ablations reveal how each component of DRACO impacts credit attribution and training efficiency.

The authors measure credit inertia on Qwen2.5‑32B‑Instruct under DRACO, observing a rise from 15.7% to 30.9% across identical windows. This metric reflects how often credit remains unallocated, not whether the reallocation reaches the truly responsible steps.

Removing the dynamic credit‑reallocation mechanism (i.e., using a static rubric) increases inert credit from 15.7% to 30.9%.

Measured on Qwen2.5‑32B‑Instruct under DRACO across the same evaluation windows.

Static rubrics are created once before training: 8 rollouts per each of 90 tasks (720 trajectories) are generated by the untrained base policy, GPT‑5.4 proposes task‑specific criteria, which are then reduced to a single set and pruned for rarity and redundancy.

In contrast, the dynamic setting generates fresh criteria at every training step, allowing the rubric to adapt continuously to the evolving policy.

Both static and dynamic generation calls enforce MECE (mutually exclusive, collectively exhaustive) criteria. For the static set, pruning removes criteria that are rarely applicable or discriminative, ensuring the remaining 21 criteria cover the task space effectively.

Table 9 orders the 21 criteria by failure rate over the 720 rollouts. Thirteen criteria apply to at least 99% of rollouts, while eight apply to fewer, with the lowest coverage at 29.9% for “Handles time correctly”. Failure rates range from 81.8% (“Protects secret values”) down to 2.8% (“Uses allowed tools only”), motivating the discriminative dropout described in Section 3.2.

Table 10 details the training configuration shared by all four reward variants. The setup uses 16 prompts and $G=6$ rollouts, yielding 96 rollouts per step, runs for at most 20 epochs (100 optimizer steps) with a checkpoint after each step, and is executed on 8 NVIDIA H100 GPUs.

Three design choices are easy to mis‑interpret: (1) only LoRA adapters (rank‑16) are trained, leaving the base model frozen; (2) a single PPO epoch per batch makes updates on‑policy, rendering the clipping range effectively inactive; (3) the learning rate stays constant at 5 × 10⁻⁵ with no warm‑up, so no schedule interferes with the final‑checkpoint reporting.

For serving, Qwen2.5‑32B‑Instruct runs with its native 32,768‑token context, using its chat template and a JSON‑based tool‑call parser. In $\tau$‑bench the window is stretched to 131,072 tokens via YaRN rope scaling, and thinking mode is disabled.

$\tau$‑bench retrieval employs Qwen embeddings for all‑tools settings, ensuring consistent retrieval behavior across base policies.

Credit Reallocation Derivation

DRACO learns a process rubric and then redistributes the trajectory‑level advantage to individual steps.

Recall that DRACO first defines success with a process‑based rubric and then reallocates the resulting reward from the whole rollout to the individual reasoning steps.

Compute weights: $w_1 = 1.0$, $w_2 = 0.5$, $w_3 = 0.0$; sum of weights $\sum_k w_k = 1.5$.

Total tokens $N = 2 + 4 + 2 = 8$.

Calculate $a_1 = 1 \times \frac{8 \times 1.0}{1.5 \times 2} = 2.67$.

Calculate $a_2 = 1 \times \frac{8 \times 0.5}{1.5 \times 4} = 0.67$.

Calculate $a_3 = 1 \times \frac{8 \times 0.0}{1.5 \times 2} = 0.00$.

Check conservation: $2\cdot2.67 + 4\cdot0.67 + 2\cdot0.00 = 8 = A_i N$.

The highest‑quality step receives the largest total influence (here $2\cdot2.67$), while a step with zero quality gets no influence at all, even though it contains tokens.

The following properties follow directly from the formulation: (P1) total advantage is conserved; (P2) unanimous rubric verdicts yield uniform step totals; (P3) the sign of $A_i$ is never flipped; (P4) on winners higher $Q_j$ yields larger $a_j$, and on losers lower $Q_j$ yields larger suppression; (P5) step totals are independent of length; (P6) scaling the reward rescales all $a_j$ uniformly; (P7) the loser branch mirrors the winner branch with quality flipped.

The table lists five rubric checks (R1-R5), their corresponding verdicts (FAIL or PASS), and the specific steps cited by each check.

**Table.** Baseline GRPO would put $a_j = 1.0$ on all seven steps ($\sum_j n_j a_j = 310$). Credit keeps the same total, $\sum_j n_j a_j = A_i N = 310$, but rebalances it: the strong steps take totals of 72.47 against 24.16 for the flagged ones, a $3\times$ gap matching their weights, while the uncited step sits between them. Changing the token counts changes the per-token advantages but never the step totals (they depend on $w_j$ only) and never the conserved total. The $a_j$ column is therefore not a ranking of the steps: a short step spreads its share over fewer tokens and so shows a larger per-token value, which is why the step totals rather than $a_j$ are the quantity to read across rows.

**Stage 1: tally passes/fails per step.** For each step $j$, count the citing checks that passed ($p_j$) and failed ($f_j$). Step 7 is cited by nothing.

This table outlines specific configuration knobs for a model, including "Length handling" with symbol $1/n_j$ and value "step total is length-free", "Gap tokens" with value "advantage 0, outside $N$", and "Uncited-step quality" with value "$\bar{Q}$ over cited".

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers