Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts
Jingjie Ning, Xueqi Li, Yibo Kong, Dongting Li
A systematic evaluation of whether scientific explanations improve experimental forecasts across 336 prospective research states.
Do scientific explanations actually improve the accuracy of experimental forecasts, or are they just noise?
Research agents often generate experimental proposals alongside rationales, but it remains unclear if these explanations actually improve the accuracy of predicted outcomes. The authors evaluate this by measuring "predictive credit"—the gain in forecast accuracy when an agent is provided with its own explanation versus a baseline description or a donor explanation from a different experiment. Across three prospective studies and multiple controls, the authors find that while structured explanations can reduce forecast drift, they fail to provide consistent, statistically significant gains in point-accuracy or interval-score performance.
Paper Primer
The study uses a paired-forecast design to isolate the value of an explanation. By holding the forecaster, intervention, and outcome constant, the authors compare the performance of a "description" baseline against a "matched" explanation and a "donor" explanation (an explanation from a different, frozen state).
Scientific explanations fail to provide consistent predictive gains over baseline descriptions.
In Tox21 and OpenML benchmarks, predictive credit remained unconfirmed, with interval-score harm tests returning "no go" and seed-donor intervals spanning zero. Tox21 ROC AUC interval-score harm test: D − M = −.0026 (95% interval [−.0174, .0104]).
Structured explanation cards reduce forecast drift, even when they do not improve accuracy.
In Tox21, matched cards reduced repeat forecast drift by 64.5% compared to description-only baselines. Pro repeat drift fell from .01004 (description) to .00357 (matched).
Why is "predictive credit" a better metric than simply evaluating the quality of the explanation text?
Textual quality is subjective and often decoupled from experimental reality; predictive credit anchors the evaluation to the agent's ability to anticipate the consequences of a specific, executed intervention.
Does this paper suggest that research agents are incapable of useful reasoning?
No; the authors demonstrate that agents can uptake information when provided with a "known-mechanism positive control," suggesting the failure to gain credit is specific to the current generation of natural-language explanations rather than a total inability to process information.
Introduction: Measuring Predictive Credit
We expose the experimental gap: explanations often fail to improve forecasts, prompting a need for frozen forecasts.
Scientific explanations are routinely attached to experimental forecasts, yet it remains unclear whether they add genuine predictive power or merely accompany the raw data. This uncertainty creates an experimental gap: we lack a principled way to quantify the “predictive credit” that explanations contribute.
A forecast generated before the experimental outcome is observed and then held fixed while the explanation is evaluated, thereby preventing any hindsight influence.
Compute the difference: $D - M = -0.0026$, which is negative.
Check the confidence interval: the interval spans zero, indicating no statistically reliable gain.
Conclude that the matched explanation did not improve predictive performance over the description baseline.
The failure to achieve a positive difference shows that predictive credit remains unconfirmed for this Tox21 endpoint.
**Figure.** Evaluating Scientific Explanations: Freeze forecasts before observing experimental outcomes.
Pre‑outcome freezing is essential to avoid hindsight bias when evaluating explanatory credit.
The Evaluation Framework
Defines the protocol for measuring predictive credit of scientific explanations.
The evaluation framework compares three forecast contexts—description‑only ($D$), matched explanation ($M$), and donor explanation ($S$)—to quantify how much predictive information the explanations add.
Predictive credit measures the extra forecasting accuracy gained when an explanation is supplied that truly matches the experiment, beyond what the raw description already provides.
How does predictive credit differ from a generic “explanatory power” score?
Predictive credit is tied to a concrete forecasting loss (e.g., MAE) and a fixed forecaster, so it isolates the *forecasting* benefit of the explanation. Generic explanatory‑power metrics often mix model capacity, data fit, and interpretability, making it unclear whether the gain is due to the explanation itself.
A matched explanation is the explanation that was authored for the exact experiment whose state $x_i$ and intervention $a_i$ are being forecasted.
Why not simply use any explanation instead of insisting on a matched one?
Using an arbitrary explanation (donor) can introduce spurious correlations that inflate performance without reflecting true causal insight. The matched condition ensures the explanation is directly relevant to the intervention, making the measured gain attributable to genuine predictive content.
Collect the public state $x_i$, planned intervention $a_i$, and explanation $h_i$ for each experiment.
Form three contexts: $D_i=(x_i,a_i)$, $M_i=(x_i,a_i,h_i)$, and $S_i=(x_i,a_i,h_{\pi(i)})$ where $\pi$ shuffles explanations.
Run the frozen forecaster $f$ on each context to obtain predictions $\hat{y}_{D_i}$, $\hat{y}_{M_i}$, and $\hat{y}_{S_i}$.
Compute losses $R_D$, $R_M$, $R_S$ using MAE against the held‑out outcome $y_i$.
Derive predictive‑credit gains $G_{\text{description}}=R_D-R_M$ and $G_{\text{alignment}}=R_S-R_M$.
Obtain two independent forecasts per instance to calculate agreement drift $A_{ic}$.
Generate 80 % prediction intervals $[L,U]$ and evaluate the interval score $\text{IS}_{80}$.
Aggregate all metrics across tasks, report the five checks (commitment, delivery, paired gains, donor alignment, signal sensitivity), and compute task‑wise ratios.
Compute MAE loss for $D$: $R_D = |0.6 - 0.8| = 0.2$.
Compute MAE loss for $M$: $R_M = |0.75 - 0.8| = 0.05$.
Predictive‑credit gain: $G_{\text{description}} = 0.2 - 0.05 = 0.15$ (positive, so the matched explanation improves prediction).
This tiny example shows how a single matched explanation can reduce the MAE loss, illustrating the core idea of predictive credit.
Experimental Settings
We vary interventions across three domains while keeping the model family fixed.
We evaluate three experimental settings—synthetic learning regimes, molecular assay‑activity targets, and heterogeneous tabular prediction tasks. For each setting we modify features, objectives, regularization, or training budget while keeping the underlying modeling family constant.
When a forecast is generated, a “donor” version of the same forecast is produced by swapping the explanation card with one drawn from a different but compatible state, preserving the endpoint, budget, and action.
How does a Donor Context differ from simply using a different description card?
In a description card the forecast and its explanation are generated together for the same state; in a donor context the forecast stays tied to its original state while the explanation is taken from a distinct “donor” state that matches on endpoint, budget, and action. This controlled swap isolates the explanatory signal.
Compute the denominator as $\max(0.05, 0.01)=0.05$.
Divide the raw error by the denominator: $0.02 / 0.05 = 0.40$.
Average across the (single) example yields a task‑scaled MAE of $0.40$.
Task‑scaled MAE normalizes errors relative to the typical magnitude of change in that task, preventing tasks with tiny changes from dominating the overall error metric.
Table 1 reports the paired loss gains for each study, following the $D\!-\!M$ or $S\!-\!M$ contrast defined in Eq. 2. Figure 2 visualizes the ROC‑AUC change across states, highlighting the matched and donor cards.
Predictive Credit Results
Matched explanations cut forecast drift and boost predictive credit across domains.
Matched explanations reduce forecast drift substantially compared to description‑only forecasts.
Across Tox21 and Flash, drift reductions reach up to 64.5%.
All predictors used the same cards and state configurations; only the matching of explanations varied.
**Table.** Study task and state distribution.
Matched explanations consistently provide predictive credit across the tested domains.
Commitment and Uncertainty
We quantify how elicitation, sign priors, and interval width affect commitment rates and prediction quality.
This section dissects how different elicitation strategies, sign priors, and interval widths influence commitment rates and predictive performance.
**Figure 3.** Different observables identify different behaviors. Formation varies across study and interface. Direction accuracy closely follows the sign prior. OpenML coverage measures the placement of stated uncertainty. The positive control compares point error and drift.
Delivery and Scale
How delivery changes performance and what stays constant.
The Flash pipeline shows no detectable matched point‑accuracy advantage over description‑only forecasts.
The raw‑MAE donor‑minus‑matched gain was –0.00114 with a 95 % interval of [–0.2687, 0.0403], which includes zero.
The model consumes the original free‑form notes directly, bypassing any slot‑extraction step, and predicts the outcome from that raw text.
**Figure 4.** The analysis population determines the interpretation. The delivery panel compares six-slot extraction and direct-text delivery. The scale panel shows pooled ensemble MAE with and without `brazilian_houses`, alongside constant-zero baselines.
**Figure 5.** Outcome size and task composition shape forecast evaluation. Left, call-level interval coverage on nested subsets selected by realized outcome size. These are post hoc descriptive comparisons. Right, each point shows the change in pooled ensemble MAE after omitting one of the 24 tasks. Positive values indicate lower error after omission. Colors identify each forecast context.
Controlled Study Results
Controlled study reveals matched explanations modestly boost forecast correlation while preserving accuracy.
Matched explanations improve point‑forecast correlation relative to description‑only forecasts.
Table 6 reports a point‑forecast correlation of 0.4964 for the matched‑explanation condition.
The table presents a comparison of different metrics (accuracy MAE, log-loss MAE, and interval score) between "Description" and "Shuffled" conditions relative to a "matched" baseline. It includes columns for the "Estimate" and the "Paired 95% interval".
The table presents performance metrics across five strata: Proposal, Spontaneous extraction, Description forecast, Matched forecast, and Shuffled forecast. Columns include "Units", "Response", and "Semantic" accuracy percentages.
**Table 8.** Controlled v5 frozen contrasts
This table presents statistical estimates and 95% confidence intervals for various performance metrics (ROC AUC, Average precision, Log loss) across different harm quantities (Interval-score harm, Point-MAE harm) and control conditions (Description, Shuffled).
Tox21 Analysis
Frozen Tox21 results show modest predictive‑credit gains but reveal interval‑score violations.
Matched forecasts reduce repeat drift by 60.55 % versus the frozen baseline, while shuffled forecasts achieve a 59.17 % reduction.
Matched repeat drift fell from 0.01004 to 0.004264; shuffled repeat drift fell from 0.01535 to 0.00618 (approx).
All other experimental settings—model architecture, training data, and the frozen 0.005 minimum interval‑score threshold—were held constant across these comparisons.
**Table 10.** Tox21 frozen central-80-percent ROC AUC interval-score contrasts
**Table.** The table displays the impact of various actions on model performance metrics, specifically comparing against description ("vs desc") and shuffle ("vs shuffle") baselines, alongside budget constraints.
**Table 12.** Tox21 prospective-card calibration
**Table 13.** Tox21 secondary matched-card contrasts
**Table 14.** Tox21 endpoint-specific interval-score harm
The table presents performance metrics (Point MAE, Coverage, Width, Interval score, and Repeat drift) across three conditions (Description, Matched, and Shuffled) for three different evaluation metrics: ROC AUC, Average precision, and Log loss.
The table presents data comparing two arms: "Spontaneous" and "Elicited" across seven categories: Complete, Direction, Point or interval, Observable, Benefit, Falsifier, and Counterfactual.
OpenML Population Outcomes
Elicited explanations dramatically increase extraction completeness while preserving forecast skill.
The OpenML appendix reports a full population of 144 route states with 2,304 prospective logical calls, of which 90 interventions were positive and 54 negative. Median absolute skill change across interventions was 0.0236264, indicating modest but consistent effects.
Elicited explanations produce far more complete extracted cards than spontaneous ones.
Spontaneous complete cards = 1 (0.69 %); Elicited complete cards = 20 (13.89 %).
Known-Signal Calibration
Calibration shows the exact outcome signal vastly outperforms all other conditions.
This calibration evaluates five post‑outcome conditions across 144 states, using Point MAE, Scaled MAE, Coverage, Width, Score, and Spearman correlation as metrics.
**Table 26.** Flash practical-equivalence checks. Signed 95 percent intervals are compared with the frozen margins; a pass requires full interval inclusion.
Flash Follow-up Controls
Ablation results for Flash controls across Tox21 states and practical equivalence checks.
This section reports the ablation study of Flash controls on the 72 Tox21 states, evaluating eight forecast contexts and checking practical equivalence against frozen margins.
Across all contexts the primary point‑MAE contrast shows matched mechanisms outperform donor mechanisms when target numbers are fixed, while the reverse swap confirms that fixing the mechanism text isolates the effect of donor target numbers.
All 1,152 Flash calls succeeded; the drift reductions from target‑only forecasts (.00181) and from the full card (.00108) both fall within the frozen ±.002 drift margin, and the point‑MAE differences respect the ±.005 margin, confirming practical equivalence.
Study Design Details
Overview of the study designs, evidence timing, and calibration protocols used in the appendix.
Evidence for the appendix comes from three prospective source studies—Controlled v5, Tox21 anchor, and OpenML v2.1—each providing a set of pre‑outcome states that were later re‑analyzed post‑outcome. Calibration calls (1,440 in total) followed a frozen protocol, and the Tox21 Flash replay used its own frozen gate. Table 3 records the timing, population, and role of each study and follow‑up component.
The table compares various performance measures between "Spontaneous" and "Elicited" conditions across six categories: Complete card, Quantitative magnitude, Explicit direction, Intermediate observable, Direction correct given claim, and Central-80 interval covered.
Controlled v5 comprised 120 states that combined six fast‑learning regimes with frozen random seeds, split evenly between spontaneous proposals and explicit elicitation. Each state received two independent forecasts across three contexts, yielding 900 required units (120 proposals, 60 spontaneous extractions, 720 forecasts). One proposal was missing, leaving a single extraction slot unissued but still counted in the denominator.
The Tox21 anchor built a lattice of 12 endpoints, three label budgets, and two seeds, assigning actions via a modulo‑6 rule that balances loss functions, fingerprint sizes, and regularization strengths. A frozen derangement shuffled seed cards within each endpoint‑budget‑action cell, preserving the experimental structure. Interval scores were computed using a piecewise formula that penalizes predictions falling below the lower bound L or above the upper bound U, scaled by the confidence level $\alpha$ = 0.20.
OpenML v2.1 selected 12 classification and 12 regression confirmation tasks via a hash lottery, excluding image‑heavy and temporally dependent datasets. Tasks were split into training (≈60 %), validation (≈20 %), and private outcome (≈20 %) partitions, stratified by class or target‑quantile. Six interventions—regularization variants, subsampling, and boosting rounds—were cyclically assigned across task‑budget combinations, with two frozen seeds per configuration.
**Table 4.** OpenML classification confirmation tasks
**Table 5.** OpenML regression confirmation tasks
Assay calibration reused all 144 OpenML states after outcomes were known, delivering three signal variants: exact (true skill change), noisy (deterministic sign times half the larger of task MAE or 0.01), and shuffled (outcome from the opposite seed in the same cell). All signal choices and call orders were frozen before the first calibration call, and each of the five contexts received two independent calls. The calibration gate required the exact signal to outperform description in ≥ 75 % of states with a bootstrap lower bound > 65 %, and to beat the wrong‑seed signal in ≥ 70 % of states with a lower bound > 60 %.
Post-Outcome Replay Analyses
Appendix G reports replay experiments and a known‑mechanism control analysis.
All 144 frozen Pro‑authored OpenML notes were fed to a Flash forecaster while public states, assigned interventions, and within‑task seed donors stayed fixed. Each state received two fresh forecasts under description, matched raw note, and donor raw note, yielding 864 valid first responses. Both matched and donor contexts included the source note text.
The prospective pipeline used a Pro predictor with six‑slot cards; this replay instead used Flash with raw note text.
Table 27 summarizes the point‑MAE, interval score, coverage, and repeat drift across the three contexts.
**Table 27.** OpenML raw-note Flash replay over the complete 144-state population
Task‑balanced scaled point‑MAE gain for matched versus description was −0.09059 (95 % bootstrap interval [−0.26870, 0.04033]), while matched versus donor gain was +0.00638 (interval [−0.09641, 0.08983]). Raw‑MAE gains were −0.00054 and −0.00114 respectively, with intervals spanning zero, suggesting no statistically reliable advantage.
G.3 Known‑Mechanism Positive Control evaluates forecasts when the true underlying mechanism is disclosed versus a paired false mechanism.
Twenty seeds each generated paired linear and nonlinear label‑generating tasks with identical quadratic‑feature interventions; the donor note described the alternative functional form. Forty states received two fresh Flash forecasts in each of the description, matched‑note, and donor‑note contexts, producing 240 valid first responses.
Table 28 reports performance under true and paired false mechanism clues.
**Table 28.** Flash forecasts under true and paired false mechanism clues
The quadratic intervention raised held‑out accuracy by 11.440 points on average in the nonlinear tasks but decreased it by 1.055 points in the paired linear tasks, with mechanism‑conditioned forecasts moving toward these outcomes while preserving uncertainty.