Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
Jingjie Ning, Shanshan Zhong, Xiaochuan Li, Ji Zeng
The Discovery Certification Protocol (DCP) provides a verifiable, executable audit framework for AI research agent claims.
How can we rigorously verify that an AI research agent's claimed discovery is genuine rather than a lucky artifact of its training data or search process?
AI research agents increasingly automate complex workflows, but their reported results often lack evidence connecting the final output to the specific research process that produced it. The Discovery Certification Protocol (DCP) addresses this by turning research claims into executable, replayable audits that validate utility, test for alternative routes to the same result, and measure the causal effect of feedback. In controlled audits of SQLite optimization and virtual catalyst design, the protocol successfully distinguished between genuine discovery and recoverable outcomes, providing a deterministic, machine-verifiable certificate for each claim.
Paper Primer
DCP organizes an audit around three gates: validating useful improvement on a sealed test, testing for "recovery" by a matched agent given the same starting information, and measuring the effect of truthful feedback versus a neutral policy. The core move is the use of a shared, deterministic verifier that recomputes decisions from frozen evidence bundles, ensuring that any researcher can independently replay the audit.
DCP provides a robust, portable evidence language for AI research contributions.
The protocol was validated across two complete three-gate audits (SQLite optimization and virtual catalyst control) and three diagnostic cases, with all decisions reproduced by a deterministic, LLM-free verifier. In both primary audits, the system recorded zero recoveries in 96 episodes, establishing a recovery probability upper bound of 0.0468.
The protocol successfully isolates the causal effect of feedback on agent performance.
Paired studies comparing truthful feedback against a neutral policy showed a clear performance increment, with the target lower bound exceeding the required total margin of 0.51. Both paired studies yielded 30/30 truthful recoveries versus 0/30 neutral recoveries, with a binary policy effect of 1.0.
Why is a "recovery" test necessary for AI research agents?
A matched agent might reach the same score as the target run using only initial data and public material, meaning the reported "discovery" may not be a unique result of the agent's specific research history. DCP uses recovery tests to determine if an outcome is easily reproducible by a challenger, thereby clarifying the contribution of the agent's unique research actions.
What happens if a challenger finds an alternative route to the same result?
If a challenger produces a valid artifact that meets the score threshold, it triggers a "Core veto," recording the outcome as recovered. This ensures that the audit protocol remains neutral to the specific method used, focusing instead on whether the outcome is truly novel given the registered information boundary.
The Problem of Agent Discovery
We expose the gap between high-scoring AI outputs and true discovery, and introduce the Discovery Certification Protocol.
AI research agents can generate impressive scores that mask whether the result stems from genuine discovery or from exploiting leaked data. This ambiguity creates a concrete gap: high scores do not guarantee that the underlying research insight is novel or reproducible.
The community needs a principled way to separate true scientific progress from artifacts that merely achieve high performance.
The DCP proceeds through three gates: Gate 1 checks that the reported improvement exceeds a sealed baseline; Gate 2 provides the agent with registered background data while hiding the target’s experimental history; and an optional Gate 3 quantifies the causal impact of truthful feedback versus a neutral policy.
```text {jening, szhong2, xiaochu4, jizeng}@cs.cmu.edu [Homepage] [PyPI / dcp-audit] [PyPI / dcp-harness] [GitHub] ```
The essential gap is that high scores alone do not certify genuine discovery; DCP bridges this gap by demanding recoverable evidence under strict information boundaries.
The Discovery Certification Protocol
The protocol defines a three‑gate audit that registers tasks, checks improvement, tests recovery, and measures feedback impact.
The Discovery Certification Protocol (DCP) operationalizes a rigorous audit of AI research agents by chaining three verification gates.
The DCP is a three‑gate audit that forces an AI research agent to prove a claimed result by first showing a useful improvement, then demonstrating that a fresh challenger can recover the outcome, and finally measuring how feedback changes performance under a sealed information boundary.
How does the DCP differ from a standard validation split?
Standard validation only measures performance on held‑out data; DCP adds a precommitment stage, a recovery challenge that must match the original score within a tolerance, and a feedback‑policy effect measurement, all under a strict information boundary.
Register the task with $\delta_{min}=0.05$ and $\varepsilon=0.02$.
Run the target agent, obtain a sealed score $x=0.85$.
Gate 1: compute LCB[µ(A*)‑µ(b)] = 0.06 ≥ $\delta$ₘᵢₙ, so the improvement passes.
Gate 2 challenger receives the same $K$ and $E_0$, produces score $0.84$, which satisfies $0.84 \geq x-\varepsilon$.
Gate 3: from checkpoint $c$, the truthful branch yields utility $0.90$, the neutral branch $0.78$, giving $\Delta_{feedback}=0.12$.
Even a modest improvement $\delta_{min}=0.05$ can be certified as genuine, and the feedback‑policy effect $\Delta_{feedback}=0.12$ demonstrates that the protocol captures downstream utility.
Register the task, fixing $K$, $E_0$, the selection rule for $A^*$, budgets, and analysis procedures.
Run the target agent to produce outcome $A^*$ and record the sealed score $x$.
Compute Gate 1: verify that the lower‑confidence bound on the mean utility gain over baseline exceeds $\delta_{min}$.
If Gate 1 passes, launch a fresh challenger with the same $K$ and $E_0$ but without $L^*$.
Apply Gate 2: require the challenger’s score to satisfy $score \ge x - \varepsilon$.
Collect multiple challenger episodes to estimate recovery probability $p_B$ and compute the upper bound $p_{upper}=1-\alpha/n$.
Select a shared checkpoint $c$ and run two branches—one receives truthful feedback, the other a neutral policy.
Compute Gate 3 feedback effect $\Delta_{feedback}=E[u(A_{truthful})-u(A_{neutral})\mid c]$ and check that its lower‑confidence bound exceeds $\delta_{evidence}+\delta_{sham}$.
Log every web query, response time, and bytes in the recording gateway to enforce the information boundary.
The audit authority verifies all registered conditions and reports the final decision, combining Gates 1‑3 and the evidence from feedback evaluation.
**Discovery Certification Protocol (DCP)**. The protocol consists of three main stages: (01) Validate a useful result, (02) Try to recover the outcome, and (03) Is feedback doing useful work? The process involves research agents, experiments, and feedback loops to establish a DCP Core and DCP Evidence.
Experimental Audit Design
Calibrating the protocol with controlled audits reveals strong recovery and large efficiency gains.
We calibrated the Discovery Certification Protocol by running controlled three‑gate audits on three distinct tasks. The design isolates information flow, allowing us to measure recovery and efficiency under strict boundaries.
The protocol recovered near‑optimal performance in the audits, achieving a top recovered score of 0.9363.
Table 2 shows the highest recovered score of 0.9363 among all cases; additional scores confirm consistent recovery across tasks.
**Figure.** DCP decisions across positive, negative, and underpowered cases.
Audit Results and Evidence
Key quantitative outcomes of the DCP audits and their implications.
This section reports the quantitative results of the Discovery Certification Protocol audits across the three test domains.
All four outputs cleared Gate 1 and the deterministic verifier achieved DCP Core in every scope.
SQLite‑Web, catalyst, device calibration, and knapsack each passed Gate 1; the verifier reported Core for all three audit scopes.
Positive controls achieved a recall lower bound of 0.8889, well above the 0.8 minimum, while the zero‑hit upper bounds (0.0468 and 0.0477) remained far below the threshold $\rho$.
**Table 2.** Primary DCP audit outcomes. Scores are normalized and higher is better within each task. Flash and Pro identify the models recorded in the provider responses.
**Figure 2.** DCP applies the same outcome rule across artifacts. The best no-lineage challenger stays below the registered recovery line in the three Core cases and crosses it in knapsack. The challenger may use any valid method.
**Figure 3.** Complete three-gate evidence in two domains and model scopes. The upper row shows all 96 no-lineage scores per audit. The lower row shows the registered 99% intervals for the checkpoint-conditional feedback effect and the separate null calibration. Each column is one audit.
Discussion and Implications
We outline limits, deployment scenarios, and broader implications of the Discovery Certification Protocol.
The Discovery Certification Protocol (DCP) forces agents to prove their results by recovering outcomes under strict information boundaries and showing that feedback yields genuine utility. This refresher restates the core premise without re‑deriving it.
6.1 An outcome contract admits alternative methods. The DCP evaluates recovery across every admissible implementation a challenger registers, so a recombination of known modules, a transfer from another domain, or a different program can all supply the same numerical witness. The knapsack case illustrates this rule directly, with task adapters providing validity and utility definitions while the shared protocol supplies the information boundary and decision procedure.
6.2 Reporting a research contribution. A reusable report records the outcome threshold, model and information scope, complete episode budget, qualified recovery witnesses or probability bound, and any checkpoint‑conditional feedback interval. Authors can state precisely what their experiments support; reviewers can inspect alternative routes and replay the numerical decision; later studies can register a new model or budget and compare recovery under that scope. Core and Evidence fields summarize these records while preserving the quantities needed for comparison.
6.3 Deployment and retrospective recovery. Prospective use begins with an approved registration and sealed evaluator, then proceeds through evidence collection, control execution, offline replay, and independent countersigning. Auditors may stop at the first qualified recovery and reserve fresh paired studies for claims that request feedback evidence. The catalyst and SQLite‑Web audits used 435 and 507 recorded sessions and cost 56.40 USD and 61.17 USD respectively; the expensive unit is the registered generation episode, while verification operates on its saved evidence. Retrospective use reconstructs an outcome target and seeks a qualified recovery witness, with pre‑publication‑equivalent audits testing the original scope and post‑publication audits measuring present‑day recoverability. Positive Core and Evidence decisions rely on prospective registration and complete evidence collection.
7 Conclusion. The DCP makes evidence about AI research contributions executable by following a single numerical outcome through sealed utility validation, recovery by matched agents given the starting information, and an optional randomized feedback comparison. Any qualified alternative method can supply a recovery witness. Core requires an audit with zero recoveries and a finite‑sample bound for one fresh registered episode; Evidence adds a checkpoint‑conditional effect of truthful feedback over a neutral policy, backed by independent null calibration and a registered margin. These decisions provide utility, alternative routes, and feedback effects as separate, checkable meanings.
Two controlled audits exercised the full protocol under distinct models. DeepSeek‑v4‑flash reduced traffic‑weighted SQLite virtual‑machine work by 88.55 %, while DeepSeek‑v4‑pro reached the optimum in virtual catalyst control. Each audit recorded zero recoveries in 96 episodes and 30 truthful recoveries against zero neutral recoveries, with passing 60‑pair null calibration. Device, knapsack, and affine cases exercised Core, recovered, and incomplete decisions, and a shared deterministic verifier reproduced the complete decisions from frozen records. The contribution is a portable experimental language that authors can report, auditors can challenge, and readers can verify across domains.
Reproducibility Statement. Frozen machine‑readable bundles and evidence records support the numerical claims; the appendices report task construction, registrations, model and interface contracts, evaluation units, thresholds, statistics, budgets, costs, and identifiers. The release provides the installable dcp‑audit, task‑specific evidence producers, reusable dcp‑harness, example bundles, and figure‑generation scripts. Decision replay is deterministic and offline, and fresh evidence production uses the model and environment specified in the registration. The harness exposes capture, challenge, paired feedback testing, finalization, and bundle generation as separate operations.
Ethics Statement. AI research agents can accelerate useful discovery while increasing the need for clear and contestable evidence. DCP supports accountable reporting by binding each decision to a registered model, information boundary, budget, probability bound, and replayable record. Certificates should be presented with this scope, and captured Web data should follow applicable privacy, licensing, and security requirements.
AI Use Statement. Generative AI tools supported ideation, protocol and statistical design, adversarial review, experiment design, implementation, testing, evidence inspection, result interpretation, literature work, and manuscript editing. GPT and Codex assisted these activities; Claude interfaces supported review and provided the registered CLI execution layer. DeepSeek‑v4‑flash and DeepSeek‑v4‑pro served as the audited agents and matched challengers. Image generation informed early teaser drafts, quantitative plots were generated with Matplotlib from frozen evidence, and executable evaluators and statistical code determine scores and decisions. The authors review the scientific claims, check evidence and primary sources, test the code, and take responsibility for the final content and artifacts.
Supplementary Information
Appendices detail the statistical decision process, audit validity, and case records for the Discovery Certification Protocol.
A.1 Main result – For each paired private evaluation unit, the DCP computes the difference between the optimal score $A^\star$ and the baseline, using a bounded empirical‑Bernstein interval on the $[0,1]$ score scale; fully enumerated sets use their exact mean.
A.2 Candidate decisions – Each challenger candidate is scored on the same private units as $A^\star$, with $d_i$ denoting its paired score difference; a candidate is a confirmed hit if its adjusted lower confidence bound exceeds $-\varepsilon$, a confirmed miss if its adjusted upper bound falls below $-\varepsilon$, otherwise it remains unresolved.
A.3 Recovery witnesses and repeated search – For independent Bernoulli episode outcomes with recovery probability $p_B$, the probability of zero recoveries in $n$ episodes is $(1 - p_B)^n$, leading to a required sample size $n \ge \lceil \log(\alpha_{\text{recovery}}) / \log(1 - \rho) \rceil$; with $\alpha_{\text{recovery}}=0.01$ and $\rho=0.05$, the audits needed 90 episodes, and 96 were recorded.
A.4 Feedback decisions – The paired feedback estimator evaluates normalized utility differences $\Delta_{\text{feedback}}$; a decision is made by comparing the lower confidence bound LCB($\Delta_{\text{feedback}}$) to $\delta_{\text{evidence}}+\delta_{\text{sham}}$ and the upper confidence bound UCB($\Delta_{\text{feedback}}$) plus $\delta_{\text{sham}}$ to $\delta_{\text{evidence}}$.
B Audit Validity – Validity requires that registration is sealed before any private result, that model identity, interface, tools, information packet, and resource budget match their registered values, and that every attempt, including timeouts and infrastructure failures, is recorded in the ledger.
C.1 SQLite with captured Web access – Using DeepSeek‑v4‑flash, the main score was $0.8855470$, baseline $0$, recovery line $0.8805470$, with 96 challenger episodes all confirmed misses; positive controls (45) gave a one‑sided recall lower bound of $0.8889268$.
C.2 Virtual catalyst optimization – With DeepSeek‑v4‑pro, the main score $1.0$ exceeded baseline $0.5989989$; recovery tolerance $\varepsilon=0.05$ and $\delta_{\text{min}}=0.35$ were met, and 96 challenger episodes were confirmed misses.
C.3 SQLite with Web disabled – The core audit recorded a main score $0.9432886$, baseline $0$, recovery line $0.9382886$, and 80 challenger episodes all confirmed misses; the zero‑hit upper bound was $0.0477239$.
C.4 Device calibration – The main score $1.0$ and baseline $0.3667$ yielded a recovery line $0.95$; with 80 challenger episodes, the zero‑hit upper bound was $0.0477$, and 60 positive controls gave a Clopper‑Pearson lower bound of $0.9155$.
C.5 Multidimensional knapsack – The main score $0.9348968$ surpassed baseline $0.9140206$; two candidate opportunities within one challenger episode produced scores $0.9362833$ and $0.9355607$, triggering a core veto and confirming a gain of $0.0208762$ above $\delta_{\text{min}}=0.01$.
C.6 Development checks retained as inconclusive – An affine parity run exercised all three gates with limited episodes, yielding main and feedback scores of $1.0$ but a positive‑control recall lower bound of $0.0707$, leading to an incomplete audit for challenger adequacy.
D Audit Record – Each audit record contains the claim, information boundary, final output hash, baseline, sealed score data, recovery predicate, challenger contract, complete attempt ledger, control adequacy results, statistical intervals, and decision; an optional feedback record adds checkpoint, paired branch contracts, neutral channel calibration, and paired estimates.