TestPrism: Rethinking Test Evaluation Beyond a Single Reference
Han Li, Lingxiang Hu, Jiacheng Huang, Ziqian Jiang, Jingkai Luo, Wei Gao, Yunfan Tan, Zun Wang, Jiaheng Liu
TestPrism evaluates coding agents against diverse implementation panels, revealing that single-reference testing overstates reliability.
How can we evaluate LLM-generated tests more reliably by moving beyond a single reference solution?
Coding agents are often evaluated against a single "gold" reference solution, which fails to account for the fact that many programming tasks admit multiple valid implementations. This narrow focus allows flawed tests to pass by missing required behaviors or imposing unnecessary constraints. TestPrism addresses this by evaluating generated tests against a hidden panel of valid and invalid implementations, requiring tests to correctly accept all valid alternatives and reject all invalid ones. Across fourteen coding agent configurations, the best-performing model achieves only 28.00% on this joint metric, compared to 59.67% under traditional single-reference evaluation.
Paper Primer
The authors introduce TestHelix to improve test reliability through a three-part workflow: heterogeneous synthesis of diverse test-repair pairs, peer cross-validation to refine assertions against multiple implementations, and recursive self-improvement to optimize generation strategies. This approach shifts the focus from matching a single reference to ensuring tests generalize across the entire behavioral space of a task.
TestHelix significantly improves test reliability over native coding agents.
Ablation studies show that combining heterogeneous synthesis, peer cross-validation, and recursive self-improvement raises the Joint Success Function. An improvement of 11.67 to 12.00 percentage points over single-reference evaluation under matched inference budgets.
Current frontier models struggle to generate tests that generalize across implementations.
Analysis of 286 failed trajectories reveals that tests frequently suffer from coverage gaps (34.6%), faulty checks (50.0%), and execution logic errors (15.4%). The highest baseline Joint Success Function is only 28.00%, less than half of the 59.67% success rate observed under single-reference evaluation.
Why is single-reference evaluation insufficient for modern coding agents?
It fails to distinguish between tests that are truly robust and those that are merely overfitted to one specific implementation, allowing tests to pass while missing required behaviors or accepting invalid patches.
What does the Joint Success Function measure that traditional metrics miss?
It measures the ability of a test suite to correctly label a diverse panel of implementations, requiring initial failure on the program state, acceptance of all valid alternatives, and rejection of all invalid ones.
Reliable test generation requires moving beyond single-reference benchmarks; developers should adopt multi-implementation panels to ensure tests are sensitive to specification violations rather than implementation artifacts.
Introduction and Motivation
We expose the flaw of single‑reference test evaluation and introduce a multi‑reference benchmark.
LLM‑based coding agents can generate tests for a wide range of programming tasks, yet the prevailing evaluation practice—checking a test suite against a single reference solution—fails to account for alternative valid implementations. This oversight lets a test pass even when it misses essential behavior, thereby inflating reported performance.
To address this gap we introduce TestPrism, a benchmark of 300 tasks drawn from 17 sources together with 3000 candidate implementations (half valid, half invalid). Its primary metric, the Joint Success Function (JSF), demands that a generated test suite fail on the initial program state, accept every valid candidate, and reject every invalid one. Under this stricter criterion, fourteen baseline coding‑agent configurations achieve only 28.00 % JSF, whereas single‑reference evaluation reports a much higher 59.67 % success.
**Table 1.** Comparison of TestPrism with existing benchmarks.
The core limitation of single‑reference evaluation is that it cannot guarantee that generated tests generalize to all valid implementations while rejecting incorrect ones.
Constructing the TestPrism Benchmark
How TestPrism builds a multi‑reference benchmark that forces generated tests to accept valid variants and reject defective ones.
Evaluating generated tests against a single reference solution inflates success rates because any test that merely reproduces the reference passes, even if it fails on alternative correct implementations.
TestPrism turns test evaluation into a multi‑reference problem by pairing each test‑generation task with a hidden panel of several valid and several invalid implementations, forcing a test to discriminate correctly.
How does TestPrism differ from simply aggregating multiple reference solutions?
Aggregating references would treat any passing implementation as correct, but TestPrism explicitly includes invalid implementations and requires the generated test to reject them. The panel’s diversity constraints (AST distance for valid variants, Jaccard mismatch for invalid ones) ensure the test cannot succeed by over‑fitting to a single solution.
Screen 17 public benchmarks, keep tasks with executable test provenance, and map them to Harbor’s instruction, environment, tests, and solution components.
Select tasks using Solution Size, Test Count, and Test Density thresholds, yielding 1,700 training tasks and 300 final test tasks.
Generate candidate patches with multiple coding agents, run the gold verifier, and label each patch as valid (all checks pass) or invalid (any check fails).
Apply diversity filtering: enforce a mean AST distance for the four valid alternatives and a Jaccard distance threshold for the five invalid variants, then package the panel into a TestPrism task.
Compute pairwise AST distances among the four valid patches; the mean distance is $0.42$, satisfying the diversity threshold $>0.35$.
For the invalid set, compute Jaccard similarity of failed verifier checks; each pair shares at most $0.2$ of failing checks, meeting the Jaccard threshold.
Assemble the hidden panel: $\{G, V_1,V_2,V_3,V_4, I_1,\dots,I_5\}$ and store it alongside the task’s instruction and entry script.
During test generation, the coding agent sees only the instruction and the gold solution; the hidden panel remains unseen until evaluation.
The panel’s diversity guarantees that a test cannot succeed by merely reproducing the gold solution—it must correctly separate the four valid alternatives from the five invalid ones.
**Figure 1.** Overview of TestPrism construction and evaluation.
Evaluation Metrics and Protocol
Defines the multi‑reference metrics that assess test reliability beyond a single reference.
Evaluating tests against only a single reference solution masks failures, because many correct implementations are ignored. The authors therefore adopt a multi‑reference protocol that records pass/fail outcomes for every candidate patch.
The core metric demands that a test first fails on the reference, then accepts every valid alternative and rejects every invalid one.
Task 1: valid candidates $\{p_{1a}, p_{1b}\}$ both accepted ($a_i=1$), invalid candidate $\{p_{1c}\}$ rejected ($r_i=1$).
Task 2: one valid candidate accepted, one valid candidate rejected ($a_i=0$) → product over valid becomes 0.
Task 3: all invalid candidates rejected, but the reference is not rejected ($r_i(b_i)=0$) → overall contribution 0.
JSF = $\frac{1}{3}\bigl(1\cdot1\cdot1 + 1\cdot0\cdot1 + 0\cdot1\cdot1\bigr) = \frac{1}{3}$.
JSF penalizes any missed acceptance or rejection, so even a single error on a task reduces the overall score.
Measures how often the reference solution passes when it should, averaged over all tasks.
Proportion of valid alternative implementations that the test suite accepts.
Proportion of deliberately invalid candidates that the test suite correctly rejects.
The TestHelix Synthesis Method
TestHelix combines diverse synthesis, peer validation, and self‑improvement to boost test reliability.
Relying on a single reference implementation inflates test reliability metrics because alternative correct solutions are ignored. TestHelix tackles this by generating multiple, mutually‑checking test‑repair pairs.
TestHelix orchestrates three interacting modules—Heterogeneous Synthesis, Peer Cross Validation, and Recursive Self Improvement—to produce diverse, self‑validated test suites.
Agent A runs T₁ on S₀ (fails) and on P₁ (passes); Agent B does the same with T₂, P₂.
Peer Cross Validation executes T₁ against P₂ and T₂ against P₁, revealing that T₁ incorrectly rejects P₂.
Corrective feedback modifies T₁ to satisfy both repairs, while T₂ remains unchanged.
Recursive Self Improvement updates the synthesis strategy based on which pairs succeeded.
The final suite merges the validated pairs, yielding two robust test‑repair pairs within the fixed budget.
This toy walk‑through shows how diversity, cross‑validation, and feedback combine to prune false positives without expanding the number of generated pairs.
Distinct coding agents follow different exploration strategies, each producing a unique test‑repair pair.
Why not just randomize test generation instead of using heterogeneous agents?
Random sampling yields unrelated tests but lacks coordinated exploration; heterogeneous agents deliberately pursue complementary strategies, ensuring the generated pairs cover different semantic aspects rather than producing redundant or trivial samples.
Each generated test is executed against repairs from other agents, revealing over‑restrictive or incomplete tests.
How is Peer Cross Validation different from standard k‑fold cross‑validation?
Standard k‑fold splits a fixed dataset to evaluate a model, while Peer Cross Validation swaps generated tests and repairs across agents, using execution and semantic review to validate each other's outputs rather than partitioning a static dataset.
RSI iteratively refines the test synthesis strategy using execution feedback, keeping the number of pairs and inference budget constant.
Why not simply increase the number of test‑repair pairs instead of using RSI?
Adding more pairs raises computational cost and exceeds the fixed inference budget; RSI improves the quality of existing pairs by learning from their outcomes, achieving higher success rates without extra resources.
High‑level TestHelix workflow
**Figure 1.** The TestHelix architecture, illustrating the workflow of Heterogeneous Synthesis, Recursive Self Improvement, and Peer Cross Validation.
Performance of Coding Agents
RQ1 reveals the limited joint success of frontier models under multi‑reference evaluation.
Claude Fable 5.1 achieves the highest Joint Success Function of 28.00% among all evaluated models.
The Joint Success Function values range from 28.00% (Claude) down to 11.33% (GLM), as shown in the evaluation table.
Five of the six configurations exhibit a higher Invalid Rejection Rate than Valid Acceptance Rate, revealing a systematic difficulty in accepting permissible implementations.
GLM 5.3 attains the highest IRR (86.33%) yet records the lowest RAR (43.00%) and Joint Success Function (11.33%), showing that aggressive defect rejection alone does not guarantee overall test reliability.
The harmonic mean H correlates positively with Joint Success Function across all configurations, and both Change Coverage and Entry Coverage exhibit strong Pearson correlations (r = 0.91 and 0.96) with joint success.
The performance gap between reference‑based and multi‑reference evaluation remains large: the best Joint Success Function is only 28.00%.
Failure Analysis and Patch Diversity
TestPrism evaluates tests against multiple references, exposing failures hidden under single-reference evaluation.
We examined the failures of generated tests by sampling 50 trajectories that failed (joint success = 0) per configuration, discarding 14 timeouts and leaving 286 annotated cases.
Two experts labeled each trajectory with seven error subtypes, achieving strong agreement (Cohen’s $\kappa$ = 0.81), ensuring reliable categorization.
Coverage gaps account for 34.6 % of the observed failures.
Panel (b) of Figure 4 reports 99 out of 286 trajectories as coverage gaps.
Faulty checks are the dominant failure mode, comprising 50.0 % of failures.
Figure 4’s hierarchy lists 143 faulty‑check cases.
Execution‑logic errors represent 15.4 % of failures.
Figure 4 shows 44 trajectories with execution errors.
**Figure 4.** Model profiles across all 300 benchmark tasks (a) and failure mechanisms in 286 sampled failed trajectories after excluding 14 timeouts (b).
Coverage gaps arise when tests miss input combinations that expose defects, e.g., a poker suite never compares an ace‑low straight against three of a kind, allowing a mis‑classifying candidate to pass.
Faulty checks occur when a test exercises relevant behavior but judges it incorrectly, such as a bracket‑parser suite that only verifies output token count, letting an invalid candidate slip through.
Execution‑logic errors stem from bugs in the test code itself, for instance an EvoEval suite that unintentionally binds an extra argument via self, causing the test to fail even on the reference repair.
GPT 5.6 sol‑max and GLM 5.3 more reliably reject invalid candidates than they accept valid alternatives, whereas Qwen 3.8 Max shows the opposite trend.
Evaluating TestHelix Components
How each TestHelix component affects the Joint Success Function under a fixed inference budget.
Table 3 reports five TestHelix ablations—heterogeneous synthesis (HS), peer cross validation (PCV), and recursive self improvement (RSI)—against native coding agents for two models, all under the same total inference cap per benchmark task.
Native agents achieve 54.00 % (GPT 5.6 sol) and 61.33 % (Kimi K3), establishing the baseline that all TestHelix variants must surpass.
Heterogeneous synthesis (HS) raises the Joint Success Function by at most 1 percentage point compared with single‑pair generation.
Table 3 shows HS (K) scores of 86.00 % and 71.60 % versus the single‑pair baseline of 85.00 % and 70.60 % (implicit from the table).
Enabling peer cross validation (PCV) adds more than 6 percentage points to the Joint Success Function.
When PCV is applied (K = 3), $JSF$ rises from the HS‑only level to 92.33 % (GPT 5.6 sol) and 77.67 % (Kimi K3), a gain of >6 %.
Increasing PCV to $K\!=\!5$ yields only a marginal 0.67 percentage‑point improvement.
Table 3 reports the $K\!=\!5$ PCV row with $JSF$ 93.00 % versus the $K\!=\!3$ row at 92.33 %.
Recursive self improvement (RSI) contributes an additional 4.33–4.67 percentage‑point boost to $JSF$.
RSI (K = 3) raises $JSF$ from the PCV‑only level to 96.66 % (GPT 5.6 sol) and 82.34 % (Kimi K3), a gain of roughly 4.5 %.
The full TestHelix configuration (HS + PCV + RSI) improves $JSF$ by 11.67–12.00 percentage points over the single‑pair baseline.
Comparing the final row (HS=3, PCV=check, RSI=check) to the single‑pair baseline yields $JSF$ values of 98.67 % (GPT 5.6 sol) and 73.33 % (Kimi K3), representing an 11.67–12.00 % increase.
Benchmark Comparison Details
Benchmark comparison conventions and coverage targets across related evaluation suites.
This appendix formalizes the comparison criteria used throughout the paper, aligning benchmarks on behavioral and coverage targets.
Only TestPrism satisfies all targets, leveraging 2,700 variant implementations and 300 reference solutions to evaluate the Joint Success Function.
Task Screening and Construction
Describes how tasks are screened, adapted, and selected for the TestPrism benchmark.
We begin by selecting public benchmarks that provide traceable task origins and documented test validation. The SWE bench and Terminal Bench families are kept because they expose public leaderboards with results from multiple agent developers, while other sources such as Exercism and USACO contribute community‑maintained exercises.
Tasks are adapted only if they centre on code and can be executed as tests; any task whose Docker image or environment cannot be reproduced is discarded. For the retained tasks we map the original benchmark’s instruction, environment, tests, and solution components into Harbor’s unified schema, preserving each source’s dependencies, services, and execution limits.
Selection uses three quantitative signals: the number of effective lines in solution.sh (excluding blanks and comments), the count of test functions extracted from source files, and the test‑coverage ratio defined as the test count divided by the number of modified files (minimum denominator = 1). Source‑specific lower bounds are calibrated to each benchmark’s typical distribution, ensuring that smaller‑scale sources are not unfairly pruned.
After normalising and deduplicating tasks, we enforce source quotas that yield a fixed training set of 1,700 tasks and a disjoint candidate test pool. From this pool we later sample a final test set of 300 tasks, preserving proportional representation from every source and using a fixed random seed (42) for reproducibility.
Table 5 shows the filtering funnel: starting from 15,148 source tasks, the pipeline retains 1,700 training tasks and 300 test tasks. The remaining tasks are excluded by source‑quota limits rather than by quality failures. Candidate test tasks then pass verifier checks, an initial panel assembly (requiring one reference, four valid alternatives, and five invalid implementations), and a diversity refinement stage before final admission.
The provided image is a table mapping sections of the "Main text" to their corresponding "Appendix" sections and page numbers.
**Table 5.** Task filtering stages in TestPrism construction.
The table outlines the screening stages for tasks, categorized into "Shared screening" and "Candidate test pool only," detailing the number of items "Retained," "Excluded," and the exclusion "Rate" for each stage.
**Table 6.** Source test provenance and prior validation evidence.
Patch Generation and Evaluation
This section details the patch generation pipeline, benchmark statistics, and evaluation metrics used to assess TestPrism.
The test generation adapter updates task.yaml to declare submission requirements and builds a Dockerfile for the execution environment. The original behavioral contract remains unchanged, and candidate patches are kept for later evaluation. Submitted tests, fixtures, and the entry script live in the designated artifact directory.
Benchmark statistics are summarized across the 300 selected tasks and 17 retained source benchmarks. Figures 5, 6, and 7 visualize the distribution of tasks by source‑test count, by benchmark, and by taxonomy respectively, while Table 8 lists raw counts per benchmark.
**Figure.** Distribution of tasks based on the number of enumerated source tests.
**Figure.** A bar chart illustrating the distribution of 17 benchmarks across 300 tasks, categorized by type: Other benchmarks (blue), Terminal / CLI (green), SWE benchmarks (pink), and USACO (orange).
**Figure 7.** Task taxonomy of TestPrism. Numbers indicate task counts; sector angles and areas are not proportional to these counts.
The table displays counts for various categories: Tasks (300), Reference (300), Other valid (1200), Invalid (1500), Verified files (3000), and Missing (0).
Language‑wise statistics are reported in Table 9, where Python dominates with 205 tasks and 327 627 source tests, while other languages contribute fewer tasks and patches.
**Table 9.** TestPrism test set statistics by language. #Test counts enumerated source tests and verifier checks, including partial counts.
Behavioral metrics are defined per task $i$ using outcome indicators $a_i(p)$ (pass) and $r_i(p)$ (fail). The reference success indicator $F_i = r_i(b_i)\cdot a_i(g_i)$ and the joint success indicator $S_i = F_i\cdot\prod_{p\in V_i} a_i(p)\cdot\prod_{p\in I_i} r_i(p)$ are averaged over $N=300$ tasks to obtain the Reference Acceptance Rate and the Joint Success Function (JSF).
Task‑level acceptance and rejection rates are $A_i = \frac{1}{|V_i|}\sum_{p\in V_i} a_i(p)$ and $R_i = \frac{1}{|I_i|}\sum_{p\in I_i} r_i(p)$. Aggregating over tasks with at least one valid (set $TV$) or invalid (set $TI$) observation yields the Valid Acceptance Rate and Invalid Rejection Rate. Their harmonic mean $H = \frac{2\cdot\text{VAR}\cdot\text{IRR}}{\text{VAR}+\text{IRR}}$ summarizes balanced performance.
Coverage metrics are collected as Change Coverage $\Delta C_i = \frac{|C_i\cap H_i|}{|C_i|}$ and Entry Coverage $E_i = \frac{|E_i\cap H_{gi}|}{|E_i|}$, then averaged across all tasks to obtain $\Delta C$ and Entry Coverage. All baseline configurations have complete observations for these metrics.
Stored fields map to reported quantities: the $f2p\_rate$ field records $F_i$, which contributes to Reference Acceptance Rate; the valid acceptance field includes the reference solution and must be recomputed over the other four valid candidates for VAR; the mutation score mirrors Invalid Rejection Rate. Joint Success Function averages the task resolution indicator $S_i$ over the benchmark.
Additional Strategy Details
Appendix C details the TestHelix strategy, prompts, and validation protocols.
This appendix expands the TestHelix methodology introduced earlier, laying out the prompt templates that drive heterogeneous synthesis, the peer‑cross‑validation workflow that checks generated tests, and the recursive self‑improvement loop that refines strategy fields. It also records the concrete schema tables and the validation experiments that support the final release.
**Table.** Reference success often fails to extend to other implementations. Claude Fable 5.1 leads Joint Success Function at 28.00%, compared with 59.67% against the reference. Five of the six configurations score higher on Invalid Rejection Rate than Valid Acceptance Rate, indicating difficulty accepting permissible implementations. GLM 5.3 has the highest rejection rate but the lowest acceptance and joint success rates: defect rejection alone is insufficient. Across all fourteen configurations, the positive association between H and Joint Success Function supports balancing both requirements. Change Coverage and Entry Coverage also correlate with joint success (Pearson r = 0.91 and 0.96). Reaching relevant behavior helps, but does not ensure that assertions judge it correctly (Inozemtseva & Holmes, 2014; Zhu et al., 1997).
**Figure 3.** Joint Success Function and candidate diversity under three patch addition schedules.
The table lists configuration fields, their default values, and their allowed bounds.
**Table 12.** This table lists the fields or resources and their corresponding values used in the study, including parameters like `maxRounds`, `agentSteps`, `repairSteps`, `contextTokens`, `toolOutputChars`, and `reviewMaxOutputTokens`, alongside various task constraints and execution limits.
**Table 12.** Mutable strategy ranges and fixed inference caps.
**Table 13.** Independent release validation with Opus 4.6. Success counts use 100 paired tasks; confidence intervals describe the Reference Acceptance Rate gain in percentage points (pp).
**Table.** Performance metrics for various coding agent models, including Reference Acceptance Rate (RAR), Candidate metrics (VAR, IRR, H), Coverage metrics ($\Delta$C, EC), and the Joint Success Function (JSF).