False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agents
Meijia Chen, Hao Li, Zheng Lu, Hongshan Lin, Junbai Tian, Yichen Liu, Zijun Tian, Yufan Zou, Shuhan Sun, Hanxin Chen, Zeyu Zhang, Weizhi Du, Yueting Li, Tianyu Shi, Alaa Khamis
Self-evolving search agents often fall into "co-cheating," where proposer and solver reinforce shared errors; cross-fitting feedback breaks this loop.
How can we prevent self-evolving search agents from "co-cheating," where the proposer and solver learn to agree on incorrect answers to maximize training rewards?
Self-evolving search agents generate their own training data by having a proposer create questions and a solver answer them. This closed loop often leads to "co-cheating," where both agents converge on the same incorrect answers, causing internal reward to rise while external correctness stagnates. CrossFit mitigates this by partitioning source documents into two folds and using auxiliary solvers trained on one fold to evaluate questions from the other. This ensures the feedback signal is never derived from the same source-document errors that the proposer is currently generating. This intervention significantly reduces false agreement and improves downstream search performance by up to 8.8 percentage points over standard coupled self-evolution.
Paper Primer
The core issue is a feedback loop: a solver trained on pseudo-labels from a source document can later "verify" new, incorrect questions derived from that same document. CrossFit is a source-level partitioning strategy: it assigns each source document to one of two folds, ensuring that the auxiliary solver providing reward feedback has never seen the pseudo-labels generated from that specific source.
CrossFit significantly reduces false agreement and boosts downstream search accuracy.
Across seven benchmarks, CrossFit improved average Cover-EM by 8.8 and 8.4 points for Qwen3.5-4B and 9B models, respectively, compared to standard coupled self-evolution. False-agreement mass dropped from 6.1% to 3.0% (4B) and 8.8% to 3.7% (9B).
Verification alone is insufficient to stop co-cheating.
Multi-sample verification (MSV) only improved average performance by 0.7–0.8 points, suggesting that feedback provenance is more critical than individual label reliability. Combined MSV and CrossFit yielded only a 0.3-point gain over CrossFit alone.
Why does the paper use source-level partitioning instead of just splitting individual questions?
Splitting individual questions is ineffective because related questions derived from the same source document can still end up in both folds, preserving the feedback path that allows the solver to "recognize" and reward its own source-based errors.
Does CrossFit require a completely new training architecture?
No; it only changes the feedback mechanism. The main solver continues to train on all admitted questions, while the auxiliary solvers are used solely to provide a cleaner reward signal to the proposer.
The Co-Cheating Failure Mode
Self‑evolving agents generate their own curriculum but fall into a self‑reinforcing error loop.
When a proposer and a solver are trained together, their mutual feedback can turn errors into a self‑reinforcing signal, causing the training reward to rise while true performance stalls.
**Figure 1.** **Co-cheating.** The training signal and false agreement rise together.
In a closed proposer‑solver loop, the two models start to “help” each other by reproducing the same mistakes, which inflates the internal reward without any real gain in correctness.
The proposer and solver are tightly linked: the proposer’s generated data feeds the solver, and the solver’s performance directly shapes the proposer’s future proposals.
The closed‑loop nature of self‑evolving agents creates a perverse incentive for models to agree on errors.
Mitigating Co-Cheating with CrossFit
Two interventions break the self‑reinforcing loop: MSV filters proposals, CrossFit reshuffles feedback.
The closed‑loop of proposer and solver quickly collapses into “co‑cheating”: an incorrect pseudo‑label trains the solver, the solver reproduces the error, and the proposer is rewarded for the same mistake. To stop this self‑reinforcing cycle we intervene at two points – before a proposal enters training (MSV) and when the proposer receives its reward signal (CrossFit).
MSV is a spell‑checker for a draft question: before the draft ever reaches training, the model generates several answers with and without seeing the source document, and only if the two majority votes agree does the draft get admitted.
Source‑aware majority $\text{Maj}$ picks $y_{\text{src}}=1998$ (two votes for 1998).
Source‑blind majority picks $y_{\text{blind}}=1998$ (two votes for 1998).
Since $y_{\text{src}}=y_{\text{blind}}=1998$, $\text{IMSV}=1$ and the draft is admitted with label 1998.
MSV forces the answer to be consistent whether the model sees the source or not, catching many drafts that would otherwise be accepted on a single noisy view.
Why not just use a single majority vote on source‑aware answers and skip the source‑blind check?
Because source‑aware answers can agree for the wrong reason (e.g., copying a spurious pattern from the document). The source‑blind view forces the answer to be justified without that context, exposing hallucinations that would otherwise pass the single vote.
CrossFit treats the two auxiliary solvers like a pair of peer reviewers: each reviewer grades the other’s essays, so no author ever grades their own work.
In round 2, a new question from Doc A is evaluated by Solver 1, which has never been trained on Doc A’s pseudo‑labels.
Solver 1 generates five answers $z_1\ldots z_5$; suppose three of them match the adopted label $\tilde{y}=42$.
The indicator sum is $k=3$, so the reward is $f(3)=\frac{5-3}{4}=0.5$.
The proposer receives this reward and updates its policy; the main solver still trains on all admitted questions from both folds.
Even though the auxiliary solver may still make mistakes, the fact that it never evaluates its own source prevents the self‑reinforcing “co‑cheating” cycle.
How does CrossFit differ from ordinary k‑fold cross‑validation, which also trains on one fold and evaluates on another?
CrossFit’s folds are fixed at the *source document* level, not at the individual question level, so related questions from the same document never end up on opposite sides. Ordinary cross‑validation would split questions arbitrarily, allowing a solver to be evaluated on a question whose pseudo‑label originated from the same source, which would not break the co‑cheating path.
Assign each source document to fold 0 or fold 1 (once for the whole training run).
Train auxiliary solver 0 on all admitted questions from fold 0; train solver 1 on all admitted questions from fold 1.
When evaluating a new question, look up its source fold $h$ and query the *opposite* solver $S_{r,1-h}$ for five answers.
Count how many of the five answers agree with the adopted label $\tilde{y}$ and compute the reward $R_P(q)=f(k)$.
Feed the reward back to the proposer; the main solver continues to train on the union of admitted questions from both folds.
**Figure 2.** Co-cheating and its mitigation. (a) Incorrect agreement creates false frontier credit. (b) MSV verifies each proposal with source-aware and source-blind samples. (c) CrossFit scores each source group with a solver trained on the other group. The two interventions target label quality and feedback provenance, respectively.
**Figure 4.** The CrossFit algorithm. Auxiliary solvers train on one source fold and score the other; the main solver trains on all admitted questions.
Table 1 (see the paper) later demonstrates that MSV improves label quality and that CrossFit dramatically reduces the false‑agreement mass, confirming that the two interventions together break the co‑cheating cycle.
Experimental Results
CrossFit and MSV + CrossFit achieve the highest Cover‑EM scores across all benchmarks.
CrossFit attains the highest Cover‑EM scores, 0.488 on Qwen3.5‑4B and 0.512 on Qwen3.5‑9B.
Table 1 reports these values, surpassing Dr. Zero and Search‑R1.
MSV alone yields only a modest 0.7–0.8 pp increase, while MSV + CrossFit reaches 0.491 (4B) and 0.515 (9B), merely 0.3 pp above CrossFit. Gains are largest on multi‑hop tasks (+10.0 pp / +10.9 pp) and smaller on single‑hop tasks (+7.3 pp / +5.2 pp).
**Figure 5.** Training dynamics across three rounds. (a,b) Round means of agreement and solver truth; hollow, light, and solid markers denote rounds 1–3. Above the diagonal, agreement is optimistic. (c,d) Faint points retain all 129 steps; solid segments average each round's 43 step rates, and end labels give round-3 means (%). Gray bands mark proposer phases, short dashes mark round boundaries, and the long dash marks step 62. Full traces and coverage appear in Figures 8–9.
**Figure 6.** Round-3 audit relative to Dr. Zero. (a,b) Joint truth gains; arrows lead to the combined treatment. (c) Reduction $F_{\text{Dr. Zero}} - F$. All values are percentage points.
Related Work and Context
We situate our approach among prior self‑play methods and outline its broader impact.
Related work spans self‑generated curricula, proxy‑reward hacking, and data‑exclusion techniques, each probing different facets of self‑evolving agents.
Dr. Zero is a baseline self‑evolving agent that alternates between proposing tasks and learning from its own solutions without explicit safeguards.
Works that use self‑play or self‑questioning to create curricula, enabling agents to discover goals, align language models, and improve reasoning without external supervision.
Literature on reward hacking and proxy‑reward optimization, showing how agents can exploit imperfect objectives and favor agreement over truth.
Approaches that enforce independence between training signals and evaluation, such as cross‑fitting, retrieval‑augmented generation, and iterative search.
In conclusion, we isolate the co‑cheating failure mode and show that CrossFit mitigates it by scoring each source with an auxiliary solver trained on the complementary fold, while the main solver still learns from all admitted tasks.
Ablation Studies and Dynamics
Self‑evolving agents can co‑cheat; CrossFit forces solvers to validate on unseen data.
The paper’s core premise is that self‑evolving agents can fall into a “co‑cheating” loop, and CrossFit breaks this loop by forcing solvers to evaluate proposals on data they did not help generate.
We now examine four ablations: the effect of CrossFit itself, the necessity of source‑level exclusion, the role of a fixed replay bank, and the interaction between verification (MSV) and CrossFit.
CrossFit cuts false‑agreement mass by roughly 50 % compared with the coupled baseline.
Figure 8 shows the false‑agreement curves dropping from ~6 % to ~3 % for the 4B model and from ~9 % to ~4 % for the 9B model.
Source‑ID exclusion drives false agreement down to 0.004, far below the 0.064 observed with a same‑source auxiliary solver.
Figure 8 (right column) reports a false‑agreement mass of 0.004 for the source‑ID treatment.
CrossFit improves the downstream metric by 8.8 points for the 4B model.
Figure 8’s “downstream advantage” panel shows an 8.8‑point gain after round 3.
CrossFit improves the downstream metric by 8.4 points for the 9B model.
Figure 9’s corresponding panel records an 8.4‑point gain after round 3.
Source‑ID exclusion raises accuracy to 91.5 % for the 4B model, versus 88.1 % for the coupled baseline.
Figure 7 (bottom‑right panel) lists 91.5 % accuracy for the source‑ID treatment.
Source‑ID exclusion raises accuracy to 91.7 % for the 9B model, versus 87.0 % for the coupled baseline.
Figure 7 (bottom‑right panel) lists 91.7 % accuracy for the source‑ID treatment.
**Figure 7.** Mechanism ablations on a fixed replay bank. (a) Truth versus false agreement; arrows compare coupled and source-ID feedback. (b) Accuracy on 3,000 replay questions, mean ± SD over five seeds. Numbers identify methods; shading marks source exclusion.
**Figure 8.** **Complete Qwen3.5-4B audit trajectories.** Columns identify treatments. Rows show adopted-label truth $T_P$, solver truth $T_S$, and agreement $A$; false-agreement mass $F$; lost-credit mass $L$; and coverage $J/E$. Every metric is expressed in percent. Shared limits support comparison across treatments and model scales. Gray bands mark proposer phases, short dashes mark round boundaries, and the long dash marks step 62.
**Figure 9.** Complete Qwen3.5-9B audit trajectories. Layout, metric colors, units, and axis limits match Figure 8. Every scheduled step is retained. The separate coverage strip prevents audit coverage from obscuring the correctness curves.
Implementation and Configuration
Implementation details enumerate training configs, resource costs, and evaluation metrics for each treatment.
This section details the concrete training schedule, the resource impact of each intervention, and the metrics used to evaluate performance.
The table presents a comparison of different treatments across several metrics: H200-hours (for 4B and 9B models), Tokens (Input and Output in millions), Judge requirements (in thousands), and Hours (for 4B and 9B models).
**Table 3.** Resource cost per training run. H200-hours are reserved budgets: each run holds eight H200 GPUs for its end-to-end duration, including waiting time, so they exceed the accelerator time actually used; the GPU usage of the external audit service is unknown and not included. Token (millions) and judge-request (thousands) counts are per single-scale run, not summed over the two scales, and exclude training-replay tokens. Hours are the end-to-end wall-clock duration of the full pipeline. “25 total” and “25 per fold” denote the auxiliary-update budget per round.
MSV contributes six additional labeler generations per proposal, while CrossFit inserts 50 auxiliary‑solver updates per round (25 per fold), increasing the overall compute budget without altering the main‑solver update count.
Downstream evaluation extracts the final answer, normalizes it, and computes Cover‑EM; micro and macro scores aggregate these per‑question results across benchmarks.
The independent audit saves the source document, the pseudo‑label, and the five solver responses for each step, allowing a post‑hoc verifier to assess evidence quality without influencing training.
Broader Literature Map
We map related retrieval and search methods, present benchmark tables, and discuss limitations of CrossFit.
The map separates complementary research questions rather than treating every cited system as a direct experimental baseline. Only methods with accessible implementations and matched protocols can support comparative performance claims.
Dense retrieval, fusion‑in‑decoder, retrieval‑enhanced language modeling, and few‑shot retrieval pretraining study how evidence is retrieved and represented. Query rewriting and active retrieval change when and how evidence is requested.
Black‑box retrieval augmentation, hierarchical retrieval, and chain‑of‑thought processing provide other evidence interfaces. Robustness to irrelevant context, corrective retrieval, and retrieval‑aware tuning address evidence quality or utilization.
Table 5 reports Cover‑EM of every round‑end main solver, bolding the best performance within each scale. Each cross‑fitted treatment shares round 1 with its coupled counterpart.
Table 6 shows micro‑averaged Cover‑EM for Qwen3.5‑4B and Qwen3.5‑9B across rounds, with the highest scores highlighted in bold.
These directions motivate holding the retrieval backend fixed: a change in evidence access must not be mistaken for an effect of source‑excluded feedback.
Search‑agent optimization literature includes evolving search, efficient search training, simulated search, deep‑research agents, and tool‑use frameworks, providing contextual references without direct evaluation claims.
Self‑generated learning and reinforcement‑learning methods (self‑play, multi‑agent games, etc.) are cited for context; the study focuses on the ancestry of the policy supplying proposal feedback rather than a new gradient estimator.
Discussion notes that CrossFit changes who supplies feedback, not what makes an answer true. Source exclusion is valuable only if checkpoint ancestry and data routing enforce it.
Lower false‑agreement mass may stem from rejecting difficult tasks rather than improving learning. Coverage, task difficulty, fixed‑probe performance, and downstream capability must accompany the reported numbers.
Results establish empirical mitigation across two model scales, but end‑to‑end cost and robustness to connected sources remain open questions.
Round summaries are arithmetic means of the displayed step rates, not additional runs or uncertainty estimates.