Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation

Yifan Zhou, Qihao Yang, Yan Li, Donggang Li, Xiru Hu, Hokin Deng, Ziyang Gong, Xuanyi Zhou, Huacan Wang, Xiangchao Yan, Wanghan Xu, Wenlong Zhang, Shaofeng Zhang, Yue Zhou, Yifan Yang, Zhihang Zhong, Xue Yang

IG-Bench evaluates whether AI research agents can trace and extend the specific mechanism-level lineages of scientific ideas.

Can we quantitatively measure how well AI models understand and generate scientific ideas by treating them as evolving "genomes" rather than static documents?

Current AI research systems generate plausible text, but they often fail to maintain the structural continuity required for genuine scientific progress. They struggle to distinguish between topical proximity and the actual inheritance of core mechanisms, leading to proposals that sound novel but lack coherent descent from prior work. The authors introduce IdeaGene, a framework that represents papers as sets of typed, evidence-grounded "Idea Genome" objects. By aligning these objects across papers using GenomeDiff records, the framework makes idea inheritance, mutation, and recombination explicitly auditable. On the IG-Bench evaluation, the strongest systems reach only 27.3% accuracy on lineage reasoning, revealing a compositional bottleneck where models recover local signals but fail to maintain consistency across a research lineage.

Paper Primer

Scientific lineage competence requires identifying which specific mechanisms persist, mutate, or disappear across papers. IdeaGene operationalizes this by abstracting research into typed objects—such as mechanisms, limitations, and deltas—and using GenomeDiff to map how these objects evolve between a predecessor and a successor.

The benchmark evaluates this through two modes: IG-Exam, a closed-form test of lineage reasoning, and IG-Arena, an open-ended generation task. In IG-Arena, proposals are scored by a Population-Evolution Score (PES): a lineage-conditioned metric that decomposes quality into Heredity (mechanism preservation), Variation (meaningful novelty), and Selection (downstream viability).

LLM-based scientists exhibit a significant compositional bottleneck in lineage reasoning.

The strongest system (GPT-5.5 + Claude Code) achieved only 27.3% exact accuracy on IG-Exam, with performance dropping sharply as tasks moved from simple abstraction to complex lineage verification.

Structured lineage context reshuffles system rankings rather than providing a uniform performance boost.

While lineage context improves PES on average, the gains are driven primarily by Heredity (mechanism grounding) rather than Variation, separating systems that can operationalize lineage evidence from those that merely benefit from more text.

Why is paper-level retrieval insufficient for evaluating scientific progress?

Paper-level relevance is too coarse; two papers can share a research topic without inheriting the same driver mechanism. IdeaGene is necessary because it audits the specific idea structures transmitted across papers, allowing for the detection of coherent descent versus mere topical co-location.

What is the primary difference between the PES metric and standard pairwise ELO rankings?

Pairwise ELO captures surface-level fluency and preference, which can favor polished but lineage-incoherent proposals. PES measures lineage-conditioned insertion quality, penalizing proposals that fail to inherit the correct parent mechanisms or maintain coherence with the existing research population.

The authors conclude that current tool-using scaffolds (like CLI harnesses) amplify retrieval-dependent capabilities but leave compositional reasoning largely unchanged, suggesting that future auto-research systems require explicit compositional verification modules.

Research systems must move beyond generating plausible text toward verifying and extending the inherited mechanisms of scientific lineages; IG-Bench provides the substrate to measure this transition.

Scientific Lineage and AI Benchmarking

We expose the missing lineage evaluation gap and introduce a genome‑centric benchmark to measure scientific reasoning.

Scientific ideas evolve by inheriting mechanisms, repairing known limitations, and recombining prior work, much like biological genomes. Current AI benchmarks, however, evaluate only surface‑level properties such as retrieval quality or fluency, leaving a critical gap: they do not assess whether models can follow this lineage structure, which is essential for genuine scientific reasoning.

Experiments with 14 LLM‑based scientists reveal a compositional bottleneck— the best system attains only 27.3 % exact accuracy on lineage reasoning, and providing structured lineage context reshuffles system rankings rather than uniformly improving performance.

Scientific Lineage captures the evolutionary relationship between ideas, tracking which mechanisms are inherited, which limitations are repaired, and how components are recombined across successive papers.

**Figure 1.** From paper-centric search to genome-centric lineage. A paper-centric view leaves a model with many related papers but no explicit unit for deciding which ideas are inherited, mutated, recombined, or merely co-located. The genome-centric view first extracts Idea Genome objects, then aligns them through GenomeDiff records into populations and lineages, making evolutionary relations explicit enough to evaluate.

The shift from paper‑centric to genome‑centric evaluation is the key to measuring scientific lineage competence.

Evaluation Paradigms in Science AI

Positioning IG‑Bench among existing evaluation paradigms.

IG‑Bench occupies a niche between representation‑focused benchmarks and generation‑only evaluations, addressing the gap where lineage reasoning is required.

Traditional scientific retrieval and QA benchmarks such as LitSearch, SciRepEval, and various embedding or summarization tasks test whether systems can locate or represent relevant papers, but they stop short of assessing idea transmission.

IG‑Bench asks a distinct question: given the relevant work, can a model pinpoint which idea structures are actually inherited across papers?

Recent automated research pipelines automate literature search, hypothesis generation, experimentation, and paper drafting, making lineage‑grounded evaluation increasingly urgent.

Science‑of‑science studies map fields through citation networks and intent classification, yet they typically operate at the paper or sentence level rather than aligning full Idea Genomes.

IG‑Bench leverages citation and temporal cues as candidate evidence, evaluating the GenomeDiff alignment itself as the core evaluative object.

LLM‑as‑judge and Chatbot Arena have popularized scalable model‑judge protocols, while agent benchmarks such as SWE‑bench, GAIA, and $\tau$‑bench assess tool use but not scientific lineage competence.

IG‑Arena extends this methodology, conditioning judgments on genome‑centric lineage packets and neighboring populations to capture lineage reasoning.

**Table 1.** Positioning of IG-Bench relative to adjacent evaluation paradigms.

The IdeaGene Framework

Defines the core objects and alignment process that enable lineage‑aware evaluation.

The IdeaGene framework operationalizes scientific lineage by turning papers into auditable objects and aligning them across versions. This section details the two central constructs—Idea Genome and GenomeDiff—and the pipeline that turns them into measurable evolutionary dynamics.

An Idea Genome is a minimal, evidence‑grounded unit that captures a single functional idea from a paper, making it possible to trace inheritance, mutation, or loss across works.

How does an Idea Genome differ from a generic document embedding?

A document embedding collapses an entire paper into a single dense vector, losing the ability to isolate individual ideas and their evidential provenance. An Idea Genome, by contrast, preserves discrete, typed units with explicit links to the source text, enabling fine‑grained lineage reasoning.

GenomeDiff records how two sets of Idea Genome objects relate, labeling each object as inherited, mutated, lost, novel, or external.

Why can’t we simply compare raw text strings to detect inheritance?

Raw string comparison ignores the functional role of ideas; two sentences may use different wording but convey the same mechanism, while identical phrasing can appear in unrelated contexts. GenomeDiff relies on typed roles and evidence links to capture true semantic inheritance.

Apply the extraction operator $E$ to each paper $p$ to obtain the set $G(p)$ of Idea Genome objects.

For a predecessor–successor pair $(p_s, p_t)$, compute $\Delta_{s\to t}$ to align the two genome sets and label each object.

Classify the resulting alignment according to the Evolutionary Dynamics taxonomy (Mutation, Adaptive Radiation, Hybridization, Speciation, Niche Competition, Isolation).

Feed the aligned records into IG‑Bench’s two evaluation tracks: IG‑Exam (closed‑form lineage understanding) and IG‑Arena (lineage‑grounded idea generation).

Correlate performance on Exam and Arena to assess how well understanding translates into generative capability.

**Figure 2.** IG-Bench evaluation design. IG-Bench converts input papers into a lineage substrate of audited Idea Genome objects and GenomeDiff records, then evaluates two complementary capabilities: IG-Exam for closed-form lineage understanding and IG-Arena for lineage-grounded idea generation. The same substrate supports the correlation between understanding and generation.

**Table 2.** Evolutionary dynamics as operational GenomeDiff criteria. Each dynamics type corresponds to a specific alignment pattern between two Idea Genome objects.

IG-Bench: Dataset and Evaluation

We detail the IG‑Bench dataset, its construction pipeline, and evaluation components IG‑Exam and IG‑Arena.

IG‑Bench operationalizes the Idea Genome framework by offering two complementary evaluation tracks: IG‑Exam for assessing lineage comprehension and IG‑Arena for testing generation of lineage‑grounded scientific ideas.

IG‑Exam presents a set of closed‑form questions that require a model to trace the evolutionary lineage of a scientific idea.

How does IG‑Exam differ from standard multiple‑choice QA benchmarks?

Standard QA asks for factual recall or inference, while IG‑Exam requires reconstructing a chain of scientific innovations—identifying which prior works contributed specific mechanisms and how they were transformed.

IG‑Arena pits models against each other in generating novel scientific proposals that respect the documented lineage of ideas.

Why does IG‑Arena focus on lineage‑grounded generation instead of free‑form text generation?

Free‑form generation lacks a concrete correctness signal; lineage grounding provides a structured target—models must inherit documented mechanisms and demonstrate meaningful extensions, which yields a measurable notion of scientific progress.

Seed collection – domain experts nominate landmark and frontier papers for each of the ten scientific domains.

Trace expansion – citation graphs, semantic similarity search, and manual curation extend each seed into lineage traces of three to seven papers.

Genome extraction & diff alignment – a multi‑pass LLM extracts typed Idea Genome objects; experts audit and align them into pairwise GenomeDiff records with driver and fate annotations.

Benchmark‑level audit – automated schema checks and held‑out annotators verify temporal consistency, answer contracts, anonymization leakage, and overall trace coherence.

Quality assurance involved 50 graduate annotators who validated GenomeDiff labels, stratified IG‑Exam difficulty, and adjudicated IG‑Arena pairwise battles; inter‑annotator agreement on dynamics labels reached 84.7 % before adjudication.

**Table 3.** IG-Exam main challenge: 42 task types, 1,029 instances, four capability axes.

IG-Exam: Lineage Understanding

We detail the IG‑Exam setup, specifying variables, constants, and evaluation metrics.

The IG‑Exam evaluates lineage understanding by presenting a structured set of challenges. Each challenge probes a distinct capability axis while keeping the evaluation protocol uniform.

Sample a test instance $τ$ = ($c$, $x$, $A$, $y$, $m$) from the benchmark.

Feed the anonymized context $x$ to the model and obtain its predicted answer set $A_{\text{pred}}$.

Compare each field of $A_{\text{pred}}$ to the gold target $y$ using exact‑match across all required components.

Mark the instance as correct only if every field matches; otherwise it is a failure.

Aggregate per‑capability‑axis accuracy by averaging over all instances of the same task type.

Step 1: The benchmark presents the tuple $τ$ to the model.

Step 2: The model outputs $A_{\text{pred}}$ = {“field type” = “metabolism”, “driver” = “enzyme X”, “contribution role” = “regulator”, “lineage position” = “ancestor”}.

Step 3: Exact‑match comparison finds all four fields identical to $y$, so the instance is counted as correct.

Step 4: This result contributes a 1 to the accuracy tally for the T1 Genome Abstraction axis.

This concrete walk‑through shows how the strict exact‑match rule penalizes any single‑field error, enforcing full lineage consistency.

IG-Arena: Lineage-Grounded Generation

We describe the IG‑Arena evaluation protocol and introduce the PES metric for lineage‑conditioned proposal quality.

PES judges a generated proposal like a biologist judging a new species: it must inherit traits from its ancestors, bring meaningful novelty, and be viable in its ecosystem.

Select 30 frontier tasks across 10 research domains, yielding 1,260 proposals from 14 LLM‑based scientists.

For each task, generate proposals under three information settings: (i) Question‑only, (ii) Library of unordered paper summaries, (iii) Lineage with ordered Idea Genome objects and GenomeDiff evidence.

Construct the context packet for each proposal: frontier question, ordered lineage $L$, neighboring population $P$, relevant Idea Genome objects, GenomeDiff evidence, and the candidate proposal $x$.

Present the packet to a panel of three model judges (positions randomised) who assign scores $H$, $V$, and $S$ on a 0–100 scale.

Compute the PES for $x$ by averaging the three scores (see the PES formula).

Run pairwise judge battles; fit a Bradley–Terry model to obtain ELO ratings as a complementary preference diagnostic.

Validate metric reliability via inter‑judge agreement (Krippendorff’s $\alpha$ = 0.74) and 80 % human‑judge concordance on pairwise rankings.

Compute the arithmetic mean: $(70 + 40 + 80) / 3 = 63.33$.

Thus $\text{PES}(x \mid L, P) = 63.33$, indicating a moderately strong lineage‑conditioned insertion.

This concrete calculation shows how a proposal can score high on heredity and selection yet be penalised by low variation, pulling the overall PES down.

How does PES differ from a standard accuracy metric used in IG‑Exam?

PES does not evaluate a proposal in isolation; it explicitly conditions on an ordered lineage and a peer population, breaking the score into heredity, variation, and selection. Accuracy in IG‑Exam simply checks whether a generated answer matches a ground‑truth label, ignoring any evolutionary context.

IG-Exam Performance Analysis

IG-Exam results reveal accuracy drops across lineage tasks and show tool‑scaffolding benefits.

Scientific ideas evolve through lineage, yet existing benchmarks fail to capture this process, leaving models weak at genuine scientific reasoning.

CLI harnesses raise exact IG‑Exam accuracy, especially on the Inheritance Tracing tier.

The strongest direct LLM (GPT‑5.5) scores 23.1 % while the best harness (GPT‑5.5 + Claude Code) reaches 27.3 %.

Tool scaffolding mainly aids retrieval; it boosts T2 performance but offers little gain for compositional consistency on T3 and T4. Research agents remain close to the direct LLM baseline on most axes.

**Figure 9.** IG-Exam exact-accuracy heatmap. Exact accuracy across T1–T4 for all evaluated systems, with verification remaining the hardest axis.

**Figure 10** IG-Exam failure flow. Sankey diagram tracing errors from capability tier (T1–T4) through failed field family to error class (E1–E9). T3 and T4 errors dominate in volume. The thickest flows connect T3/T4 to dynamics and driver fields, which then fan out to E1 (dynamics misclassification) and E2 (driver misidentification). Genome-fate and relation errors (E3–E4) are the next largest classes, confirming that compositional field interactions—not isolated label failures—drive the accuracy bottleneck.

IG-Arena Generation Results

Lineage context separates systems, revealing a median PES gain of +4.4 points.

Lineage context separates systems, with weaker models gaining larger PES improvements.

GPT‑5.5 gains +2.3 PES, Kimi‑K2‑Thinking gains +6.9 PES, and the median gain across 14 systems is +4.4.

The PES gain is driven almost entirely by Heredity; Variation (82.7–84.7) and Selection (79.7–82.2) remain flat across settings.

Dynamics labels only become informative when tied to lineage: Question‑only prompts yield a mix of Mutation, Adaptive Radiation, and Hybridization, whereas Library and Lineage prompts collapse onto Hybridization.

PES rankings diverge from ELO rankings (Spearman $\rho$ = 0.82), especially when a fluent but lineage‑incoherent proposal beats a better‑grounded one, underscoring PES as the primary metric for lineage‑grounded insertion.

**Figure 5. Domain-level PES snapshot.** Lineage-setting PES across 10 domains for the evaluated systems. Strong systems remain relatively uniform, while weaker systems show sharper domain-specific failures.

**Figure 6.** **PES decomposition and information-setting breakdown.** (a) Heredity, Variation, and Selection sub-scores under the genome-centric scoring packet. Variation consistently exceeds Heredity across systems, confirming the plausibility–coherence gap identified in Finding 3. (b) PES under Question, Library, and Lineage settings. Question→Lineage gains are heterogeneous: GPT-5.5 moves only from 85.2 to 87.5, while weaker systems gain more from explicit lineage context.

**Figure 7. Generated dynamics distribution by information setting.** Stacked proportions of post-hoc dynamics labels for generated IG-Arena proposals. The main text replaces this pure frequency view with a quality map because the same dynamics label can have different Heredity and PES depending on whether lineage evidence is available.

**Figure 8. Per-system H/V/S bar chart.** Lineage-setting Heredity, Variation, and Selection for all evaluated systems, sorted by aggregate Lineage PES. Variation consistently exceeds Heredity, confirming the plausibility–coherence gap across the full participant set.

**Figure 3. IG-Arena PES analysis.** (a) Question–Library–Lineage trajectories show that structured lineage context improves generation quality unevenly across systems. (b) PES decomposed by setting–dynamics group, sorted by PES ascending. Only groups with $n \ge 15$ are shown: Question produces four dynamics modes while Library and Lineage concentrate on Hybridization ($\ge 97\%$). Variation and Selection are nearly constant across all rows, while Heredity alone explains the PES gap—from 61.9 (Question|Mutation) to 84.2 (Lineage|Hybrid).

**Figure 4. Capability profile and understanding–generation bridge.** (a) Radar plot over five capability dimensions for five GPT-5.5-backbone systems—the direct LLM, two research agents (AI Scientist v2, CoI-Agent), and two CLI harnesses (Codex, Claude Code)—reducing backbone confounds. Agents nearly overlap with the direct LLM, while CLI harnesses redistribute strengths rather than uniformly improving every capability. (b) Systems with stronger closed-form lineage understanding tend to preserve lineage-setting heredity better during generation; the moderate correlation shows the two evaluations remain complementary.

Technical Appendices

Appendix details task inventories, example prompts, and additional analysis for IG‑Bench.

Table 5 lists the full IG‑Exam challenge inventory, covering 42 task types across four capability dimensions (T1–T4) and totaling 1,029 instances.

**Table.** Distribution of domains and task types.

The contribution‑role matrix enumerates specific reconstruction and grouping tasks such as Five‑Genome Lineage Reconstruction and Genome Alignment.

The dynamics matrix lists single‑step and multi‑step tasks, including Genome‑Field Fate, Driver Dynamics, and Multi‑Hop Genome Tracking.

The consistency‑check matrix defines validation tasks like Domain Intruder Detection and Next‑Hop Prediction.

Selected counts per task type are provided (e.g., N values ranging from 18 to 30) to illustrate instance distribution.

**Table 6.** Active IG-Arena task inventory for the main run: 30 tasks balanced across 10 scientific domains.

Each domain (Computer Science, Biology, etc.) contributes three tasks, yielding a balanced benchmark across disciplines.

IG‑Exam example T1 demonstrates a Genome Abstraction question where the correct answer is “method”.

IG‑Exam example T2 illustrates Inheritance Tracing, requiring partitioning eight Idea Genome objects into two lineage groups.

IG‑Exam example T3 focuses on Evolutionary Dynamics, asking to identify the primary dynamic (answer: Mutation).

IG‑Exam example T4 tests Structural Integrity by selecting the missing‑link paper (answer: Y).

IG‑Arena physics example evaluates two fusion‑control ideas, with the judges preferring the uncertainty‑aware foundation model.

IG‑Arena biology example compares protein‑binder designs, with the causal multi‑objective approach (Idea A) winning.

Evaluation details note that token‑budget caps were increased to 16,384 for GPT‑5.5 reruns to avoid empty completions.

Human agreement was measured with 50 graduate annotators, achieving 80 % agreement on IG‑Arena pairwise battles.

Additional IG‑Arena analysis includes domain‑level PES, PES decomposition, dynamics distribution, and per‑system H/V/S bar charts.

Further IG‑Exam analysis provides a full leaderboard heatmap and an error‑flow Sankey diagram highlighting failure sources.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers