Measuring the Gap Between Human and LLM Research Ideas

LLM-generated research ideas are systematically narrower than human ideas, favoring integrative synthesis over diverse gap-framing.

How do the research ideas generated by LLMs differ from those of human researchers when both are given the same literature context?

Research ideation is increasingly automated, but we lack a clear understanding of whether LLMs produce the same breadth of research ideas as human experts. The authors construct a literature-grounded framework that pairs human-written papers with LLM-generated ideas derived from the same prior-work context, then map both to a two-axis taxonomy of research taste. LLM ideas are disproportionately concentrated around "bridge" opportunities and synthesis methods, whereas human ideas span a significantly wider range of gap-framing and contribution strategies.

Paper Primer

The authors define "research taste" as the distribution of problem-framing and contribution-style choices a source makes across many contexts. While individual LLM ideas often appear coherent, the authors hypothesize that the aggregate output of these models is behaviorally narrow, repeatedly relying on the same templates rather than the diverse strategies found in human research.

The method hinges on a two-axis taxonomy: the "opportunity pattern" (why a study is needed) and the "method paradigm" (how the contribution is constructed). The core move is a distributional comparison: the authors reverse-engineer the prior-work context for 11,683 real papers, prompt LLMs to generate ideas from that same context, and measure the divergence between the resulting human and model corpora.

LLM-generated ideas exhibit a systematic distributional shift toward integrative and synthesis-oriented research moves.

Across nine evaluated LLMs, 47.1%–64.2% of ideas are motivated by "bridge" opportunities, compared to only 12.1% of human ideas. The models' reliance on synthesis or unification as a method paradigm is 4–7 times higher than the human baseline (up to 38.7% vs. 5.1%).

Extended reasoning (thinking mode) fails to broaden the distribution of ideas.

Enabling thinking mode consistently increases the concentration of bridge and synthesis moves, further lowering the entropy of the generated idea distribution. For Qwen3-8B, thinking mode increased bridge opportunities from 49.7% to 71.1%.

Why is this distributional view more informative than simply judging individual ideas for novelty or feasibility?

A single idea can appear novel and coherent while still reflecting a narrow, repetitive research taste. By analyzing the distribution, the authors reveal that LLMs are prone to "template-like" ideation, which limits their utility for diverse scientific discovery even when individual outputs seem high-quality.

Does providing models with full-paper context instead of just abstracts bridge the gap?

No. The authors found that richer context does not move model distributions closer to human research taste; instead, it often increases the divergence, as models continue to default to their preferred synthesis-oriented templates.

Current LLM ideation is not just a matter of generating "reasonable" ideas, but of overcoming a systematic bias toward integrative templates. Future research agents must move beyond synthesis-heavy archetypes to match the broader range of gap-framing strategies used by human researchers.

The Research-Taste Gap

We define the Research‑Taste Gap and motivate measuring AI research taste.

LLMs are being deployed to brainstorm research ideas, yet most evaluations focus on single‑idea novelty or feasibility. The open question this paper tackles is: what systematic patterns do LLM‑generated ideas exhibit compared to human researchers, and how far do they lie from the full spectrum of human research taste?

The gap captures the mismatch between the breadth of problem framings and methodological styles humans employ and the narrower, template‑driven patterns LLMs tend to produce.

**Figure 1.** Overview of our research-taste gap analysis. From a shared literature context, humans contribute the paper idea while LLMs generate new ideas from the same prior works. Each idea is decomposed into a *motivation* and a *method*, then annotated with a research-taste taxonomy. Comparing the resulting distributions reveals that LLM ideas are substantially narrower than human ideas, with strong biases toward bridge-like motivations and explicit synthesis methods.

The core motivation for measuring AI research taste is to reveal systematic biases that limit the diversity and usefulness of LLM‑generated research ideas.

Taxonomy of Research Ideas

We detail the evaluation pipeline and annotation scheme for measuring LLM versus human research ideas.

To compare LLM‑generated and human research ideas we construct a controlled ideation task and a systematic annotation pipeline. By anchoring both sources to the same set of prior papers we isolate how each identifies gaps and proposes contributions.

This axis captures *why* a new study is needed by labeling the type of research gap the motivation addresses.

How does the Opportunity Pattern Axis differ from a simple “research problem” label?

A “research problem” label merely states the topic, whereas the Opportunity Pattern Axis explicitly categorizes the *nature* of the gap (e.g., missing explanation vs. resource bottleneck), enabling fine‑grained analysis of what motivates proposals.

This axis captures *how* the proposed work addresses the identified gap by labeling the high‑level contribution strategy.

Why isn’t the Method Paradigm Axis just a “method description”?

Because it abstracts away implementation details and focuses on the *type* of contribution (e.g., “Robustification” vs. “Artifact/System”), enabling comparison across disparate domains where concrete methods differ but high‑level strategies are comparable.

Collect a set of $k$ related prior works ($t_{ij}$ titles, $a_{ij}$ abstracts) for each target idea.

Prompt humans and LLMs to generate a structured idea $(m_i, s_i)$ anchored to the same prior‑work set.

Run the automated annotator to assign primary and secondary labels on both Opportunity Pattern and Method Paradigm axes, plus diagnostic scores.

Validate the annotator on a held‑out subset of 150 papers, measuring Cohen’s $\kappa$ for each axis.

Compute distributional distances (TVD, JSD) between human and LLM label distributions for each axis.

The annotator scans $a_{11}$ and $a_{12}$, detects that neither addresses robustness to noisy graph data, and assigns the primary opportunity label “Bridge Opportunity.”

It then matches the method description to the “Robustification” paradigm, assigning that as the primary method label.

Confidence scores are $0.92$ for the opportunity label and $0.88$ for the method label; diagnostic scores indicate low boilerplate ($0.15$).

This concrete run shows how the annotator maps a textual motivation and method to the two axes, producing the data used for the TVD/JSD comparison.

**Figure 2.** The two-axis research-taste taxonomy. The opportunity pattern axis labels *why* a new study is needed and captures a research gap, while the method paradigm axis labels *how* the proposed work turns that gap into a contribution.

Empirical Comparison of Ideas

LLMs stay far from human research‑idea distributions, especially on opportunity moves.

LLMs remain substantially distant from the human baseline, especially on the opportunity axis.

Table 1 shows the best opportunity‑axis TVD (Claude‑Sonnet‑4.6) at 0.348 versus 0.0 for humans.

The human baseline is the empirical label distribution of 11,683 real research ideas, serving as the target that models should emulate.

Why is high entropy important for the human baseline?

High entropy means human ideas spread across many distinct research moves; a low‑entropy model distribution signals over‑concentration on a few template moves.

These scores measure how much a model’s label distribution must shift to match the human baseline.

How does JSD differ from TVD in what it penalizes?

JSD penalizes both the magnitude of mismatches (like TVD) and the asymmetry of the distributions by averaging them, so it is less sensitive to outliers and always lies between 0 and 1.

An archetype operation is the high‑level verb (e.g., integrate, replace) that characterizes the core action of a research idea.

Why does “integrate” dominate model outputs while “replace” dominates human ideas?

Models tend to select high‑frequency technical nouns from the context and then attach a generic “integrate” verb, whereas humans often target a specific weak component and explicitly “replace” it.

**Figure 3.** Full label distributions over opportunity patterns and method paradigms. On the opportunity axis, LLMs strongly amplify fragmentation and bridge opportunities (orange), where the research gap is framed as disconnected literatures, methods, or evidence streams that should be connected. On the method axis, the largest shift is toward explicit synthesis or unification (blue), where the contribution is constructed by integrating or reconciling separate ideas. Thinking-mode outputs are shown for Qwen3-8B and DeepSeek-V4-Flash.

**Figure 4.** Mechanism analyses for the distributional gap. (A) Circle areas show archetype operation shares by source: model outputs concentrate on reusable template moves such as *integrate* and *unify*, while human ideas retain more local interventions such as *replace*, *decouple*, and *formalize*. (B) Same-paper similarity distributions show that the two model ideas are closer to each other than either is to the human idea. (C) Core concept enrichment ranks clusters by model-vs-human log-odds, with circle area indicating support. Model-heavy concepts are high-frequency technical motifs, whereas human-heavy concepts are narrower mechanism or representation clusters.

Taxonomy Design Principles

Defines the taxonomy used to evaluate research‑taste gaps and lists experimental details.

The taxonomy was built to compare research taste rather than topic, field, or technical substrate. It therefore focuses on why current knowledge is insufficient and what new contribution would address that insufficiency.

We consulted four well‑known proposal‑writing guides—DARPA’s Heilmeier Catechism, NIH application guidance, NSF’s PAPPG instructions, and AHRQ’s research‑gap framework—to inform the label set. These sources motivated the distinction between motivating limitations and proposed solutions.

Opportunity Pattern categories enumerate the kinds of gaps we observe: Puzzle/Contradiction (paradoxes or surprising failures), Explanation Gap (missing causal or theoretical accounts), Scope Mismatch (unrealistic assumptions or unclear transferability), Evidence Gap (lack of measurement or validation), Bridge Opportunity (disconnected literatures that could be linked), Failure/Risk Gap (brittleness, bias, safety concerns), and Resource Bottleneck (cost, compute, data scarcity).

Method Paradigm categories capture the types of methodological moves: Synthesis/Unification (bridging separate bodies of work), Relax/Extend Scope (weakening assumptions or expanding regimes), Robustification (reducing brittleness or bias), and Formal Derivation (introducing a formal model, theorem, or taxonomy).

**Table 6.** Guideline primitives used to ground the research-taste taxonomy. We report the related items extracted from each source and how they informed our taxonomy.

Artifacts used are publicly available paper metadata (titles, abstracts, prior‑work contexts) and the model‑generated ideas with their assigned taxonomy labels. The released dataset preserves source attribution.

The corpus contains only public scholarly text; no private user‑generated content or human‑subject data are included.

Model runs use temperature 0.6, top‑p 0.95, top‑k 20, and a maximum of 2,048 new tokens per input; the API variant runs at temperature 1.0 with a JSON‑schema constraint. Embedding analysis employs Qwen3‑Embedding‑4B (2,560‑dimensional, batch size 12, bfloat16).

Analysis pipelines apply TF‑IDF (lowercasing, stop‑word removal, 1–2 grams) followed by MiniBatchKMeans (k = 30, batch = 512, seed = 13). Linear probes use balanced logistic regression with 5‑fold stratified cross‑validation.

Two authors audited a stratified sample of 150 annotations, judging the primary Opportunity Pattern, primary Method Paradigm, and diagnostic‑score profile against the codebook; disagreements were examined via confusion matrices.

LLMs are employed throughout the pipeline—for idea extraction, proposal generation, annotation, and final polishing—mirroring the uses described in the method and prompt appendices.

Prompting and Annotation Pipeline

Appendix D details the prompt templates used to extract prior work, generate ideas, and annotate research taste.

This appendix records the exact prompts that drive the three‑stage pipeline: extracting proximal prior work, synthesising a novel research idea, and annotating the resulting proposal on the Opportunity‑Pattern and Method‑Paradigm axes.

Prompt: Prior Work Extraction

Prompt: LLM Idea Generation

Prompt: Research‑Taste Annotation

Domain-Specific Results

Domain‑specific and ablation analyses reveal persistent bridge‑opportunity bias.

Bridge‑opportunity labels dominate across both corpora and persist under all ablations.

Claude‑Sonnet‑4.6 achieves the lowest $TVD$ (0.105) in the Opportunity Pattern axis, and Qwen3‑8B’s bridge count rises to 551 in the full‑paper context.

All other label categories (e.g., formal derivation, empirical mapping) show negligible shifts, indicating the pattern is robust to context and prompt changes.

**Table 1.** Comparison of model performance across Opportunity Pattern and Method Paradigm metrics.

**Table 8.** Nature Communications distributional distances against the human distribution. Ent. is normalized entropy. Header arrows indicate the direction closer to the human distribution.

The table presents performance metrics for various AI models compared to human performance across four categories: ML Bridge, ML Synthesis, NC Bridge, and NC Synthesis. The columns represent the different tasks, and the rows list the specific models (e.g., Claude-Sonnet-4.6, Gemini-3.1-Pro, GPT-OSS-20B, Qwen3-8B, DeepSeek-V4-Flash, etc.) and the Human baseline.

**Table.** Comparison of Opportunity Patterns and Method Paradigms between Original and Full context settings for Qwen3-8B and DeepSeek-V4-Flash models.

Discussion and Conclusion

Summarizes the study’s takeaways, its constraints, and future directions.

Recall the central premise: LLMs and humans draw ideas from the same literature, yet LLMs gravitate toward generic “bridge” or synthesis templates rather than the fine‑grained, problem‑specific moves that characterize human contributions.

Our evaluation framework—reconstructing a shared literature context, extracting human ideas, prompting LLMs, and labeling outcomes with a two‑axis research‑taste taxonomy—reveals that LLM‑generated ideas occupy a markedly narrower region of the research‑taste space.

The dominant pattern is an overproduction of bridge‑like opportunities and synthesis‑oriented method paradigms; diagnostic analyses show many outputs are less specific, more boilerplate‑like, and repeatedly follow the same integrate archetype.

Consequently, ideation should be framed as a distributional alignment problem: a useful system must preserve LLM fluency and scale while diversifying problem identification, contribution construction, and departure from familiar synthesis templates.

Our corpus, though broad, is STEM‑centric; other domains (social sciences, humanities, clinical research) may exhibit different research‑taste distributions.

The task reconstructs a local literature context from related works, whereas real researchers also rely on tacit expertise, failed attempts, collaborations, reviewer feedback, and long‑term agendas.

Our taxonomy and annotation pipeline, while human‑validated, compress nuanced ideas into discrete labels; some proposals naturally blend multiple research moves, which the current labeling cannot fully capture.

We evaluated a finite set of models, prompts, and one‑shot settings; interactive agents, domain‑specific systems, retrieval‑heavy pipelines, or prompt interventions could narrow the observed gaps.

Thus, our results should be read as evidence about current LLM ideation under a controlled, literature‑grounded setting, not as a universal claim about all AI‑assisted research workflows.

We thank the U.S. National Science Foundation (award No. 2541654) for partial support.

The following bibliography lists all works cited throughout the paper.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers