ResearchStudio-Idea: an Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes

Qihao Zhao, Yangyu Huang, Yalun Dai, Lingao Xiao, Jianjun Gao, Xin Zhang, Wenshan Wu, Scarlett Li, Yang He, Yan Lu, Yap Kim Hui

ResearchStudio-Idea operationalizes 15 reusable ideation patterns from ML conference outcomes into an end-to-end research-ideation skill suite.

How can we systematically extract and apply successful research-ideation patterns from top-tier ML conference papers to improve the quality of AI-generated research ideas?

Research agents can generate plausible proposals, but they struggle to ground ideas in literature, identify meaningful bottlenecks, and differentiate from prior art before implementation. This "first-mile" gap often leads to ideas that are fluent but lack scientific utility. ResearchStudio-Idea addresses this by inducing 15 reusable ideation patterns from 1,947 machine learning conference papers. The system packages these patterns into an end-to-end skill, IdeaSpark, which performs evidence-grounded generation, collision checking, and outcome-informed auditing. Blind automated-judge evaluations show that IdeaSpark consistently produces stronger research proposals than generic baselines while maintaining competitive novelty.

Paper Primer

The core mechanism is the non-parametric, data-induced pattern card. IdeaSpark treats ideation as a structured workflow: it maps a research problem to a specific ideation pattern, then uses that pattern's card—which contains success conditions and failure modes distilled from accepted and rejected papers—to guide generation and audit the candidate against prior art.

IdeaSpark improves the quality of generated research proposals over no-skill and generic-skill baselines.

Blind automated-judge evaluations of generated idea cards. Consistent improvement in proposal strength while maintaining competitive novelty.

Rejected papers do not occupy a separate strategy space from accepted papers.

Reject-only re-clustering maps every rejected-paper cluster back onto the existing 15-pattern taxonomy. 100% mapping coverage, indicating that failure is a matter of execution and boundary-case handling rather than a distinct negative strategy class.

Why is this system focused on "ideation patterns" rather than just training a model to predict paper acceptance?

The authors argue that acceptance prediction is a weak proxy for useful ideation. By inducing procedural patterns—how a gap connects to a method—the system provides actionable guidance for researchers to organize a problem and audit its distance from prior art before experiments begin.

What is the scope of the IdeaSpark skill suite?

It is scoped strictly to the "first mile" of research: grounding evidence, generating one defensible direction, and auditing it. It does not automate implementation, human peer review, or program-committee selection.

Research Ideation as a Skill

LLM agents ease idea generation, yet turning vague notions into defensible research proposals remains a bottleneck.

LLM‑based research agents make idea generation cheap, but turning a vague notion into a defensible research direction remains hard. The bottleneck is organizing early‑stage work into reusable skills that ground evidence, propose a single viable direction, and audit its novelty before any experiment. ResearchStudio‑Idea tackles this gap by inducing a taxonomy of 15 ideation patterns from 1,947 conference papers and using them to structure LLM‑driven idea generation (the central premise of the paper).

Researchers need a systematic way to convert literature evidence into a concrete, auditable research proposal.

The shift from generative ideation to evidence‑grounded ideation.

Prior Work in LLM Ideation

Survey of recent LLM-driven research ideation approaches.

End-to-end “AI scientist” systems aim to automate the full research lifecycle. AI Scientist drafts, implements, evaluates, and writes up ideas, but its output is often incremental; AI‑Researcher adds structured architectures and decoupled pipelines, while Agent Laboratory starts from a human‑supplied idea and produces a report with variable human‑in‑the‑loop control. Idea2Plan bridges generation and planning via ReAct‑style scaffolding, and later systems such as Sakana AI Scientist v2, Google AI co‑scientist, and Auto Research extend this direction with evolutionary search, multi‑agent frameworks, and full‑pipeline coverage.

Critiques of full‑lifecycle autonomy highlight recurring failure modes such as training‑default bias and long‑horizon context degradation. Trehan et al. distill six failure modes from autonomous ML‑paper attempts, and Bisht et al. argue that current agentic scientists suffer from problem‑selection bias and missing tacit laboratory knowledge. These observations motivate a narrower scope that improves ideas before downstream execution, as IdeaSpark does.

Multi‑agent and search‑based approaches focus directly on ideation. VirSci and IRIS model scientific teamwork with role‑separated agents; Deep Ideation adopts an explore‑expand‑evolve workflow, and Nova adds iterative planning with external knowledge retrieval. FlowPIE treats generation as a test‑time evolution driven by Monte Carlo Tree Search, while the multi‑workflow benchmark compares reflection‑based refinement, Sakana‑style evolution, and other long‑context workflows, finding decomposition‑based methods achieve the highest novelty scores.

Pattern‑induction work extracts reusable structures from top‑conference papers. Liu et al. annotate oral papers with a 12‑category taxonomy; MoRI extracts motivation→innovation pattern pairs from ICLR accepted papers; MotivGraph‑SoIQ builds a motivational knowledge graph paired with Socratic dialogue; and Navigating Ideation Space decomposes ideas into conceptual representations. Parallel efforts ground generation in large literature graphs: SciMuse constructs a graph from 58 million papers, KG‑grounded hypothesis generation retrieves sub‑structures for biomedical hypotheses, and Graphs of Research fine‑tunes LLMs on citation‑evolution DAGs.

Novelty‑focused benchmarks evaluate ideas rather than generate them. NovBench provides a four‑dimensional framework for relevance, correctness, coverage, and clarity; RINoBench offers expert‑judged novelty judgments; and OpenNovelty produces evidence‑based novelty reports for thousands of submissions. Complementary evaluation suites such as IdeaBench, AI Idea Bench 2025, and ResearchBench decompose discovery into retrieval, hypothesis composition, and ranking, while studies like Si et al. show LLM‑generated ideas are more novel but less feasible than expert ideas.

Recent surveys map the broader landscape. “From Automation to Autonomy” proposes a Tool/Analyst/Scientist taxonomy, while “Towards Scientific Intelligence” surveys architectures, benchmarks, and ethical considerations. Anthropic’s skill‑authoring practices inspire IdeaSpark’s two‑tier design, emphasizing progressive disclosure and deterministic routines for repetitive steps.

IdeaSpark differs by binding pattern induction, acceptance contrast, and retrieval‑grounded generation onto a single non‑parametric pattern card. The card serves both as a move selector and an audit artifact, encoding failure signals from an Oral/HC/Reject contrast that pure pattern‑induction or knowledge‑graph methods lack. By stopping at the reviewer‑facing idea card, IdeaSpark targets the upstream bottleneck that many downstream failures stem from.

Corpus and Data Pipeline

We assemble a labeled corpus of 1,947 papers, defining three contrastive tags for pattern mining.

We harvest papers from ICLR, ICML, and NeurIPS (2021–2025) via the OpenReview API and Semantic Scholar metadata, retaining title, abstract, authors, decision, review fields, and citation count. From this raw pool we derive three orthogonal labels—Oral, High‑Cited ($HC$), and Reject—to create a balanced contrast for downstream pattern induction.

**Figure 2.** IdeaSpark data-to-skill workflow. The upper band constructs reusable ideation assets from the 1,947-paper ICLR / ICML / NeurIPS corpus: papers are outcome-labeled, normalized into strategy signatures, mined into 31 sub-patterns, and induced into 15 operational pattern cards. The lower band shows how the skill uses those cards at inference time: evidence grounding and full-text retrieval feed a staged reasoning loop for bottleneck diagnosis, pattern-guided candidate generation, and collision/audit verdicts, followed by expansion, validation, and idea-card deliverables.

Three complementary tags capture success (Oral), community impact (High‑Cited), and explicit failure (Reject), giving the pattern miner both positive and negative signals.

P1 receives the Oral tag only.

P2 receives the $HC$ tag only.

P3 receives both Oral and $HC$ tags, illustrating overlap.

P4 receives the Reject tag, providing a negative example.

P5 remains unlabeled, showing that not every paper fits the contrast.

The three‑label system yields a balanced training set where success, popularity, and failure are all represented, enabling the downstream miner to learn discriminative ideation patterns.

How does this tri‑label scheme differ from a simple binary “accepted vs. rejected” labeling?

Binary acceptance conflates reviewer preference with community impact and provides no explicit negative signal. The tri‑label scheme separates program‑committee endorsement (Oral), long‑term influence (High‑Cited), and concrete failure (Reject), allowing the pattern induction to reason about distinct success dimensions and to learn from genuine rejections.

After deduplication by OpenReview ID we retain 1,947 unique papers. All contain the eight base fields; 1,891 also have at least one of the four abstracted fields (abstract, parsed reviews, introduction, meta‑review). The remaining 56 lack abstracted content and are excluded from the embedding pool. Field‑level coverage is: abstract (1,074 papers), parsed reviews (716), extracted introductions (682), and meta‑reviews (340).

Extracting Innovation Signatures

Extract a compact, strategy‑only vector from each paper to enable meaningful clustering.

Embedding raw paper text tends to latch onto domain nouns (“Transformer”, “molecule”), which produces topic clusters rather than the desired strategy clusters. The two‑stage Innovation Signature extraction removes that failure mode.

We turn a paper into a twelve‑field vector and then strip away domain nouns so only the strategic essence remains for clustering — like taking a color photograph and converting it to a grayscale sketch that preserves shape but discards hue.

Stage 1 LLM returns the eight base fields; the first four are: (1) innovation approach = “Accelerate GNN inference by pruning edges”, (2) key step = “Prune edges before message passing”, (3) why non obvious = “Edge importance is hidden in the graph topology”, (4) trigger condition = “When graph size > 10⁴ nodes, prune edges to achieve sub‑linear scaling”.

Stage 2 rewrites these into abstract forms: (1) abstract strategy = “Accelerate a processing pipeline by removing redundant connections”, (2) abstract key step = “Remove connections before the main computation”, (3) abstract why non obvious = “Redundant connections are not obvious without structural analysis”, (4) abstract trigger condition = “When data size exceeds a threshold, drop superfluous links to meet a speed target”.

The four abstract fields are vectorised (e.g., using a sentence encoder) and concatenated into a 4‑dimensional Innovation Signature for clustering.

The example shows how concrete domain terms disappear while the underlying strategic move (“prune before compute”) stays intact, enabling papers from different domains to be grouped together.

How does this differ from simply embedding the raw abstract and introduction?

Raw embeddings retain domain‑specific nouns, so papers about “Transformer” and “diffusion” cluster together even if they use completely different reasoning strategies. The two‑stage process first extracts a structured description of the reasoning, then rewrites it into a domain‑agnostic form, guaranteeing that clustering is driven by the *strategy* rather than the *topic*.

Unsupervised Pattern Discovery

We uncover unsupervised ideation patterns by embedding, reducing, and clustering paper strategies.

The corpus contains thousands of papers with heterogeneous strategy descriptions, so recurring ideation moves are hidden in plain sight.

We embed each paper’s four abstracted strategy fields, compress the embeddings, and let a density‑based clusterer reveal natural groups — the trick is to let geometry, not labels, define the patterns.

How does this clustering differ from conventional topic modeling?

Topic models group documents by word co‑occurrence statistics, whereas our pipeline groups papers by the geometry of their strategy embeddings. The density‑based HDBSCAN step also tolerates irregular cluster shapes and leaves ambiguous papers “unclustered” instead of forcing a topic assignment.

Concatenate the four abstracted fields for each paper and embed with OpenAI text‑embedding‑3‑large.

L2‑normalize the 3,072‑dimensional vectors.

Reduce to 10 dimensions with UMAP (`n_neighbors` = 15, `min_dist` = 0, seed = 42).

Cluster the reduced vectors with HDBSCAN, sweeping min cluster size and selecting the configuration with the highest silhouette.

Assign automatic tactic‑level labels to each cluster via an Opus 4.7 pass over the nearest representatives.

Export the 31‑cluster inventory for downstream ideation‑pattern induction.

**Table 2.** HDBSCAN sweep on the 1,891-paper embedding. Selected row bolded.

Normalize each vector to unit length (already normalized).

Apply a toy UMAP that projects to 2 D by dropping the third coordinate, giving q₁=(0.9,0.1), q₂=(0.85,0.15), q₃=(0.1,0.9), q₄=(0.0,0.95).

Run HDBSCAN with `min_cluster_size` = 2; q₁ and q₂ fall within a dense region and form Cluster A, q₃ and q₄ form Cluster B.

Because the distance between the two clusters exceeds the density threshold, no point is labeled “noise”; all four papers are clustered.

Assign labels: Cluster A → “Sequential Control”, Cluster B → “Operator Substitution”.

The toy illustrates how density‑based clustering groups papers that are close in strategy space while leaving truly isolated points “unclustered” in the full dataset.

With the 31‑cluster inventory in hand we can later tag new papers with the discovered ideation patterns and expose the risk signals (Oral‑safe, Reject‑warn, Mixed) to downstream generation.

Inducing Ideation Patterns

Inducing a 15‑pattern taxonomy from clustered papers via LLM prompting.

The previous section showed that raw clusters capture thematic similarity but give no actionable move. Without a concrete taxonomy, researchers cannot translate a cluster into a repeatable ideation step.

We ask a powerful LLM to synthesize a concise, reusable list of reasoning moves that explains each cluster, then force every cluster to adopt exactly one of those moves as its primary label.

How does this LLM‑driven taxonomy differ from manually crafting a taxonomy after inspecting clusters?

Manual taxonomy requires hand‑written definitions and iterative refinement, which introduces author bias and scales poorly. The LLM approach enforces the four constraints automatically, guarantees a single primary label per cluster, and preserves compositionality across patterns.

LLM returns the three pattern names and their signatures.

Cluster‑level mapping assigns C1→Audit Assumption, C2→Substitute Operator, C3→Reframe Object, C4→Audit Assumption, C5→Substitute Operator.

Secondary labeling later adds “Manufacture Signal” to C2 because its papers also contain a self‑supervised signal.

The final representation of a paper in C2 is the composition [Substitute Operator + Manufacture Signal].

This toy illustrates that a single LLM call yields a compact, compositional taxonomy while still allowing papers to carry multiple operators.

Prompt‑and‑assign pipeline for Ideation Pattern Induction.

**Figure 3.** Acceptance composition of the 31 clusters, sorted by Oral rate among $O+R$. Six clusters clear the 65% Oral threshold; one clears the 65% Reject threshold. The remaining 24 are mixed. Cluster labels are colored by risk flag (green = Oral-safe, red = Reject-warn, gray = mixed; deliberately off the bar palette); the threshold uses $p_O$ among $O+R$, so it is shown via label color rather than an $x$-axis line, because the bars are $O/HC/R$ shares that include HC.

**Figure 4.** 2D UMAP projection of the 1,891-paper embedding. Oral, HC, and Reject papers are interleaved across essentially every cluster: the embedding captures strategy, not acceptance.

**Table.** The 15 induced ideation patterns. $n_{cl}$=fine-grained clusters mapped as primary; Oral, HC, and Reject counts are paper-level primary-mapping counts by cluster-level inheritance (989 of the 1,891 papers); the rest are HDBSCAN-unclustered and receive their pattern via the multi-label pass. $n$ is the unique paper count under cluster-level primary; because 49 papers are both Oral and High-Confidence, Oral+HC+Reject can exceed $n$. Figure 5 sizes each wedge by the Oral+HC+Reject sum.

**Figure 5.** Ideation-pattern hierarchy across 989 clustered papers. Three concentric rings share one angular layout, ordered by size clockwise from 12 o'clock; categorical colors identify the 15 patterns and are reused across all rings, with white gaps separating segments. Inner ring: the 15 induced Level-1 ideation patterns, each wedge sized by its clustered-paper count (the count is printed inside the larger wedges; full pattern names are in the right-hand legend). Middle ring: each pattern's arc is subdivided into its constituent sub-clusters (31 in total), shaded as a lighter tint of the parent pattern; the number of segments within a wedge shows how finely that pattern fragments, while the segment angle is the equal share within the pattern, not the per-sub-cluster paper count. Outer ring: a thin heat band encoding each pattern's Oral acceptance rate $p_O = n_O / (n_O + n_{HC} + n_R)$ on the light-to-dark gray scale shown at left (light = low, dark = high), so reviewer outcome can be read against methodology at a glance. Totals: 15 patterns, 31 sub-clusters, 989 clustered papers (of 1,891 embedded).

With the taxonomy in place, the skill can now match a user’s research gap to one or more ideation patterns, compose them, and generate concrete suggestions.

Pattern Distribution and Taxonomy

Top 2‑way pattern pair boosts oral acceptance to 85 %.

Algebraic Equivalence Unification + Generative Process Redesign yields the highest oral acceptance rate among 2‑way pattern pairs.

Oral acceptance = 85.0 % (26.6 pp above the 58.4 % baseline).

The distribution of ideation patterns exhibits a long tail: the three most common patterns—Audit and Pivot an Assumption (181 papers), Substitute the Operator or Representation (109), and Liberate a Fixed Generative Component (94)—cover 384 papers, while the three least common—Adapt by Conditioning (18), Characterize a Limit (15), and Design a Property‑Targeting Pretext Objective (15)—cover only 48 papers.

Methodology fragmentation is uneven: Audit and Pivot splits into six sub‑clusters, Substitute the Operator into four, while nine patterns remain single‑cluster, indicating coherent tactical moves that the taxonomy does not further divide.

Cluster sizes are tight (13–86 papers, mean = 31.9, median = 22) and yield a silhouette score of 0.584, confirming well‑separated yet cohesive groups.

At the cluster level, 73.7 % of papers (729/989) carry both a primary and a non‑trivial secondary pattern, whereas 26.3 % (260/989) present only a primary pattern.

Paired papers achieve an oral acceptance rate of 58.2 % (366/263), essentially matching the overall baseline of 58.4 %, while single‑pattern papers lag at 52.7 % (116/104), a drop of 5.7 pp.

The two most isolated patterns—Design a Confound‑Isolating Diagnostic (cosine = 0.904) and Characterize a Limit, Then Surpass It (cosine = 0.917)—form their own pockets in the embedding space, reflecting low geometric proximity to other patterns.

Audit and Pivot an Assumption, Reframe as a Solvable Object, and Substitute the Operator each serve as nearest‑neighbor hubs for three other patterns, providing balanced default secondaries for composition.

Across the full corpus, 59.2 % of papers execute exactly two ideation patterns, 7.2 % a single pattern, and 33.6 % three or more, with a class‑conditional mean of ≈2.3 patterns per paper (Oral = 2.28, HC = 2.28, Reject = 2.32).

**Table 4.** Nearest-neighbor ideation pattern and centroid cosine. Sorted by descending cosine. The two most isolated patterns are *Design a Confound-Isolating Diagnostic* (0.904) and *Characterize a Limit, Then Surpass It* (0.917); they do not share a neighborhood and form their own pockets in the embedding.

**Figure 6.** Centroid-cosine similarity between the 15 ideation patterns. Rows/columns ordered by mean similarity (most “central” ideation patterns first). All off-diagonal entries fall in [0.79, 0.96]: ideation patterns are not orthogonal in this embedding; they form a dense neighborhood on a low-dimensional manifold.

**Table.** Ideation patterns and their associated probabilities ($p_{oral}$, $p_{hc}$, $p_{reject}$) and differences ($\Delta_{OR}$, $\Delta_{OH}$).

Acceptance and Impact Analysis

IdeaSpark extracts 15 ideation patterns and this section measures how they relate to paper acceptance.

IdeaSpark extracts 15 ideation patterns and uses them to steer LLM‑generated ideas; this section quantifies how those patterns correlate with paper acceptance.

The “Prove Equivalence to Unify” pattern exhibits the strongest acceptance bias, with $\Delta_{\text{OR}} = +10.6$ pp.

Table 7 and Figure 9 report a $\Delta_{\text{OR}}$ of +10.6 percentage points for this pattern.

Acceptance Bias measures how much more frequently a given ideation pattern appears in accepted (Oral) papers than in rejected ones.

**Figure 9.** Per-ideation-pattern acceptance bias. Bars show $\Delta_{OR}$; positive bars are Oral-enriched and negative bars are Reject-enriched. Diamonds show $\Delta_{OH}$ where $n_{HC} \ge 5$.

The table lists various "Ideation patterns" and compares their frequency or usage across two categories: "Reject-only" and "Reject-in-full", along with a delta ($\Delta$) column representing the difference between the two.

The table displays the distribution of ideation patterns ($k$) across different paper categories (all, Oral, HC, Reject). It shows the count and percentage for each $k$ value from 1 to 4, with $k=2$ identified as the mode for all categories. The final row provides the mean value $\bar{k}$ for each category, with the highest value (2.32) highlighted in bold for the HC category.

Domain-Specific Pattern Usage

Domain distribution reveals a dominant PC‑vs‑community bias far exceeding oral‑reject variance.

The PC‑vs‑community bias axis spans a 13.1 pp range, roughly 4.5× wider than the oral‑reject bias axis.

Observations O2–O4 show $\Delta$OH up to +13.1 pp versus $\Delta$OR confined within ±2.9 pp.

**Figure 8.** Top combinations under multi-label tagging. (a) 2-way combinations ($n \geq 20$); (b) 3-way combinations ($n \geq 10$). Dotted line: dataset-wide baseline $p_O = 58.4\%$.

Temporal Trends in Ideation

Temporal shifts reveal which ideation patterns are gaining or losing traction.

We compare pattern shares between an early window (2021‑2022) and a late window (2024‑2025) to expose the most dynamic ideation trends.

“Decompose for Differentiated Treatment” loses 8.5 pp, dropping from 34.7 % to 26.2 %.

Early‑period average 34.7 % (2021‑2022) vs. late‑period average 26.2 % (2024‑2025).

The early window rests on only ~150 papers, so absolute changes below 2 pp may not be statistically robust.

“Decompose and Delegate to Solvers” gains 6.3 pp, rising from 0.0 % to 6.3 %.

Early‑period share 0.0 % (no papers) vs. late‑period share 6.3 %.

ICML papers over‑index “Audit and Pivot an Assumption” by 3.8 pp (56.2 % vs. 52.4 % elsewhere).

Measured as the percentage of ICML papers carrying the pattern.

ICLR papers over‑index “Encode Structure by Construction” by 4.4 pp (20.6 % vs. 16.2 % elsewhere).

Pattern share computed across ICLR submissions.

NeurIPS papers over‑index “Decompose for Differentiated Treatment” by 4.7 pp (31.4 % vs. 26.8 % elsewhere).

Pattern share across NeurIPS submissions.

Analysis of Rejected Papers

Reject papers map onto existing ideation patterns with only modest divergences.

Four main patterns receive no Reject‑only mapping, and the empirical reading is that Reject papers do not occupy a separate strategic space. Instead, they fall into the existing 15‑pattern taxonomy with high cosine similarity. This observation motivates a closer look at the quantitative distribution.

The table presents a comparison of different embedding models and configurations across various metrics including "mcs", "k", "Unclustered %", and "Silhouette".

Ten of the fifteen patterns exhibit comparable Reject counts across the two computations (|$\Delta$| ≤ 10), so the Reject pool’s strategic shape is stable regardless of clustering mode. The notable divergences cluster in two directions: Reject‑only clustering inflates “Unify Heterogeneous Inputs into One Space” and “Audit and Pivot an Assumption”, while it under‑represents “Encode Structure by Construction”, “Reframe as a Solvable Object”, and “Manufacture the Supervisory Signal”. These asymmetries suggest that rejected work still aligns with the broader ideation taxonomy, but certain constructive patterns are less common among rejected submissions.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers