Large Discovery Models: Empirically-Grounded Model-Based Open-Ended Search

Zhongwei Yu, Yan Song, Xue Yan, Anjie Liu, Xingyu Lu, Yihang Chen, Huichi Zhou, Siyuan Guo, Luoyang Sun, Sihan Chen, Xiangning Yu, Jun Wang

Large Discovery Models (LDM) couple generative LLMs with Bayesian surrogates to guide scientific search via uncertainty-aware acquisition.

How can we unify generative LLMs with Bayesian optimization to perform efficient, open-ended scientific discovery across heterogeneous domains?

Scientific discovery requires searching vast, open-ended spaces like molecular structures or protein sequences where feedback is expensive, noisy, and sparse. Standard LLMs can generate plausible candidates but lack the empirical grounding to reliably predict which designs merit costly experimental evaluation. The Large Discovery Model (LDM) addresses this by coupling a generative foundation model with a continually updated Bayesian surrogate. The LLM proposes structured candidates, while the surrogate predicts their performance and epistemic uncertainty, allowing an acquisition function to prioritize experiments that balance exploitation of known high-performers with exploration of uncertain regions. This approach significantly outperforms traditional methods, achieving a 62.4% improvement in molecular multi-objective performance and an 18.2% reduction in antibody binding energy compared to LLM-only reflection.

Paper Primer

LDM treats scientific discovery as a sequential inverse design problem. It uses the generative model as a "reservoir" of structured candidates and the acquisition function as a "tilt" that reweights this reservoir toward designs with high decision value, effectively expanding the search frontier beyond the model's initial training distribution.

LDM achieves superior performance in multi-objective molecular optimization compared to both LLM-only and classical Bayesian optimization baselines.

Hypervolume improvement of the Pareto front on molecular design tasks. 62.4% gain over LLM-only reflection and 63.1% over classical Bayesian optimization.

LDM provides more effective guidance for experimental resource allocation than LLM-based self-reflection.

Validation BPB reduction on AutoResearch neural-network training programs.

Why is a Bayesian surrogate necessary if LLMs already possess broad scientific knowledge?

LLM likelihoods are unreliable proxies for task-specific scientific rewards, especially for novel candidates outside the training distribution. The surrogate provides calibrated epistemic uncertainty, allowing the system to distinguish between designs that are merely plausible and those that are empirically valuable to test.

How does LDM differ from standard Bayesian optimization?

Traditional Bayesian optimization struggles to propose candidates in complex, structured spaces like programs or sequences. LDM uses the LLM to generate and refine these structured candidates, while the surrogate provides the empirical grounding to navigate the resulting frontier.

LDM shifts the role of LLMs in scientific discovery from autonomous "research agents" to structured proposal engines governed by empirical, uncertainty-aware value signals. This framework allows researchers to scale discovery efforts by treating test-time compute as a search-effort dial rather than relying on model-internal confidence.

Introduction to Discovery Models

Scientific discovery demands guided search over vast, open‑ended spaces under tight evaluation budgets.

Scientific discovery requires optimizing expensive, black‑box objectives over enormous, structured hypothesis spaces—molecules, proteins, programs—while only a handful of evaluations can be afforded.

We need a system that both expands the searchable frontier (via a generative model) and evaluates candidates with an uncertainty‑aware value signal (via a surrogate), so that each costly experiment is maximally informative.

Existing LLM‑only pipelines excel at generating diverse candidates but lack a reliable estimate of scientific value, while Bayesian Optimization (BO) supplies an uncertainty‑aware ranking yet struggles to propose candidates outside its current representation.

**Figure 1.** From generation and reasoning to open-ended discovery. LLM generation produces responses through autoregressive prediction. LLM reasoning adds inference-time computation (e.g., chain-of-thought, tree search, or best-of-N) with a cheap verifier to select solutions (Brown et al., 2020; OpenAI, 2024; Wang et al., 2024; DeepSeek-AI, 2025). Large Discovery Models augment LLMs with an empirically grounded signal that guides exploration and discovery, enabling search in open-ended scientific domains under expensive black-box evaluation.

The key shift is moving from fixed‑domain, verifier‑driven search to open‑ended, generative discovery guided by an uncertainty‑aware value model.

Defining the Discovery Space

Scientific discovery must navigate an open‑ended hypothesis space under costly, noisy evaluations.

Discovery proceeds in an open‑ended hypothesis space $X$, where the objective is to find a design $x$ that maximizes an unknown reward function $R(x)$. In practice the evaluator can only query a reachable subset $X_0\subset X$, and each query is expensive, noisy, and black‑box. Moreover, the full space $X$ is not known a priori; it must be gradually constructed as the search progresses.

BO treats the search as a sequential decision problem, using a surrogate model of past evaluations to pick the next candidate that best balances learning about the unknown reward and exploiting promising regions.

To reason about what is currently known, the authors partition the space $X$ into four epistemic regimes. “Known knowns” are designs already evaluated with low surrogate uncertainty $\sigma_t(x)$. “Known unknowns” are reachable designs with high uncertainty that can be resolved by further evaluation.

“Unknown knowns” refer to designs that exist externally (e.g., in a database) but have not yet been retrieved into the current search context. “Unknown unknowns” are designs beyond the current reachable frontier or outside the surrogate’s modelling support, so the surrogate cannot provide meaningful predictions for them.

The search frontier marks the boundary of what the current procedure can formulate, reach, and model. Within this frontier, three operations drive progress: exploitation selects high‑reward, low‑uncertainty designs (known knowns); exploration evaluates reachable but uncertain designs (known unknowns); discovery expands the frontier by generating hypotheses that were previously outside reach, turning unknown unknowns into known unknowns.

**Figure 2.** The empirical scientific discovery loop. An agent generates candidate hypotheses or designs, selects a subset for costly evaluation, observes the outcomes, and uses those observations to update its beliefs and guide the next round of search, as described in Definition 1.

The LDM Architecture

Defines the Large Discovery Model and its acquisition‑tilted search mechanism.

The Large Discovery Model (LDM) turns a generative language model into a discovery‑oriented policy by repeatedly coupling generation, surrogate‑driven evaluation, and acquisition‑guided refinement.

LDM is a recurrent policy that draws candidate designs from a generative model, evaluates them, updates a probabilistic surrogate, and uses an acquisition function to tilt future draws toward high‑value experiments.

The surrogate is a Bayesian model that turns past design–reward pairs into a predictive distribution over the unknown reward function, providing both an expected reward and a calibrated uncertainty.

Why is a surrogate needed when the LLM already encodes domain knowledge?

The LLM’s internal probabilities are not calibrated to the task‑specific quantitative reward and provide no principled uncertainty estimate. The surrogate supplies an empirically grounded, uncertainty‑aware estimate of R, enabling the policy to reason about where additional evaluations would be most informative.

The acquisition function turns the surrogate’s mean and uncertainty into a scalar “decision value” that tells the policy which candidates are worth evaluating or refining.

How does an acquisition value differ from a plain reward‑model score?

A reward‑model score predicts the expected reward $\mu$ₜ(x) only. An acquisition value augments this with uncertainty $\sigma$ₜ(x) (and possibly other factors), yielding a decision‑centric metric that values information gain as well as immediate payoff.

Compute the unnormalised weights w(x)=p(x)·exp($\eta$·aₜ(x)): w(A)=0.25·$e^{0.2}$=0.25·1.22≈0.305, w(B)=0.25·$e^{0.8}$=0.25·2.23≈0.558, w(C)=0.25·$e^{0.4}$=0.25·1.49≈0.373, w(D)=0.25·$e^{0.6}$=0.25·1.82≈0.455.

Sum the weights Zₜ≈0.305+0.558+0.373+0.455=1.691.

Normalise to obtain the tilted distribution: $\pi$ₜ(A)≈0.305/1.691≈0.18, $\pi$ₜ(B)≈0.33, $\pi$ₜ(C)≈0.22, $\pi$ₜ(D)≈0.27.

Design B receives the highest probability because its acquisition value is largest, illustrating how the tilt pushes sampling toward promising candidates while still respecting the prior.

The example shows that even with a flat prior, the exponential tilt can dramatically reshape the sampling distribution, and the strength of that reshaping is governed by $\eta$.

**Figure 5.** The Large Discovery Model learning loop. The right (cyan) region denotes the *fast learning loop* of the LDM's recurrent discovery: a candidate batch $B_t$ is drawn and evaluated, and the observations update the surrogate data $\mathcal{D}_t$ and the LLM context $\mathcal{C}_t$. The left (yellow) side corresponds to the *slow learning loop*, which amortises the acquisition-tilted search policy into the LLM's weights $\theta$ via LDM-TTS fine-tuning (see §3.4). All candidates $\{\hat{x}_{i,t}\}_{i=1}^N$ generated by the LLM during test-time search, alongside their respective acquisition values, are recorded to a TTS dataset. Using this dataset, we fit the empirical search distribution into the network, endowing the foundation model with generalisable discovery expertise to enable stronger discovery performance on subsequent tasks. The fast loop iterates with each batch of reward evaluation within a specific discovery task, while the slow loop is executed only after sufficient new samples spanning diverse tasks have been collected for the TTS dataset.

**Figure 6.** Fine-tuning as value distillation across model regimes. Chat LLMs distil human preference value to align with subjective intuition. Reasoning LLMs distil verification value to navigate deterministic proof paths. In contrast, Discovery LLMs distil epistemic acquisition value. Rather than merely memorising static, domain-specific solutions, discovery fine-tuning trains a research manager that learns an intrinsic acquisition strategy that prioritising high-information experiments to expand open-world boundaries and advance an evolving research trajectory.

**Figure a.** Effect of the LLM sampling temperature $T_{LLM}$ in `LDM_DirectSoftmax`; for this controlled base-measure-exponent ablation, $T_{LLM} = 1/\alpha$. The acquisition temperature is fixed to $\eta = 1$.

The LDM Loop

How the Large Discovery Model iteratively proposes, evaluates, and refines scientific designs.

Evaluating a scientific hypothesis is costly, so naïve exhaustive search quickly exhausts the budget. The Large Discovery Model therefore needs a tight loop that extracts maximal information from each experiment while keeping compute tractable.

At test time the model repeatedly tilts a generative prior toward promising regions, draws a batch of candidates, evaluates them, and feeds the results back into the prior – a closed‑loop that turns a few expensive experiments into a refined search policy.

Round 1: Fit a GP on the empty dataset (prior mean only). Acquisition scores are all equal, so deterministic top‑$b$ picks $x_1$ and $x_2$ as $B_1$.

Observe rewards $r_{1,1}=0.4$, $r_{1,2}=0.6$; update $\mathcal{D}_2$ and append “$x_2$ performed best” to $\mathcal{C}_2$.

Round 2: GP posterior now has mean $\mu_2$ higher near $x_2$. Acquisition tilts the prior, making $x_2$ and $x_3$ highest‑scoring; $B_2=\{x_2,x_3\}$.

Observe $r_{2,1}=0.65$, $r_{2,2}=0.3$; update dataset and context (reflection notes that $x_3$ is promising despite lower prior probability).

Round 3: Tilted prior concentrates around $x_2$; top‑$b$ yields $B_3=\{x_2,x_4\}$. After evaluation, $x_2$ remains the best observed design.

The loop quickly homes in on high‑reward regions by repeatedly reshaping the sampling distribution with cheap GP‑based acquisition, even though each round evaluates only two designs.

How does Test‑Time Search differ from a standard test‑time inference pass?

Standard inference draws a single output from the model’s prior. TTS repeatedly reweights that prior with an acquisition‑driven tilt, evaluates a batch of candidates, and feeds the results back into the prior, turning inference into an adaptive search rather than a one‑shot prediction.

Acquisition‑Guided Inference‑time Sampling (Algorithm 2)

**Algorithm 1** The Large Discovery Model Loop **Require:** Reward function $R : \mathcal{X} \to \mathbb{R}$; prior mean $m(\cdot)$; kernel $k(\cdot, \cdot)$; initial dataset $\mathcal{D}_1$; initial context $\mathcal{C}_1$; evaluation budget $T$; batch size $b$; candidate pool size $N$. 1: **for** $t = 1, \dots, T$ **do** 2: $\quad$ Update the active decision domain: $A_t \leftarrow \text{supp } p_{\theta, \alpha}(\cdot \mid \mathcal{C}_t)$ 3: $\quad$ Fit Gaussian-process surrogate over $A_t$ using data $\mathcal{D}_t$ to obtain the posterior $R \mid \mathcal{D}_t \sim \mathcal{GP}(\mu_t, k_t)$, given by Eqs. (29), (30) $\quad \triangleright$ Full derivation in Appendix C.1. 4: $\quad$ Construct acquisition function $a_t$ by combining $\{\mu_t, \sigma_t\} \quad \triangleright$ e.g. Eq. (32) or (31). 5: $\quad$ Draw $B_t = \{x_{t,1}, \dots, x_{t,b}\}$ from $\pi_t \propto p_{\theta, \alpha}(\cdot \mid \mathcal{C}_t) \exp\{\eta a_t\}$ using Algorithm 2 6: $\quad$ Observe black-box noisy rewards $r_{t,i} = R(x_{t,i}) + \epsilon_{t,i}$ for all $x_{t,i} \in B_t$ 7: $\quad$ Update dataset: $\mathcal{D}_{t+1} \leftarrow \mathcal{D}_t \cup \{(x_{t,i}, r_{t,i})\}_{i=1}^b$ 8: $\quad$ Update context: $\mathcal{C}_{t+1} \leftarrow \mathcal{C}_t \cup \{(x_{t,i}, r_{t,i})\}_{i=1}^b \cup \{\text{Reflection feedback (if applicable)}\}$ 9: **return** $\arg \max_{(x,r) \in \mathcal{D}_{T+1}} r$

Theoretical Foundations

Analyzes how the LLM proposal and acquisition tilt jointly shape regret in the Large Discovery Model.

Searching vast, open‑ended hypothesis spaces with a limited budget of expensive evaluations is the core difficulty that motivates the Large Discovery Model. Standard BO analyses assume a static, fully accessible domain, which does not hold when a language model continuously reshapes the reachable set. We therefore analyse how the LLM‑driven reservoir and the acquisition‑tilted sampling interact.

The LDM draws the next candidate from a Gibbs distribution that exponentially tilts the LLM’s proposal toward high acquisition values, thereby balancing the dynamic support of the language model with the exploitation pressure of UCB.

Compute the unnormalized weights: $w_1=0.4\\,e^{1\\cdot1.0}=0.4e^{1}=1.09$, $w_2=0.3\\,e^{2}=0.3e^{2}=2.21$, $w_3=0.2\\,e^{0.5}=0.2e^{0.5}=0.33$, $w_4=0.1\\,e^{1.5}=0.1e^{1.5}=0.45$.

Sum the weights: $Z = 1.09+2.21+0.33+0.45 = 4.08$.

Normalize: $\\pi_t(x_1)=1.09/4.08=0.27$, $\\pi_t(x_2)=2.21/4.08=0.54$, $\\pi_t(x_3)=0.33/4.08=0.08$, $\\pi_t(x_4)=0.45/4.08=0.11$.

Sample $x_t$ according to these probabilities; the most likely outcome is $x_2$ because it combines a high acquisition score with a non‑negligible LLM probability.

The tilt amplifies the effect of the acquisition score while still respecting the LLM’s original distribution, so candidates that are both plausible (high $p_{\\theta,\\alpha}$) and promising (high $a_t$) dominate the sampling.

How does this sampling differ from directly maximising the acquisition function over $A_t$?

Direct maximisation would always pick the argmax of $a_t$, discarding the LLM’s semantic bias and any stochastic exploration. The Gibbs tilt instead yields a stochastic policy that can still explore lower‑scoring candidates, which is crucial when the LLM’s support $A_t$ does not yet contain the true optimum.

At each evaluation the total regret splits into a discovery gap—how far the LLM’s current support is from the global optimum—and an optimisation gap—how far the sampled point is from the best point inside that support.

Why is it useful to separate regret into discovery and optimisation components?

Because the two terms are driven by different parts of the algorithm: the first by how the LLM expands $A_t$, the second by how the Gibbs sampler $\\pi_t$ exploits the acquisition signal. This separation lets us bound each effect individually and understand which design choices (e.g., improving the LLM’s generative quality versus sharpening $\\eta$) will reduce overall regret.

We rely on two standard assumptions. First, the UCB calibration (Assumption 1) guarantees that the true reward lies within the confidence interval defined by the surrogate’s mean and variance. Second, the reservoir coverage assumption (Assumption 2) links the geometric distance of the support to the objective gap via a Hölder condition, and introduces the near‑UCB set $U_t(\\zeta_t)$ and its probability mass $\\kappa_t(\\zeta_t)$.

Under Assumptions 1 and 2, for any $\\delta_{\\text{gp}},\\delta_{\\text{samp}}\\in(0,1)$ and any choice of $\\rho_t>0$ with $\\sum_{t=1}^T \\rho_t \\le \\delta_{\\text{samp}}$, with probability at least $1-\\delta_{\\text{gp}}-\\delta_{\\text{samp}}$, $$ \\frac{1}{T}\\sum_{t=1}^T \\text{Reg}_t \\le \\frac{L}{T}\\sum_{t=1}^T (\\text{rcov}_t)^{\\chi} \\;+\\; \\frac{2 C_\\lambda \\beta_T \\gamma_T}{T} \\;+\\; \\frac{1}{T}\\sum_{t=1}^T \\left( \\log\\zeta_t + \\frac{1}{\\eta}\\right) \\frac{1}{\\rho_t \\kappa_t(\\zeta_t)}. $$

Apply Lemma 1 to bound the acquisition error $a^\\star_{t,A} - a_t(x_t)$ in terms of $\\zeta_t$ and $\\kappa_t(\\zeta_t)$.

Combine the acquisition error bound with Assumption 1 to control $\\text{Reg}_{\\text{opt},t}$.

Sum over $t$ and use the standard GP‑UCB information‑gain bound $\\sum_{t=1}^T \\beta_t \\sigma_t(x_t) \\le C_\\lambda \\gamma_T$.

Empirical Results

We quantify how LDM outperforms baselines across code, antibody, and molecule design.

The LDM framework treats scientific discovery as a sequential decision problem: a surrogate guides an LLM‑generated proposal pool, and the loop iterates until the evaluation budget is exhausted.

LDM reduces validation bits‑per‑byte by 0.0727 (a 2.4× larger drop than the LLM‑only baseline) while keeping the same five‑minute per‑run budget.

Figure 4(a) shows the LDM bar 2.4× higher than the grey LLM‑only bar; the numeric label “‑0.0727” quantifies the reduction.

After a high‑budget test‑time search, the LLM is fine‑tuned on the surrogate‑ranked proposals so that future generations inherit the discovered search strategy.

**Figure 4.** Complementary limitations of LLM-only and BO-based search. (a) LDM improves upon the LLM-only AutoResearch baseline under the same H100 hardware setting. (b) Final best-so-far binding energy on the antibody-design task, where LDM outperforms the LLM-only and BO-only baselines. Lower binding energy is better. For details, we refer to § 6.

**Figure 7.** The discover loop on autoresearch. Validation `val_bpb` (lower is better) versus search progress. The H100 panel compares the no-discover Karpathy baseline with the LDM; the B200 panel reports a matched-hardware leaderboard run of the LDM. B200 width scaled for legibility.

**Table 1.** The autoresearch run as physics-grounded discovery. Each row lists a change the LDM retained under a fixed 300s budget, the physical mechanism by which it helps, and the resulting change in `val_bpb`.

**Table 2.** Implemented LDM algorithm variants for computational antibody design.

Related Work

We position LDM among LLMs, Bayesian optimization, and hybrid methods, highlighting its unique contributions.

The literature splits into three strands that motivate LDM: the rapid progress of large language models, the principled uncertainty‑aware search of Bayesian optimization, and recent hybrid systems that fuse the two.

Introduced the self‑attention architecture that replaced recurrence and enabled parallel processing of token sequences.

Demonstrated that scaling model size, data, and compute yields emergent capabilities such as few‑shot reasoning.

A black‑box optimisation framework that builds a probabilistic surrogate (often a Gaussian process) and selects queries via an acquisition function.

Uses an LLM as an optimiser by prompting it with a history of solution–score pairs, iteratively refining candidates.

Integrates LLMs for warm‑starting, surrogate modelling, and candidate sampling within a BO loop, improving low‑data hyperparameter optimisation.

Performs offline black‑box optimisation with a masked‑diffusion tree search; leaf candidates are scored by expected improvement under a GP surrogate.

A simple pipeline that runs a fixed LLM to generate candidates, evaluates each with a cheap proxy, and selects the best without any adaptive surrogate or acquisition strategy.

Ablation Studies

We isolate each component of the LDM framework to see how its removal impacts performance.

Autoresearch runs a multi‑turn code‑editing loop, while small‑molecule and CDRH3 design are single‑step proposal tasks that invoke a surrogate‑guided acquisition at each Bayesian‑Optimization round.

The ablations answer two questions: (1) does removing the BO value model cripple the system, and (2) how does test‑time search scale when we vary inner‑loop budgets.

**Figure 20.** Pure LLM research loop on autoresearch. Validation `val_bpb` (lower is better) across 875 no-BO, no-acquisition experiments. Grey points are individual trials and the green step curve is the best value found so far; yellow stars mark self-reflection checkpoints where the agent summarises the ledger and enters a new research regime. The coloured bands show the loop’s staged progress: dense scaling, sparse-memory invention, dense-compute rebalancing around sparse memory, ablation-based confirmation of the 512-sparse frontier, and final VRAM-aware batch/schedule search. The curve demonstrates that a pure LLM loop can do real sequential research, but also its ceiling: after the large regime shifts it plateaus near `val_bpb` $\approx$ 0.956, above the LDM result of 0.93421 in Figure 7.

The pure LLM research loop ($\eta$ = 0) reaches a final val bpb of ≈ 0.956, better than the no‑discover baseline (0.9767) but still above the LDM result (0.93421).

Figure 20’s curve descends from 1.069 → 1.01 → 0.991 → 0.959 before flattening at ≈ 0.956.

In the small‑molecule ablation we vary the LLM proposal pool size K and the BO inner‑loop budget M. The balanced setting proposer 64 bo 64 yields the highest final Pareto hypervolume, while increasing K without matching M gives diminishing or unstable gains.

Mean‑ or UCB‑style acquisitions (a 0.5/0.5 scalarisation of Vina and activity) outperform strict EHVI at low budgets; once budgets grow, EHVI overtakes and continues improving, whereas mean‑style acquisitions plateau or degrade.

The CDRH3 ablation fixes the BO side at its maximum and sweeps only the LLM proposer budget. Budgets of 150–300 candidates consistently give the best final binding energies, but the optimal budget varies per antigen (e.g., 150 for ADQ A, 300 for 1FBI X), reflecting a non‑monotonic trade‑off in a rugged sequence landscape.

**Figure b.** Effect of the acquisition temperature $\eta$ in `LDM_PolicySoftmax`. The LLM sampling temperature is fixed to $T_{LLM} = 1$ (and hence $\alpha = 1$ in the controlled exponent setting).

Temperature sweeps for LDMDirectSoftmax (LLM sampling temperature Tₗₗₘ) and LDMPolicySoftmax (acquisition temperature $\eta$) show only weak sensitivity on antigen 1NSN S; Direct‑Softmax slightly favors low Tₗₗₘ, while Policy‑Softmax has a marginal edge near $\eta$ = 4, but all error bars overlap.

Fine-Tuning Analyses

Fine‑tuning distills acquisition value and yields consistent gains across tasks.

This section reports the quantitative impact of fine‑tuning the proposer to internalise acquisition value, covering single‑task fit, cross‑domain transfer, and out‑of‑distribution generalisation.

Fine‑tuned Qwen3.5‑9B with chain‑of‑thought improves KRAS hypervolume by roughly +9 HV over the base model.

Table 6 shows the base model achieving 11.64 ± 2.63 (G12C) and 16.49 ± 5.87 (G12D), while the fine‑tuned model reaches 20.98 ± 2.92 and 26.66 ± 2.61 respectively.

Cross‑molecule fine‑tuning raises hypervolume by about +6 HV compared with the base LDM on a different molecular task.

E.2 reports a base hypervolume of 16.489 ± 5.867 versus 22.279 ± 3.828 after fine‑tuning on a source task and evaluating on a new target.

When Gaussian‑process values are hidden, the proposer attains a best binding energy of ‑115.9, surpassing both the base model (‑90.9) and the GP‑in‑prompt variant (‑109.4).

Table 8 lists the GP‑hidden row achieving ‑115.9 on the 1FBI X antigen, the lowest (best) value among all compared methods.

**Figure 18.** Out-of-distribution generalisation to held-out antigens (3-train/2-test). Best binding energy versus number of evaluations on the two held-out antigens. The mixed-task fine-tuned model evaluated on antigens it never saw during training (red) matches or exceeds the in-distribution fine-tuned model (teal) and clearly beats base Qwen3.5-9B with and without chain-of-thought (dashed).

Questions & answers

What is the Large Discovery Model (LDM) and what does it contribute?

LDM is an experiment-grounded recurrent architecture that tightly couples the generative prior of a large language model with a Gaussian-process surrogate's value signal for scientific discovery. Its main contributions are a KL-regularised, acquisition-guided optimisation framework, a closed-form acquisition-tilted optimal search distribution, a tripartite exploitation-exploration-discovery regime, and a full regret decomposition isolating three error sources: generative coverage gap, surrogate fitting error, and sampling-strategy suboptimality.

What problem does LDM address and why does it matter?

LDM addresses the challenge of optimising expensive, black-box objectives over enormous structured hypothesis spaces—such as molecules, proteins, and programs—where only a handful of evaluations can be afforded. Standard LLMs generate plausible candidates but cannot reliably estimate scientific value or uncertainty, while classical Bayesian optimisation struggles to propose valid candidates in complex structured spaces.

Why is a Bayesian surrogate necessary if LLMs already encode broad scientific knowledge?

LLM internal probabilities are not calibrated to task-specific quantitative rewards and provide no principled uncertainty estimate, making them unreliable proxies for scientific value—especially for novel candidates outside the training distribution. The Gaussian-process surrogate supplies an empirically grounded, uncertainty-aware estimate of the reward function, enabling the system to distinguish designs that are merely plausible from those that are empirically valuable to test.

How does LDM work at a high level?

LDM runs a closed loop of three steps: the LLM generates a pool of structured candidates, a Gaussian-process surrogate predicts their expected reward and epistemic uncertainty, and an acquisition function reweights (tilts) the candidate pool to prioritise experiments that balance exploitation of known high-performers with exploration of uncertain regions. This loop iterates until the evaluation budget is exhausted, continuously expanding the search frontier.

How does LDM differ from standard Bayesian optimisation?

Traditional Bayesian optimisation struggles to propose candidates in complex structured spaces such as programs or sequences because it lacks a generative model for those domains. LDM uses the LLM to generate and refine structured candidates, while the surrogate provides the empirical grounding to navigate the resulting frontier, combining the strengths of both approaches.

What is Test-Time Search (TTS) in LDM and how does it differ from standard inference?

Standard inference draws a single output from the model's prior in one shot. TTS repeatedly reweights that prior with an acquisition-driven tilt, evaluates a batch of candidates, and feeds results back into the prior, turning inference into an adaptive search process and treating test-time compute as a search-effort dial.

What is the Gibbs tilt and why is it used instead of directly maximising the acquisition function?

The Gibbs tilt yields a stochastic sampling policy that can still explore lower-scoring candidates, preserving the LLM's semantic bias and stochastic exploration. Direct maximisation would always pick the argmax of the acquisition function, discarding this diversity, which is problematic when the LLM's current support does not yet contain the true optimum.

What are the four epistemic regimes LDM uses to partition the search space?

LDM partitions the hypothesis space into: 'known knowns' (evaluated designs with low surrogate uncertainty), 'known unknowns' (reachable designs with high uncertainty resolvable by further evaluation), 'unknown knowns' (designs existing externally, e.g., in a database, but not yet retrieved into the search context), and 'unknown unknowns' (designs beyond the current reachable frontier where the surrogate cannot provide meaningful predictions).

Across which domains was LDM empirically evaluated?

LDM was evaluated across three heterogeneous domains: auto-research neural-network training program optimisation, antibody CDRH3 sequence design, and multi-objective small-molecule optimisation. The paper states that the integrated LLM-surrogate system outperforms both LLM-only and BO-only baselines across all three domains.

What are the key quantitative results reported for LDM?

LDM achieves a 62.4% improvement in molecular multi-objective performance and an 18.2% reduction in antibody binding energy compared to LLM-only reflection. In ablation studies, the balanced setting of proposer pool size K=64 and BO inner-loop budget M=64 yields the highest final Pareto hypervolume for small-molecule optimisation, and CDRH3 design benefits from LLM proposer budgets of 150–300 candidates depending on the antigen target.

What do the ablation studies reveal about LDM's design choices?

Ablations show that removing the Bayesian optimisation value model significantly degrades performance, confirming its necessity. Balancing the LLM proposal pool size K and BO inner-loop budget M is important—increasing K without matching M gives diminishing or unstable gains. Mean- or UCB-style acquisitions outperform EHVI at low budgets, but EHVI overtakes them as budgets grow. Temperature sensitivity for sampling strategies is weak, with overlapping error bars across settings.

What do the fine-tuning analyses show?

Fine-tuning the proposer to internalise acquisition value via reasoning-augmented supervised fine-tuning dramatically cuts inference-time cost while preserving search quality. Fine-tuning experiments demonstrate stable gains across cross-target and cross-task transfer, indicating that LDM learns a generalisable discovery decision logic rather than memorising task-specific solutions.

What are the main limitations of LDM acknowledged by the paper?

The paper identifies five limitations: (1) the Gaussian-process surrogate incurs cubic computational complexity in the number of observations, limiting scalability; (2) test-time search imposes substantial LLM inference overhead via batch candidate sampling; (3) surrogate kernel functions and feature representations require manual domain-specific customisation; (4) performance is bounded by the LLM's pretrained domain knowledge, making superiority over specialised methods hard to achieve in domains with weak priors; and (5) the closed loop does not systematically incorporate existing external knowledge, leaving 'unknown knowns' unused.

How does LDM compare to LLM-only pipelines and pure Bayesian optimisation?

LLM-only pipelines generate diverse candidates but lack reliable estimates of scientific value or uncertainty. Pure Bayesian optimisation provides uncertainty-aware ranking but struggles to propose valid candidates in complex structured spaces. LDM's integrated LLM-surrogate system outperforms both baselines across all three evaluated domains, as reported in the empirical results.

What future work does the paper propose?

The paper proposes adding memory-retrieval mechanisms and federated discovery frameworks to ingest external knowledge bases and parallel exploration data, exploring sparse Gaussian processes to reduce computational complexity, developing end-to-end surrogate specification where the LLM autonomously learns kernel functions, and advancing discovery-oriented post-training paradigms using preference learning and reinforcement learning to internalise acquisition-guided decision logic.

Who are the authors, and where and when was this paper published?

The paper does not specify individual author names in the provided text. It is available on arXiv at https://arxiv.org/abs/2608.15669; the paper does not state a specific publication venue or date beyond the arXiv identifier.

How can a practitioner reproduce or apply LDM?

The paper includes an implementation appendix describing how the Gaussian-process surrogate, acquisition functions, and batch-sampling routine are realised in code, as well as a notation table and detailed proofs for Proposition 1, Lemma 1, and Theorem 1. The paper does not specify a public code repository or dataset release.

Key terms

Large Discovery Model (LDM)
An experiment-grounded recurrent architecture that couples a large language model's generative prior with a Gaussian-process surrogate's uncertainty-aware value signal to guide sequential scientific discovery.
Bayesian surrogate
A probabilistic model (here, a Gaussian process) that predicts both the expected reward and the epistemic uncertainty of unevaluated candidates based on past experimental observations.
acquisition function
A decision-centric scoring function that combines predicted reward and uncertainty to determine which candidate designs are most valuable to evaluate next, balancing exploitation and exploration.
Test-Time Search (TTS)
An inference procedure that iteratively reweights the LLM's generative prior using an acquisition-driven tilt and feeds experimental results back into the prior, turning a single inference pass into an adaptive search loop.
Gibbs tilt
A stochastic sampling strategy that reweights candidate probabilities proportionally to an exponentiated acquisition score, preserving exploration of lower-scoring candidates rather than always selecting the argmax.
epistemic uncertainty
Uncertainty arising from limited knowledge or data, which can in principle be reduced by gathering more observations, as opposed to irreducible noise in the system.
Gaussian process (GP)
A probabilistic model that defines a distribution over functions and provides both a mean prediction and a calibrated uncertainty estimate at any input point, used here as the Bayesian surrogate.
Bayesian optimisation (BO)
An iterative optimisation strategy for expensive black-box functions that uses a probabilistic surrogate model and an acquisition function to select the most informative next evaluation.
CDRH3
The third complementarity-determining region of an antibody's heavy chain, a short amino-acid sequence loop that largely determines antigen-binding specificity and affinity.
Pareto hypervolume
A scalar metric for multi-objective optimisation that measures the volume of the objective space dominated by the current set of non-dominated solutions, used here to evaluate molecular multi-objective performance.
EHVI (Expected Hypervolume Improvement)
A multi-objective acquisition function that quantifies the expected increase in Pareto hypervolume from evaluating a candidate, balancing exploration and exploitation across multiple objectives.
UCB (Upper Confidence Bound)
An acquisition strategy that selects candidates with high predicted reward plus a bonus proportional to their surrogate uncertainty, encouraging exploration of uncertain regions.
KL-regularised optimisation
An optimisation framework that penalises the search policy for deviating too far from a reference distribution (here, the LLM prior) using Kullback-Leibler divergence, preventing the policy from collapsing to a single candidate.
regret decomposition
A theoretical analysis that separates the total performance gap from the optimum into distinct components—here, generative coverage gap, surrogate fitting error, and sampling-strategy suboptimality—to identify which parts of the algorithm drive performance.
search frontier
The boundary of the hypothesis space that the current procedure can formulate, reach, and model; expanding it constitutes the 'discovery' operation in LDM's tripartite regime.
inverse design
The problem of finding an input (e.g., a molecule or protein sequence) that produces a desired output property, as opposed to forward prediction of properties from a known input.
reasoning-augmented supervised fine-tuning
A training procedure that fine-tunes the LLM on examples that include explicit reasoning traces aligned with acquisition-guided decision logic, amortising the cost of test-time search into model weights.
Hölder condition
A mathematical smoothness assumption stating that the objective function does not change too rapidly between nearby points, used here to link geometric distance in the hypothesis space to the objective gap in the regret analysis.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers