Intern-S2-Preview: Scientific Agentic Foundation Model

Lei Bai, Jiaqi Cao, Chiyu Chen, Guanzhou Chen, Kai Chen, Guangran Cheng, Erfei Cui, Xuanlang Dai, Shengyuan Ding, Shangheng Du, Yanhui Duan, Yue Fan, Youqing Fang, Quan Gan, Yuanyuan Gao, Jiaye Ge, Lixin Gu, Yuzhe Gu, Qipeng Guo, Junjun He, Xin Hong, Ming Hu, Zhouqi Hua, Haian Huang, Junhao Huang, Zixian Huang, Minxi Jin, Lingkai Kong, Alexander Lam, Zehao Li, Zonglin Li, Tianhao Liang, Dahua Lin, Junyao Lin, Tianyang Lin, Zhouhan Lin, Jiangning Liu, Jin Liu, Kuikun Liu, Wenran Liu, Yifei Liu, Yuhong Liu, Zhoumianze Liu, Ziyan Liu, Ziyu Liu, Haijun Lv, Han Lv, Chengqi Lyu, Le Ma, Ningsheng Ma, Zerun Ma, Haoyang Peng, Runyu Peng, Jifei Shan, Zixin Shang, Kou Shi, Xiang Shi, Qisheng Su, Xuerui Su, Hao Sun

Intern-S2-Preview is a scientific foundation model series integrating multimodal reasoning, numerical forecasting, and agentic tool-use.

How can a foundation model be architected and trained to integrate scientific tool use, long-horizon reasoning, and heterogeneous data modalities into a unified agentic system?

Scientific discovery requires models that can reason over heterogeneous data, forecast numerical trends, and execute long-horizon tasks using external tools. Existing models often treat these as isolated question-answering problems, failing to sustain adaptive planning across complex scientific workflows. Intern-S2-Preview-397B addresses this by integrating a dedicated numerical forecasting branch and a modular "Memory Decoder" that attaches domain-specific knowledge to the frozen backbone. The model is trained via a staged pipeline of visual pre-training, multi-task reinforcement learning, and agentic policy distillation. Evaluations show the model achieves leading performance on scientific benchmarks, with the Memory Decoder improving domain-specific accuracy without modifying the core 397B parameters.

Paper Primer

The core challenge is that scientific expertise is long-tailed and evolving, making it difficult to specialize a large model without degrading its general reasoning or agentic capabilities. Intern-S2-Preview solves this by decoupling general-purpose foundations from domain-specific knowledge: the 397B backbone remains frozen, while lightweight "Memory Decoders" are trained on domain corpora and fused at inference time via a token-level router.

To handle long-horizon scientific tasks, the authors implement a "pause-and-resume" partial rollout system that prevents long-tail generations from stalling training. This is paired with online speculative decoding, where a draft model is continuously updated to track the evolving policy, providing a lossless 2x speedup in rollout generation.

Modular domain specialization improves performance without backbone retraining.

The Intern-MemDec-4B extension increases the Biology-Instructions average score from 56.92 to 60.32.

The upgraded time series encoder significantly improves efficiency for long-sequence scientific signals.

The encoder supports up to 300,000 time steps, achieving 5–6x faster inference and 80% lower memory usage compared to the previous version.

Why use a separate Memory Decoder instead of standard fine-tuning?

Standard fine-tuning often perturbs the model's general reasoning and agentic capabilities. The Memory Decoder allows for plug-and-play domain extension, keeping the 397B backbone frozen while supplying specialized knowledge through dynamic fusion.

How does the model handle the computational bottleneck of long-horizon RL rollouts?

The authors use a co-located partial-rollout system that pauses in-flight requests to form training batches, combined with online speculative decoding that accelerates generation by verifying draft tokens in parallel.

Intern-S2-Preview demonstrates that scientific foundation models can achieve modular, scalable specialization through frozen backbones and efficient, entropy-aware reinforcement learning, setting a template for future domain-specific agentic systems.

Introduction and Motivation

We define the scientific reasoning gap and introduce Intern‑S2‑Preview‑397B as a unified foundation model to address it.

Current AI systems excel at isolated question answering but fall short when scientific discovery demands reasoning over heterogeneous evidence, tool interaction, and long‑horizon planning. General‑purpose LLMs lack specialization for scientific modalities, while existing scientific multimodal models remain static agents that cannot sustain iterative problem solving.

The field needs a single model that can understand scientific signals, process visual documents, and act as an agent over extended tasks.

To realize this vision, Intern‑S2‑Preview‑397B adds a dedicated time‑series forecasting branch for numerical signal understanding and a separate Memory Decoder that attaches parametric memories to the frozen 397B backbone, enabling rapid domain specialization without altering the core model.

The need for unified scientific reasoning and tool use drives the design of Intern‑S2‑Preview‑397B.

Architecture and Memory Decoder

Memory Decoder adds plug‑and‑play domain knowledge to a frozen Intern‑S2‑Preview‑397B via dynamic fusion.

Scientific domains evolve continuously, so a single frozen model cannot cover every emerging subfield without risking interference with its general reasoning abilities.

It attaches a small, trainable memory module alongside the frozen Intern‑S2‑Preview‑397B backbone, letting each token decide how much to rely on the memory’s domain‑specific prediction.

Compute the weighted contribution of the backbone: $(1-\lambda)\,p_{\text{S2}} = 0.6 \times [0.6,0.3,0.1] = [0.36,0.18,0.06]$.

Compute the weighted contribution of the memory: $\lambda\,p_{\text{mem}} = 0.4 \times [0.2,0.5,0.3] = [0.08,0.20,0.12]$.

Sum the two contributions element‑wise to obtain $p_{\text{final}}=[0.44,0.38,0.18]$.

Normalize (already sums to 1) and select the highest‑probability token A as the next output.

The router lets the model fall back to the general backbone when the memory is uncertain (low $\lambda$) and amplify domain‑specific predictions when the memory is confident (high $\lambda$), all without altering backbone weights.

**Figure 1.** Architecture of the separate Memory Decoder extension for Intern-S2-Preview-397B. The frozen Intern-S2-Preview-397B backbone and a domain memory process the same input in parallel and produce separate next-token distributions. A lightweight token-level router uses their hidden states and output-distribution uncertainty features to predict a dynamic fusion weight $\lambda$, which controls the contribution of the two distributions to the final prediction.

How does Memory Decoder differ from classic adapter fine‑tuning?

Adapters insert trainable layers inside the frozen backbone, so updating them still perturbs the backbone’s internal representations. Memory Decoder keeps the backbone entirely immutable and injects domain knowledge only at the output distribution level via a separate model and a token‑wise router, guaranteeing zero interference with the backbone’s general capabilities.

By decoupling domain specialization from the core model, Memory Decoder enables continual expansion to new scientific subfields without ever risking regression on previously mastered tasks.

Time Series Modules

We introduce efficient long‑sequence encoders and a dedicated forecasting branch for scientific time‑series.

During router training, Intern‑S2‑Preview‑397B and the Memory Decoder stay frozen while only the router is optimized on a mix of domain‑specific and general instruction data.

The encoder treats a long signal like a librarian who first groups books by genre (temporal chunks) and then compresses each shelf into a concise catalog (Q‑Former), enabling fast look‑ups while preserving detail.

Chunk 1 (steps 1‑4) is normalized, yielding per‑channel mean $\mu$ and std $\sigma$.

CNN extracts local features, producing a $2\times4$ feature map.

Q‑Former compresses the 4‑step patch into 2 tokens using learned queries.

Chunk 2 undergoes the same pipeline, yielding another 2 tokens.

The channel‑wise Transformer mixes the 2 channels across the 4 tokens, producing contextual embeddings.

Finally, the global Transformer processes the 4 tokens to generate the encoder output.

This pipeline keeps the representation size proportional to the number of chunks, not the raw length, enabling linear‑time processing of very long signals.

How does this encoder differ from the earlier mean‑pooling design?

The prior encoder collapsed each channel by simple averaging, discarding temporal structure and inter‑channel interactions. The upgraded encoder first preserves local temporal detail via patching, then explicitly models channel dependencies with a dedicated Transformer before global attention, retaining richer information while remaining efficient.

The generation module works like a weather forecaster who looks at both the current atmospheric map (encoder embeddings) and a textual briefing (LLM context) to predict future conditions, using a dedicated forecasting engine instead of writing numbers as prose.

Q‑Former extracts 2 semantic tokens from the instruction and 2 numeric tokens from the encoder.

Cross‑attention combines these 4 tokens, feeding them into the causal Transformer forecaster.

The horizon predictor sets the output length to 4 steps.

The forecaster generates 4 future tokens for each channel, which are then decoded back to numeric values.

Because the forecaster works on token embeddings rather than raw text, the predicted numbers retain the original signal’s precision.

Why not generate future values directly as text tokens like the rest of the model?

Text tokens introduce quantization and rounding errors that degrade numerical fidelity. By using a dedicated forecasting branch that operates on continuous embeddings, the model preserves the exact numeric values needed for scientific analysis.

**Figure 1.** (a) Structure of the time series encoder. (b) Structure of the time series forecaster.

Scientific Pre-training

Pre‑training adds visual understanding to scientific language models by interleaving image and text data.

Purely scaling text tokens discards the rich visual information present in scientific PDFs. Intern‑S2‑Preview therefore augments its foundation with a visual pathway that preserves layout and visual units, enabling multimodal reasoning.

Scientific Pre‑training fuses text and visual streams so the model learns from the full scientific document, not just extracted words.

Raster‑scan orders the retained tokens as $u_1=z_1$, $u_2=z_3$, $u_3=z_4$.

Project each $u_t$ with $W_{\text{in}}$ (e.g., $W_{\text{in}}u_t$) and feed the sequence into the LLM backbone.

The backbone produces hidden states $h_1,h_2,h_3$; the prediction head $\psi$ maps $h_3$ to $\hat{u}_4$.

Contrastive softmax over a batch $\mathcal{B}$ of size 2 yields $p_{33}=0.78$, contributing $-\log 0.78$ to $\mathcal{L}_{VP}$.

Combined with a text loss of $0.45$, the total loss $\mathcal{L}=0.6\,\mathcal{L}_{CE}+0.4\,\mathcal{L}_{VP}$ is back‑propagated.

Even a tiny raster‑scanned visual sequence provides a learning signal that aligns image patches with language model dynamics, without any manual annotation.

**Figure 3:** Overview of matched text and visual pre-training. The text pathway predicts tokens from parsed PDF content, whereas the visual pathway predicts foreground visual latents from rendered pages, improving alignment between textual and visual document representations.

Simple image‑caption pairs miss the richer context of scientific PDFs, where figures, equations, and tables are tightly woven into narrative flow. To capture this, we interleave visual units with surrounding text according to the original layout.

By ordering text blocks and visual units in true reading order, the model learns how images and equations function as integral parts of scientific arguments.

This figure illustrates the data processing pipeline, divided into two main stages: "Page-level processing" (steps 1-4) and "Document-level organization" (steps 5-8). - Step 1: PDF Layout Parsing & Structured Extraction using MinerU2.5-Pro. - Step 2: Visual Unit Cropping of images, equations, and tables. - Step 3: Page-level Interleaved Sequence Construction following reading order. - Step 4: Visual-Gain Filtering using perplexity (PPL) comparison. - Step 5: Document-level Concatenation of filtered pages. - Step 6: Long-Context Chunking with 256K token limit and 512-token overlap. - Step 7: Domain Focus on Life Science, Chemistry, and Materials Science. - Step 8: Final Interleaved Pre-training Corpus generation.

Building a large‑scale image vector database and retrieving high‑quality visual samples boosts the multimodal pre‑training signal.

**Figure 5.** The pipeline of the image retrieval process, including image encoding, vector database construction, and online text-to-image and image-to-image retrieval with post-processing.

How does Visual Pre‑training differ from conventional image‑caption pre‑training?

Standard image‑caption pre‑training predicts a textual caption from an image, treating the visual input as a single token. Visual Pre‑training instead raster‑scans the entire page, keeps the spatial order of visual patches, and predicts the next visual latent directly, learning a fine‑grained alignment between image structure and language model dynamics.

Why is visual‑gain filtering necessary instead of using all extracted pages?

Many pages contain decorative or unrelated graphics that add noise and increase training cost without improving language modeling. Visual‑gain filtering quantifies the actual contribution of visual content by measuring perplexity reduction, ensuring that only pages where visuals meaningfully aid text prediction are retained.

Agentic RL Framework

We describe a unified post‑training pipeline that adds reasoning, generation, and agentic capabilities.

The post‑training pipeline builds on the pretrained checkpoint to boost scientific reasoning, multimodal generation, and long‑horizon agentic behavior.

Agentic RL treats the model as an autonomous tool‑using agent that plans actions, observes tool feedback, and iteratively refines its policy to achieve long‑horizon goals.

How does Agentic RL differ from standard reinforcement learning?

Standard RL chooses actions from a fixed set and receives scalar rewards; Agentic RL expands the action set with external tool calls, treats tool outputs as observations, and optimizes a policy that reasons about both internal states and external effects.

Instead of waiting for every rollout to finish, we pause unfinished generations, keep their prefixes, and resume them after the policy update—so GPUs stay busy while still using all completed data for training.

After 3 tokens, the two long rollouts are paused; their prefixes “A‑B‑C” and “D‑E‑F” are saved.

The completed short rollouts (“G‑H‑I” and “J‑K‑L”) form a training batch; the GPU pool switches to policy update.

Policy parameters are updated; the new version is v₂.

Paused rollouts resume from their saved prefixes using v₂, generating tokens 4‑7.

All tokens now have an associated policy version (v₁ for tokens 1‑3, v₂ for tokens 4‑7) and log‑probabilities.

By keeping the prefixes, we reuse already‑spent compute, and the recorded policy versions let us correct for the mixture of policies during gradient estimation.

Why does pausing rollouts avoid the producer–consumer bottleneck?

Pausing lets the inference side keep generating new requests while the training side consumes completed trajectories; the GPU pool is never idle, and no rollout is aborted, so overall throughput improves without sacrificing data quality.

**Figure 6.** Overview of the post-training pipeline for Intern-S2-Preview. The pretrained base model is first enhanced through supervised fine-tuning, followed by multi-task RLVR and black-box agentic RL for general and specialized capability improvement. On-policy distillation then consolidates the resulting scientific reasoning and agentic capabilities into a single unified model.

RL Techniques and Stability

Core RL tricks that keep policy updates stable and reasoning efficient.

Policy mismatch and over‑long reasoning trajectories waste compute and destabilize gradients. We therefore need a way to correct stale rollouts while encouraging concise answers.

$\rho$ measures how much the current learner policy differs from the behavior policy that generated a token, letting us re‑weight that token’s contribution.

Compute $\rho = 0.2 / 0.1 = 2.0$.

The token’s gradient will be up‑weighted by a factor of 2, compensating for the under‑representation in the behavior data.

$\rho$ > 1 indicates the learner prefers the token more than the behavior policy, so the token should receive extra credit during update.

How does this differ from ordinary importance sampling used in off‑policy RL?

Ordinary importance sampling reweights entire trajectories, while here we apply the ratio per token because rollouts are partial and tokens may be mixed across trajectories. This fine‑grained weighting preserves gradient signal for every token.

We bound $\rho$ to stay near 1, preventing a few extreme ratios from exploding gradient variance.

Lower bound = $1-0.2 = 0.8$, upper bound = $1+0.5 = 1.5$.

Clip $2.0$ to the upper bound → $\bar{\rho}=1.5$.

The token’s gradient is up‑weighted by $1.5$ instead of $2.0$, reducing variance.

Clipping caps extreme ratios while still preserving a non‑zero update for every token.

Why not simply discard tokens with extreme ratios instead of clipping?

Discarding would remove gradient information entirely, whereas clipping retains a scaled contribution, keeping the learning signal alive for all tokens.

In PPO the clipping is applied to the surrogate loss, which can zero‑out gradients for out‑of‑trust‑region tokens; our weight‑clipping keeps every token alive.

During rollout we record which expert each token used; during training we replay the same expert choices so rollout and training see identical routes.

Replay of expert routing for a single token.

Is this just “store‑and‑reuse” the routing decisions?

Yes, but the key is that the stored decisions are used to align the forward pass of the training engine with the exact path taken during rollout, removing a source of stochastic mismatch.

We run expert linear layers in FP8, the rest in BF16, and keep numerically sensitive ops in FP32 to match the two engines.

We flag tokens whose probability distributions differ too much between rollout and training by measuring symmetric KL divergence.

Compute $D_{\text{BKL}}(0.8\|0.6)=0.8\log(0.8/0.6)+0.2\log(0.2/0.4)\approx0.045$.

Compute $D_{\text{BKL}}(0.6\|0.8)=0.6\log(0.6/0.8)+0.4\log(0.4/0.2)\approx0.045$.

Both divergences ≤ 0.05, so $m_{\text{BKL}}=1$ (token kept).

The symmetric check ensures that a token is only kept when the two engines agree both ways, not just one direction.

Why not use the standard (asymmetric) KL divergence instead of the bidirectional version?

Asymmetric KL can be small in one direction while large in the other, hiding a serious mismatch. The bidirectional version guarantees mutual consistency, which is crucial for stable RL updates.

We re‑weight advantages of correct answers to penalize unnecessary length, but only after the model has mastered the query.

How does this differ from adding an explicit length‑aware reward?

An explicit length reward adds a new term to the objective, which can conflict with the main task reward. Our method simply rescales the existing advantage, preserving the original reward structure while still discouraging wasteful steps.

**Figure 7.** Overview of our co-located partial-rollout system based on XTuner and LMDeploy. During rollout, completed requests are continuously replaced to keep the inference engine fully utilized. Once sufficient completed trajectories have been collected for one training batch, the remaining in-flight rollouts are paused and their generated prefixes are retained. The same GPUs then switch to policy training. After the training states are offloaded and the updated model weights are synchronized to the inference engine, the paused requests resume generation from their retained prefixes.

Speculative Decoding

Speculative decoding speeds RL rollouts by drafting tokens that the policy quickly verifies.

RL rollouts dominate training time because each reasoning trajectory is generated token‑by‑token. Even with a partial‑rollout system that improves GPU utilization, the autoregressive nature of long sequences remains a major slowdown.

Instead of waiting for the policy to emit every token, a lightweight draft model first proposes a batch of candidate tokens; the current policy then checks those candidates in parallel, accepting the ones that match its own distribution.

How does speculative decoding differ from standard beam search?

Beam search expands multiple hypotheses but still follows the policy’s probabilities; speculative decoding first generates candidates with a separate draft model and then *rejects* any that the policy would not have produced, guaranteeing the exact same distribution as pure policy sampling.

Draw a uniform random number $u\in[0,1]$; suppose $u=0.45$.

Compute cumulative policy mass over the draft list: $0.6$ (for $a$) > $u$, so token $a$ is accepted.

Tokens $b$ and $c$ are discarded because the policy would have sampled them with lower probability than $u$.

The accepted token $a$ is emitted, and the rollout proceeds to the next state $s_{t+1}$.

Speculative decoding can accept a draft token in a single parallel check, turning a sequential generation step into a constant‑time verification when the draft’s proposal aligns with the policy.

**Figure 8.** Comparison of training with and without adaptive length penalty, showing similar reward curves and shorter outputs with regularization.

Multi-Task Optimization

Adaptive draft training and entropy‑aware policy updates enable stable multi‑task RL.

Speculative decoding relies on a draft model that must stay aligned with the evolving policy. When the draft drifts, gradients become noisy and rollout speed collapses, which is especially problematic for heterogeneous RL tasks.

The draft model is trained online with a loss that smoothly shifts from KL‑driven smoothing to TV‑driven acceptance‑rate maximization as the draft becomes better aligned.

Compute KL: $D_{KL}$(p‖q)=0.6·log(0.6/0.5)+0.4·log(0.4/0.5)≈0.020.

Compute TV: $D_{TV}$=½·(|0.6‑0.5|+|0.4‑0.5|)=0.1.

$\lambda$₁ = exp(‑$\eta$·sg[0.9]) with $\eta$=3 → $\lambda$₁≈exp(‑3)≈0.05.

Hybrid loss 𝓛(·,1) = 0.05·0.020 + (1‑0.05)·0.1 ≈ 0.095.

When the draft already matches the target (high overlap), the KL term becomes negligible and the loss is driven almost entirely by the TV term, directly encouraging higher acceptance.

Why not use only the KL term for draft training?

KL alone yields smooth gradients but does not directly reward the acceptance probability that speculative decoding cares about. As the draft improves, KL gradients become tiny, slowing further alignment. The TV component explicitly measures overlap, so weighting it more when acceptance is high accelerates the final speedup.

GEPO treats each prompt group as its own entropy regime and scales advantage signals so that low‑entropy groups are not over‑exploited while high‑entropy groups retain enough exploratory pressure.

Compute log‑probabilities: A = log 0.9 + log 0.8 + log 0.7 ≈ ‑0.58; B = log 0.4 + log 0.5 ≈ ‑1.79.

Average negative log‑probability: $H_g$ = -(1/2)·(‑0.58 + ‑1.79) ≈ 1.185.

Low $H_g$ indicates the group is confident (low entropy); GEPO will apply a milder attenuation to its positive advantages.

Even though Prompt B has higher per‑token uncertainty, the group‑level average smooths these differences, allowing GEPO to treat the whole group uniformly.

How does GEPO differ from standard entropy regularization that applies a single temperature to the whole batch?

Standard regularization uses one scalar entropy term for all samples, assuming a homogeneous entropy landscape. GEPO computes entropy per prompt group, then scales advantages asymmetrically: low‑entropy groups receive weaker positive‑advantage scaling, preventing over‑exploitation, while high‑entropy groups receive weaker negative‑advantage scaling, preserving exploration. This group‑aware treatment respects the heterogeneous nature of scientific RL tasks.

RL Objective Functions

Defines the unified RL objective by shaping advantages to balance exploration and encourage concise reasoning.

Heterogeneous reasoning tasks cause uneven exploration, destabilizing policy updates. GEPO rescales each task’s advantage based on its entropy, while adaptive length regularization penalizes overly long successful trajectories. The resulting advantage feeds the REINFORCE loss.

GEPO tempers a task’s raw advantage by how uncertain its policy is: low entropy (over‑confident) tasks are shrunk, high entropy (still exploring) tasks are kept larger.

Task 1: $A_1>0$ and $\mathcal{H}_g=0.3<0.5$ ⇒ apply $\alpha_{\text{low}}$: $\hat{A}_1=0.3\times0.8=0.24$.

Task 2: $A_2<0$ and $\mathcal{H}_g=0.7>0.5$ ⇒ apply $\alpha_{\text{high}}$: $\hat{A}_2=0.7\times(-0.4)=-0.28$.

Task 3: $A_3>0$ but $\mathcal{H}_g=0.4<0.5$ ⇒ apply $\alpha_{\text{low}}$: $\hat{A}_3=0.3\times0.2=0.06$.

Resulting entropy‑adjusted advantages are $\hat{A}=[0.24,\ -0.28,\ 0.06]$.

GEPO reduces the influence of over‑confident positive advantages while still allowing negative signals to contribute, yielding a more balanced gradient across heterogeneous tasks.

How does GEPO differ from ordinary entropy regularization that simply adds a penalty term?

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers