Moving Towards Latent Space Language Models Through Next Concept Prediction

The Intern-NCP Team, Jiaqi Cao, Chiyu Chen, Shuang Cheng, Xu Cheng, Beiya Dai, Yufan Feng, Kewen Ge, Ruijun Ge, Jiayi Huang, Yang Jiao, Dahua Lin, Zhouhan Lin, Yifan Liu, Yuliang Liu, Biqing Qi, Mowen Ruan, Junzhe Shen, Yunchong Song, Hao Sun, Zhongbo Tian, Yixuan Wang, Rubin Wei, Jiaxin Xiong, Kangyu Yang, Qian Yao, Qi Zhang, Bowen Zhou

NCP-ArchPreview integrates discrete concept-level prediction into language modeling to improve training efficiency and downstream reasoning.

Can we improve language model training efficiency and downstream performance by forcing the model to predict discrete latent concepts alongside standard next-token prediction?

Standard language models rely on next-token prediction, which forces the model to learn high-level semantic abstractions only as an indirect byproduct of granular token supervision. NCP-ArchPreview introduces a latent-space architecture that explicitly predicts discrete concepts spanning multiple tokens, using a learned product-quantized vocabulary to guide token-level generation. This approach achieves a 1.95× convergence speedup in training tokens and outperforms the baseline OLMo-3-7B on downstream benchmarks, including a 5.99-point gain on GSM8K.

Paper Primer

The model architecture inserts a Concept Module between a Token Encoder and a Token Decoder. It compresses token groups into continuous representations, quantizes them into a discrete concept vocabulary, and uses the Concept Module to predict future concepts, which are then injected back into the token stream to refine generation.

NCP-ArchPreview significantly improves training efficiency and downstream performance compared to standard next-token prediction models.

The model matches the final pretraining loss of the 7B-parameter OLMo-3 baseline using only 51.3% of the training tokens and achieves a 2.45-point higher macro-average on downstream tasks. 1.95× faster convergence in training tokens and a 5.99-point gain on GSM8K.

The learned latent space provides a lightweight interface for domain adaptation. Updating only the 17M-parameter Vector Quantization (VQ) module and prediction heads allows for effective adaptation to new domains with higher throughput and lower memory usage than full-parameter fine-tuning or LoRA.

Why is predicting concepts better than just predicting more tokens?

Standard token-level objectives lack explicit guidance for semantic structure across multi-token spans. By predicting discrete concepts, the model gains a structured, latent-level objective that captures dependencies spanning multiple tokens, which improves both training efficiency and reasoning capabilities.

Does the performance gain come from the extra parameters or the architecture?

Controlled ablations confirm the gains stem from the latent hierarchy and the Next Concept Prediction (NCP) objective. The model outperforms computation-aligned baselines and approaches the performance of size-aligned models while using only 85% of the computation.

Introduction and Motivation

Motivating the need for concept-level prediction beyond token-level language modeling.

Standard next‑token prediction (NTP) forces the model to learn high‑level semantics only indirectly, which makes training slow and limits reasoning capabilities. To address this inefficiency, we propose augmenting the pretraining objective with Next Concept Prediction (NCP), an explicit task that asks the model to predict discrete concepts spanning multiple tokens.

Token‑only supervision provides a weak signal for multi‑token semantics, causing slow convergence and suboptimal downstream reasoning.

Scaling this architecture to 8.9 B parameters and training on 5.73 T tokens lets NCP‑ArchPreview reach the final pretraining loss of the 7 B‑parameter OLMo‑3 baseline while consuming only 51.3 % of its training tokens, a 1.95× speed‑up.

Beyond pretraining efficiency, the model improves downstream performance by 2.45 macro‑average points and adds a 5.99‑point boost on GSM8K, demonstrating that concept‑level supervision benefits reasoning tasks.

The learned latent space remains useful after pretraining: updating the 17 M‑parameter VQ module enables lightweight domain adaptation, and injecting concept representations into a DFlash2 drafter raises mean accepted length by 4.17 % with negligible overhead.

The inefficiency of token‑only prediction motivates adding explicit concept prediction to accelerate convergence and enhance reasoning.

NCP-ArchPreview Architecture

We augment token‑level modeling with a discrete concept‑level prediction to capture high‑level semantics.

Standard token‑level prediction struggles to capture abstract semantics, so we introduce a latent‑space prediction that runs alongside the usual token stream.

The model interleaves a token encoder, a concept module, and a token decoder, letting a discrete “concept” stream guide token generation.

Mean‑pool tokens $(t_1,t_2)$ → concept $c_1$, $(t_3,t_4)$ → $c_2$, $(t_5,t_6)$ → $c_3$, $(t_7,t_8)$ → $c_4$.

Each concept $c_m$ is a $d$‑dimensional vector obtained by averaging the two token hidden states.

The Concept Module predicts the next concept $\hat{c}_5$ (which will be repeated $k$ times later).

During decoding, the predicted concept $\hat{c}_5$ is injected back into the token stream after a causal shift of $\Delta=k$.

The Token Decoder now sees both token‑level hidden states and the injected concept to produce the next token.

Mean‑pooling reduces sequence length, allowing the model to reason over higher‑level units while still preserving token‑level detail via residual connections.

The module compresses token groups into a discrete latent space and predicts the next latent “concept” by blending learned codebook entries.

How does this concept prediction differ from a standard token‑level next‑token prediction?

Token‑level prediction directly maps the current token hidden state to the next token distribution, while the Concept Module first compresses a group of tokens into a discrete latent entry and predicts the next latent entry; the decoder then conditions on both the token hidden state and the injected latent concept, enabling higher‑level semantic guidance.

IRC mixes representations from multiple depths within a module, and CRC shuttles information between modules, so the model can reuse early features while still learning deep abstractions.

Forward pass of NCP‑ArchPreview

**Figure 2** | Overview of NCP-ArchPreview. The token-level module (comprising the Token Encoder and Token Decoder) maintains token-level representations, ensuring the overall input and output remain consistent with canonical autoregressive decoding. In the intermediate layers, the Concept Module compresses token spans into concepts and predicts the next concept via weighted combinations of entries from a vector-quantized concept vocabulary, which is learned from the Token Encoder’s hidden states. Hierarchical residual connections, including Cross-Module Residual Connections (CRC) and Intra-Module Residual Connections (IRC), allow information to flow smoothly across layers and abstraction levels.

The image displays three sequential data flow relationships: - TokenEncoder $arrow$ ConceptModule - TokenEncoder $arrow$ TokenDecoder - ConceptModule $arrow$ TokenDecoder

Training Procedure

Training aligns token and concept pathways through a balanced multi‑loss curriculum.

The model is trained end‑to‑end with three cooperating objectives, each shaping a different part of the architecture.

Three losses—token, concept, and codebook—are combined with scalar weights so the model learns to predict tokens while simultaneously shaping discrete latent concepts.

Compute the weighted NCP contribution: $0.5 \times 0.45 = 0.225$.

Compute the weighted VQ contribution: $0.1 \times 0.30 = 0.03$.

Sum all three terms: \$1.20 + 0.225 + 0.03 = 1.455$.

The total loss $L_{\text{total}}$ for this step is $1.455$, which is back‑propagated through all shared parameters.

Even modest coefficients can noticeably shift the gradient balance, illustrating why careful tuning of $\alpha$ and $\beta$ is essential for stable training.

How does this curriculum differ from naïve multitask training that simply adds token and concept losses?

In naïve multitask, the losses are summed with equal weight, which can drown the weaker concept signal. Here the explicit coefficients $\alpha$ and $\beta$ let the trainer amplify or attenuate the auxiliary objectives, ensuring the codebook learns a useful discrete space while the concept predictor receives a clear gradient signal.

Experimental Results

Key empirical gains of NCP‑ArchPreview over the OLMo‑3 baseline.

This section quantifies how the auxiliary next‑concept prediction objective reshapes training dynamics and downstream capability relative to the vanilla OLMo‑3 family.

NCP‑ArchPreview converges 1.95× faster than OLMo‑3‑7B while attaining a lower final loss.

Stage‑1 loss curves (Figure 1) show the NCP model reaching OLMo‑3’s final loss after only 51.3 % of the tokens.

**Figure a.** Stage-1 loss curves for OLMo-3-7B and NCP-ArchPreview across all 5.73T pretraining tokens. Despite trained on exactly the same data, NCP-ArchPreview converges 1.95x faster and achieves a 0.091 lower loss over the final steps.

**Figure 4** | The scaling law curves for OLMo-3 and NCP-ArchPreview. Each marker denotes the best validation loss found after searching training hyperparameters and model/data allocations at a fixed FLOPs budget. Dashed curves show the corresponding scaling fits.

OLMo‑3 is the standard dense Transformer model that predicts the next token using only token‑level attention, without any explicit concept‑level supervision.

Why does adding a Concept Module improve performance if the token‑level backbone already sees all tokens?

Because the Concept Module aggregates four consecutive token states into a single latent representation, providing a higher‑level abstraction that the NCP loss can target directly, which speeds up learning of semantic structure.

Scaling laws describe how validation loss decreases as a power of compute (FLOPs); a steeper curve means a model extracts more capability per unit of compute.

How does the 1.74× efficiency reported in the scaling‑law plot differ from the 1.95× convergence speedup?

The 1.95× figure measures token‑level progress (how many tokens are needed to reach a given loss), whereas the 1.74× figure measures compute‑level progress (how many FLOPs are needed to reach the same loss). Both reflect faster learning but at different resource granularities.

Across training stages, NCP‑ArchPreview delivers roughly double‑speed convergence and a 1.7× compute‑efficiency gain, confirming that the auxiliary concept objective accelerates learning without extra compute.

Related Work

We situate our approach among recent abstract, hierarchical, and residual innovations in language modeling.

Recent work shifts from reconstructing raw tokens to predicting higher‑level representations. Joint‑embedding predictive architectures learn to forecast latent targets that preserve semantic structure while discarding unnecessary input details. Large Concept Models extend this idea to whole sentences, mapping them into a shared continuous embedding space.

Hierarchical models introduce a compressed high‑level sequence over groups of fine‑grained units, as seen in the Hourglass Transformer and MegaByte. Other approaches such as BLT, H‑Net, ContextLM, and DLCM modify tokenization or dynamically chunk inputs to form latent units. These methods primarily adjust computational granularity rather than the prediction objective.

Recent residual designs aim to selectively reuse earlier representations. DenseFormer averages block outputs with learned weights, while DeepCrossAttention and MUDDFormer employ input‑dependent weighting schemes. NCP‑ArchPreview builds on MUDDFormer’s single‑stream formulation as Intra‑Module Residual Connections and adds Cross‑Module Residual Connections for target‑conditioned depth selection across encoder, concept module, and decoder.

The current study focuses on NCP‑ArchPreview after extensive pre‑training at standard context lengths; extending to long‑context training remains future work. Because the concept‑level pathway operates on a compressed sequence, longer contexts could especially benefit latent‑space modeling.

NCP‑ArchPreview maintains a token‑level language‑modeling loss advantage while delivering downstream gains, with larger improvements after pre‑training than mid‑training. This mirrors recent observations that reductions in cross‑entropy loss do not uniformly translate to downstream performance, highlighting a nuanced loss‑to‑capability relationship.

Overall, latent‑space language modeling at trillion‑token scale shows that joint token and concept prediction can achieve comparable loss with fewer tokens and improve downstream metrics. This establishes a scalable blueprint for future models that combine token‑level and concept‑level objectives.

Data and Proxy Analysis

We compare three Stage‑2 data recipes using proxy metrics to identify the strongest recipe without full evaluation.

Before screening new mid‑training recipes we first define a fixed held‑out capability set and a set of lightweight proxy metrics that can be computed with a single forward pass.

The architecture of the NCP‑ArchPreview model (Table 10) consists of a token encoder, a dedicated Concept Module, and a token decoder, each built from causal Transformer layers with hierarchical residual routing.

Algorithm 1 – End‑to‑end training of NCP‑ArchPreview with hierarchical residual routing

Proxy metrics are computed as per‑capability negative log‑likelihood (NLL) for free‑form tasks and as task‑native choice margins for multiple‑choice tasks, then aggregated source‑balanced to avoid domination by large benchmarks.

Across all six free‑form capability leaves the ordering V1 < V2 < V3 in proxy NLL aligns with the ordering in downstream scores, confirming that the proxy metrics reliably screen recipes; V1 emerges as the strongest recipe.

Scaling Ladder Experiments

We detail how scaling‑ladder experiments allocate FLOPs across model size and data.

We construct scaling‑ladder experiments by fixing several FLOPs budgets and varying the split between model capacity and training tokens within each budget.

The experiment treats total compute as a fixed budget and explores how different allocations of model size versus data affect performance.

Why not simply increase model size instead of adjusting token count?

Because the total compute $C$ is fixed; enlarging the model reduces the number of tokens $D$ you can afford, and the ladder isolates the contribution of each factor.

**Figure 1.** Scaling laws for downstream performance as a function of pre-training loss.

Table 12 enumerates the exact configurations for each point in Figure 10, listing hidden width $H$, depths $L_e/L_c/L_d$, FLOPs per token $F_{\text{tok}}$, token count $D$, and the resulting validation loss.

Evaluation Protocols and Results

Evaluation protocol details and key results across benchmarks.

Standard next‑token prediction (NTP) struggles to capture high‑level semantics, so NCP‑ArchPreview adds a next‑concept prediction (NCP) objective to speed convergence and boost reasoning.

Table 1 groups benchmarks into higher‑is‑better task scores (e.g., MMLU, MATH, Code) and lower‑is‑better likelihood scores (bits per byte, BPB).

All few‑shot evaluations use a fixed random seed of 42, 5‑shot prompts, and identical templates across models; scores are macro‑averaged per domain and then unweighted across the 26 listed benchmarks.

**Table 1.** Downstream performance grouped by task domain. Vanilla denotes the released OLMo-3 model for each stage. Metrics above the likelihood block are reported as percentages and are higher-is-better. The likelihood block reports bits per UTF-8 byte (BPB), where lower is better. The BPB average is computed separately and is excluded from Overall AVG.

**Table 14.** NCP-ArchPreview stage-1 downstream performance across training checkpoints. The reported benchmarks follow the same grouping as Table 1. The likelihood block reports BPB (lower is better) and is aggregated separately from the higher-is-better Overall AVG.

**Figure 3** | Training-loss ablation of NCP-ArchPreview over the first 200B training tokens. All runs use the same training recipe. The six curves correspond to three standard OLMo-3-7B baselines Vanilla, Vanilla computation-aligned, and Vanilla size-aligned (+15% computation) and three progressive configurations Vanilla+CM, Vanilla+CM+Residual, and Vanilla+CM+Residual+NCP, with the last corresponding to NCP-ArchPreview. The inset enlarges the final 8k training steps to better resolve the differences among the configurations.

Author Contributions

Lists all authors, their specific contributions, and provides the model’s full configuration and training pseudocode.

The author roster is presented alphabetically, comprising thirty‑four contributors ranging from Jiaqi Cao to Bowen Zhou.

Contributions are grouped by functional area—Architecture Design, Architecture Optimization, Model Training, Scaling Ladder, Ablation and Analysis, Inference and Deployment, Evaluation, Data Analysis, Infrastructure, Project Leadership, and Project Advisorship—each listing the responsible authors.

**Table 10.** Architecture configuration of NCP-ArchPreview.

Algorithm 1 (referenced in the text) details the end‑to‑end forward pass, employing intra‑module residual connections (IRC) and cross‑module residual connections (CRC) as defined in Equations 14–20, and aligning concept‑level with token‑level states via the causal shift operator.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers