ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search
Jiyan He, Guang Liang, Hao Liu, Haoxiang Guan, Jinbo Sun, Junyi Guo, Wenjun Feng, Yantai Xie, Yifei Shen, Bin Shao, Chuyang Wei, Kai Chen, Kexin Zhou, Minghang Zhu, Shuxin Zheng, Tie-Yan Liu, Taine Zhao, Wenhui Zhu, Xueyin Xu, Xiaoqing Zhang, Yatao Li, Yuxuan Ren
A fully open 7B foundation model that achieves frontier-grade reasoning and agentic search via system-algorithm co-design.
How can a 7B-parameter foundation model achieve frontier-level mathematical reasoning and agentic search capabilities while remaining fully open-source and compute-efficient?
Frontier-grade reasoning and agentic search are currently locked behind massive, proprietary models, leaving compute-constrained researchers unable to study or replicate these capabilities. ZGCM-1 overcomes this by coupling a compact 7B architecture with a dual engine of deliberate internal thinking and active external tool use, optimized through a high-efficiency open training recipe. The model achieves competitive performance against frontier systems orders of magnitude larger, while providing a 4.2× improvement in pre-training time-to-loss.
Paper Primer
The authors argue that compact models are inherently limited by static parametric capacity but can transcend this by shifting from passive web memorization to active, tool-augmented reasoning. The core mechanism hinges on a hybrid attention architecture—interleaving gated sliding-window and global attention—paired with a stable FP8 Muon optimizer and a curriculum that progressively scales context from 16K to 256K tokens.
ZGCM-1-7B achieves competitive performance on reasoning and agentic benchmarks against models orders of magnitude larger.
The model scores 97.1% on MATH-500 and 62.0% on Binary Function Search, outperforming or matching frontier models like GPT-4o and GLM-5.1 in specific agentic tasks. Competitive parity with 235B+ parameter systems at the 7B scale.
The training pipeline delivers a ~4.2× speedup in pre-training time-to-loss.
Efficiency gains are derived from the cumulative impact of hybrid attention (1.4×), FP8 mixed-precision systems (1.5×), the Muon optimizer (1.8×), and Pre-LN (1.1×).
Why does this paper prioritize "open" training recipes over just releasing model weights?
The authors identify an "opacity barrier" where proprietary black-box training schedules and data mixtures prevent the academic community from systematically studying training dynamics and capacity limits.
How does the model handle the memory bottleneck of 256K context lengths?
The hybrid attention architecture uses a 5:1 ratio of gated sliding-window attention to global attention, which reduces the per-token KV-cache footprint by 6.4× compared to standard full attention.
Introduction and Motivation
We expose the barriers to frontier AI and introduce an open, efficient 7B model that overcomes them.
Foundation models have pushed the frontier of long‑horizon reasoning and tool‑augmented agency, yet most researchers cannot access these capabilities because training at the required scale demands prohibitive compute and the released models conceal their training recipes.
ZGCM‑1 is a fully open 7B dense foundation model trained from scratch using an ultra‑efficient recipe that combines interleaved gated sliding‑window and full attention with a stable FP8 Muon optimizer, and a progressive curriculum that scales context length from 16K to 256K via an MDP‑style mid‑training regime.
The need for open, efficient foundation models is essential to democratize frontier‑level reasoning and agentic capabilities.
The ZGCM-1 Architecture
We detail the hybrid attention and FP8 pre‑training tricks that enable efficient 256K‑context training.
Training large models at 256K context is prohibitively expensive on academic hardware, so we must cut both compute and memory while preserving quality.
We interleave gated sliding‑window attention (SWA) with occasional global attention in a 5:1 ratio, letting most layers look only at a short local window while a few layers retain full‑sequence information.
Layer 1 (SWA): each token attends to its own token plus the next $3$ tokens, producing $4$‑token KV entries.
Layer 2 (SWA): the same $4$‑token window slides one position forward; token 0 now sees tokens 0‑3, token 1 sees 1‑4, etc.
Layer 3 (SWA): window slides again; memory usage stays at $4$ entries per token.
Layer 4 (SWA): final local window; still $4$ entries per token.
Layer 5 (Global): each token attends to all $8$ tokens, storing a full $8$‑token KV cache for this layer only.
Result: total KV entries = $4$ (per SWA layer) × 4 + 8 (global) = $24$ entries versus $8$ × 5 = $40$ entries for full attention.
The local‑window layers give most of the representational power while keeping memory linear in the window size, and the occasional global layer restores long‑range dependencies.
How does this hybrid schedule differ from a plain sliding‑window attention that never uses global layers?
Pure sliding‑window attention can never propagate information beyond the window, so distant tokens remain isolated. By inserting global layers every fifth step, the model periodically mixes all tokens, enabling long‑range reasoning without the full $O(n^2)$ cost.
We train the model using 8‑bit floating‑point (FP8) arithmetic together with the Muon optimizer and TWEO outlier regularization, cutting precision‑related overhead while keeping gradients stable.
Muon scales the gradient by a factor $0.9$ to stay within the FP8 exponent range, yielding $G' = [0.0009, -0.0018; 0.0027, -0.0009]$.
TWEO detects the $-0.0018$ outlier (magnitude > $2\times$ median) and clips it to $-0.0010$.
Update $W \leftarrow W - \eta\,G'$ (with learning rate $\eta=0.01$), producing $W_{\text{new}} = [0.1191, -0.0801; 0.0527, 0.0899]$.
All intermediate values are stored back in FP8, preserving the reduced memory footprint.
Even with aggressive quantization, the optimizer’s scaling and outlier clipping keep updates meaningful, avoiding divergence that plain FP8 would cause.
Why can’t we simply switch a BF16 training pipeline to FP8 without the Muon optimizer and TWEO?
BF16 has a larger mantissa and exponent, so it tolerates a wider range of activation magnitudes. FP8’s tiny exponent range would overflow on the same gradients, and without Muon’s adaptive scaling the optimizer would take steps that are either too large (causing NaNs) or too small (stalling learning). TWEO further protects against rare spikes that would otherwise dominate the limited precision.
**Figure 2.** Overview of ZGCM-1. Left: benchmark performance across reasoning and agentic tasks compared with 7B-scale and frontier models. Right: four key technical highlights: (1) FP8 pre-training with Muon and TWEO achieving ~4.2x time-to-loss speedup, (2) hybrid sliding-window/global attention, (3) MDP mid-training that reformulates interaction traces into state-action supervision with context scaling to 256K, and (4) AI-native R&D where each researcher directs agent swarms across the full development lifecycle. BFS denotes Binary Function Search. Qwen3-8B-Distill denotes DeepSeek-R1-0528-Qwen3-8B. Full results and evaluation protocols appear in Section 5.
**Figure 3** Hybrid attention architecture of ZGCM-1. The right panel shows the backbone formed by gated sliding-window and global attention layers; the left panel details the gated sliding-window attention module and its corresponding attention masks. The last global layer is layer 29 (0-indexed), followed by two trailing SWA layers.
**Figure 4.** Architecture experiments. (a) Quality-throughput trade-off among non-parameter-matched 7B attention configurations trained for 10B tokens (lower loss and higher throughput are better). (b) Throughput speedup of SWA schedules over full attention as context length grows from 4K to 256K. (c) KV cache memory footprint of three iso-parameter architectures (batch = 1, bf16): full attention (32 global layers), linear attention (GDN 3:1, 7 global + 21 linear, L=28), and our hybrid SWA 5:1 (5 global + 27 SWA).
Pre-Training Phases
Two‑phase pre‑training builds broad capabilities then refines them with longer contexts.
Pre‑Training proceeds in two ordered phases: a broad “General” stage followed by a more focused “Mid‑Training” stage that stretches the context window.
Think of constructing a building: first lay a solid foundation that supports any floor plan (General stage), then add specialized upper floors that require longer spans and richer utilities (Mid‑Training).
Run each candidate through the proxy model for one epoch and record the average loss on a held‑out validation set.
Candidate 2 yields the lowest loss (1.84) while Candidate 1 records 1.92 and Candidate 3 records 1.89.
Select Candidate 2 as the final mixture for Stage 1 because it balances loss reduction with domain diversity.
The proxy model dramatically cuts the cost of mixture exploration—evaluating three candidates takes minutes instead of the weeks required at full scale.
Tokenize all source documents with the GLM‑5.1 tokenizer.
Run General Pre‑Training Stage 1 on the web‑dominant mixture (≈0.99 T tokens).
Run General Pre‑Training Stage 2, adding code, mathematics, and specialized reasoning data (≈3.20 T tokens).
Enter Mid‑Training Stage 1: causal LM with 16 K context, injecting denser instruction data.
Mid‑Training Stage 2: increase context to 64 K tokens, further enrich reasoning data.
Mid‑Training Stage 3: extend context to 256 K tokens, finalizing agentic capabilities.
**Figure 5.** General Pre-Training is organized into two data stages, followed by three Mid-Training stages with progressively longer context lengths.
**Figure 6.** Data mixture composition shown as donut charts for the two General Pre-Training stages and the three Mid-Training context stages. Wedge areas represent each category's share of the corresponding stage's total tokens.
Curriculum Pre-Training
We detail the curriculum ordering and hyperparameter choices that enable fast, stable pre‑training.
Curriculum pre‑training reshapes the data stream so that easier language examples appear first, reducing wasted compute on hard samples.
We sort general‑language documents by increasing lexical complexity, while keeping code and mathematics separate, so the model sees progressively harder language but never mixes difficult code or math with the ordering.
Sort the non‑code documents by score → order: D₁, D₂, D₃, D₄.
Filter out any document with score > 10 that is also flagged as corrupted; D₄ is removed.
Interleave the remaining ordered documents with code and math: D₁, C₁, M₁, D₂, C₂, M₂, D₃.
Training proceeds over this mixed stream, exposing the model first to easy language, then gradually harder language, while code and math appear at constant difficulty.
Lexical ordering improves language modeling efficiency, but without the filter it would introduce noisy high‑complexity text that harms training.
How does this curriculum differ from random shuffling of the corpus?
Random shuffling presents hard and easy examples uniformly, causing the optimizer to waste steps on difficult samples before it has learned basic language patterns. The curriculum orders easy language first, letting the model acquire a solid foundation before tackling harder text, which yields the observed BPB gains.
Curriculum ordering reduces token‑per‑byte (BPB) for coding and mathematics while only modestly increasing general‑domain BPB.
Coding BPB drops from 1.99 to 0.81, mathematics BPB from 0.97 to 0.94, and general‑benchmark BPB rises by 0.03–0.09.
The next part specifies the optimizer, precision format, and regularizer that make the large‑scale pre‑training stable and fast.
Muon augments standard momentum with spectral scaling and a few Newton–Schulz steps, giving faster convergence than Adam while keeping a simple constant learning rate.
Why not simply replace Adam with a larger learning rate?
Increasing the learning rate alone would cause instability because gradients are not rescaled to account for curvature. Muon’s spectral scaling and Newton–Schulz refinement adaptively condition the update, achieving higher efficiency without sacrificing stability.
Matrix multiplications run in ultra‑low‑precision FP8 (E4M3 forward, E5M2 backward) while TWEO caps extreme activation values, preventing overflow and keeping training stable.
How does FP8 differ from the more common FP16/BF16 mixed‑precision schemes?
FP8 uses only 8 bits per value (E4M3/E5M2), cutting memory and bandwidth dramatically. However, its reduced dynamic range makes it vulnerable to outliers, which TWEO mitigates. FP16/BF16 retain more range and typically need no extra regularizer.
TWEO regularization is essential for stable FP8 pre‑training, enabling high throughput.
With TWEO, the system sustains ~585 TFLOP/s/GPU (≈60 % BF16‑equivalent MFU) without numerical divergence.
The co‑design of SWA, FP8, Muon, and Pre‑LN yields a ~4.2× speedup in 16K pre‑training time‑to‑loss versus a BF16/AdamW baseline.
Multiplying the individual gains (1.4× SWA, 1.5× FP8, 1.8× Muon, 1.1× Pre‑LN) gives ≈4.2× overall acceleration.
Mid-Training Strategy
Mid-Training stages expand context length while preserving short‑range capabilities.
Large‑scale pre‑training wastes tokens on duplicated content and struggles to acquire long‑range reasoning. Mid‑Training tackles this by progressively widening the context window while replaying earlier data to keep short‑range skills alive.
Instead of a single, static curriculum, the model is fed three increasingly long context stages that overlap—so it learns to handle short snippets, medium documents, and ultra‑long passages without forgetting earlier abilities.
Stage‑A trains on the first 30 tokens, each sequence limited to 16 tokens.
Stage‑B re‑uses the same 30 tokens (so short‑range skills persist) and additionally trains on the next 20 tokens, allowing sequences up to 64 tokens.
Stage‑C re‑uses the 50 tokens from earlier stages and adds the last 10 tokens, enabling sequences up to 256 tokens while still seeing the short‑range examples.
Throughout, the proportion of “code‑like” tokens stays at 20 % (12 of 60), while “knowledge‑like” tokens increase from 6 (10 %) in Stage‑A to 9 (15 %) in Stage‑C.
Because each stage re‑plays earlier tokens, the model never loses ability on short contexts even after learning to process very long documents.
How does Mid‑Training differ from a single‑stage curriculum that simply increases the context length at the end?
In a single‑stage approach the model would see only ultra‑long sequences after the switch, discarding the short‑context examples that anchor skills like code completion. Mid‑Training keeps the short‑context data in every later stage, so the model continuously practices both short and long contexts, preventing catastrophic forgetting.
All incoming reasoning examples are forced into a common schema and then sorted into five buckets—accepted, rewrite, pending, holdout, and rejected—so the training pipeline can treat them uniformly.
Why not simply keep all raw reasoning traces instead of routing them into separate views?
Raw traces often contain incomplete reasoning, duplicated content, or hidden references to the original source, which can leak proprietary data or confuse the model. By normalizing and routing, the pipeline filters out these problems and guarantees that only clean, self‑contained examples contribute to training.
**Figure 7.** Mid-Training data processing pipeline.
Data Governance and Monitoring
We describe how agentic data and continuous monitoring steer Mid‑Training toward reliable capabilities.
Training massive language models is notoriously opaque: developers cannot see which data or hyper‑parameter choices drive capability gains, and subtle system‑level regressions can go unnoticed until they cause costly failures.
We turn raw interaction traces into Markov‑decision‑process‑style next‑action prediction examples, giving the model dense supervision for planning, tool use, and iterative refinement.
State $s_0$ = “list files in /tmp”.
Action $a_0$ = “/tmp contains a.txt, b.log”.
Next state $s_1$ = model’s internal representation after emitting $a_0$ (includes the file list).
Predict $a_1$ = “What would you like to do next?” (next‑action prediction target).
This reformulation supplies a supervised signal for each decision point, enabling the model to learn fine‑grained planning without waiting for full‑sequence back‑propagation.
How does agentic data differ from the standard instruction data collected earlier?
Instruction data are static prompts paired with a single response, suitable for next‑token prediction. Agentic data, by contrast, encodes a sequence of states and actions, turning each interaction step into a supervised next‑action target, which provides much richer guidance for sequential decision making.
We continuously evaluate checkpoints with a fixed capability suite and track low‑level training signals, so that regressions are caught early and stage transitions are timed based on empirical progress.
**Figure 9** Token-level cross-entropy (CE) loss (blue, left axis) and learning-rate schedule (terracotta, right axis) for the staged Mid-Training run. The solid blue curve is a 200-point moving average, while the light-blue curve shows raw loss values. Both metrics share the cumulative-token axis; vertical dotted lines mark transitions between the 16K, 64K, and 256K context stages.
**Figure 8** AI-driven self-iterating data governance pipeline. Each source family is routed and sampled independently, then processed through source-specific rule filtering, strong-model audit, failure mining, human spot checks, iterative script revision, and held-out validation before accepted examples enter the validated reasoning-data pool.
A concrete monitoring incident illustrates the pipeline’s value: after resuming a job, throughput fell from 585 to 554 TFLOP/s/GPU over 80 steps. Inserting a periodic CUDA allocator cache clear every 100 iterations restored the original 585 TFLOP/s/GPU, and this fix was baked into the production loop.
Supervised Fine-Tuning Pipeline
Describes the curated supervised fine‑tuning pipeline and its mixed think/no‑think objective.
Raw SFT corpora are noisy and massive; training on them without careful curation yields diminishing returns despite billions of tokens.
The pipeline turns a heterogeneous raw corpus into a high‑quality, de‑contaminated training set by normalizing, filtering, balancing capability coverage, and packing examples into assistant‑only loss masks.
Start: 10 raw examples.
Heuristic filter removes 2 → 8 remain.
Tiered scoring discards 2 low‑tier → 6 remain.
Decontamination removes 1 overlapping example → 5 final examples.
Pack the 5 examples into a single training sequence with assistant‑only loss mask.
The example shows that aggressive quality filtering can halve the raw pool yet still produce a compact, high‑utility training batch.
How does this pipeline differ from a simple “filter‑then‑train” approach?
Beyond binary filtering, the pipeline adds a model‑based tiered scoring stage that ranks examples, a decontamination step that explicitly protects benchmark integrity, and a mixture‑selection stage that balances long‑CoT and direct‑response data. These extra layers preserve useful reasoning trajectories while eliminating subtle leakage.
Training simultaneously on “think” examples (full reasoning chains) and “no‑think” examples (concise answers) gives a single model the ability to switch between deliberative and fast modes at inference time.
Why not train separate models for think and no‑think behavior instead of mixing them?
Separate models would double the deployment footprint and require a heuristic to choose between them at runtime. Mixing both behaviors in one model lets a single weight set be reused, and the joint training empirically boosts the no‑think performance because the model learns to extract the essential reasoning patterns from the think examples.
**Figure 10** Supervised fine-tuning data curation pipeline. General data comprises general instruction and reasoning examples, while agentic data forms a separate branch. All sources undergo schema formatting, filtering and verification, mixture selection, benchmark decontamination, and preparation as packed training sequences with assistant-only loss.
**Figure 11** Training dynamics of the two length-specific SFT variants. The 64K run (left) and the 256K run (right) exhibit similar loss-reduction patterns: a rapid initial decrease followed by stepwise convergence as the learning rate follows a cosine decay schedule. Curves are shown from optimizer step 50 to focus on steady-state training dynamics. Solid blue curves show a 160-step moving average of token-level cross-entropy (CE) loss, light-blue curves show raw loss values, and salmon curves show the learning rate. Both panels use the same CE-loss scale. The released model uses the 256K SFT variant.
Benchmark Performance
Benchmarking shows ZGCM‑1‑7B excels across reasoning, code, and agentic tasks.
The paper’s core claim is that an open, efficient training recipe can deliver frontier‑level capabilities. Here we verify that claim by measuring ZGCM‑1‑7B on a wide suite of non‑agentic and agentic benchmarks.
ZGCM‑1‑7B outperforms all comparable 7B–8B models on reasoning benchmarks, reaching 99.09 % accuracy on AGIEval SAT Math.
Table 2 shows ZGCM‑1‑7B achieving the highest score on AGIEval SAT Math, while the next best model (DeepSeek‑R1) scores 90.88 %.
The table displays the composition of supervised fine-tuning data, categorized into "General data" and "Agentic data," along with their respective row shares and primary roles.
**Table 2.** Thinking-mode benchmark results (%) for ZGCM-1-7B and comparable-scale models. Dark blue cells in bold indicate the best result; light blue cells indicate the second-best result.
The table presents performance metrics for various models across three benchmarks: WebWalkerQA, BrowseComp, and GAIA text-only. Models are categorized into "Our Model," "Open-source Research Agent," "Open-weight Models with Tools," and "Proprietary Models with Tools."
AI-Native R&D Lifecycle
We embed LLM agents throughout the R&D pipeline, turning each stage into an autonomous, feedback‑driven process.
Traditional model development relies on ad‑hoc scripts and manual supervision, causing bottlenecks at data preparation, training, and evaluation. By delegating each stage to LLM‑based agents, we eliminate those hand‑off delays and keep the loop continuously moving.
Instead of treating data engineering, model design, training, and evaluation as isolated human tasks, we let LLM agents occupy every stage and share their outputs through a common harness.
R1 tells A1 to enforce a minimum token frequency of 5; A1 writes a filter script, runs it on a 1 k‑sentence toy corpus, and verifies the resulting token distribution meets the threshold.
A1 passes the cleaned dataset to A2; A2 launches a training job on a single GPU, monitors loss convergence, and automatically adjusts the learning rate when plateauing.
After training, A2 invokes ACE on the model, which returns a profile indicating strong “Arithmetic” capability but weak “Reasoning” capability.
A2 feeds the ACE report back to A1, which updates the data‑filter script to add more arithmetic‑focused examples, and the loop restarts.
This toy loop demonstrates how a single pair of agents can close the full R&D cycle—cleaning data, training, evaluating, and feeding the result back—without any manual hand‑off.
**Figure 13** AI-native R&D workflow. Each researcher directs agents across seven core stages of model development: data engineering, model architecture design, learning algorithm design, experimentation & monitoring, infrastructure engineering, evaluation, and deployment engineering. Completed agent work produces reusable experience (validated scripts, workflows, checklists, debugging history), and researcher discussions produce human context (conversations, meeting notes, decisions, plans); both feed into a shared agent harness that equips all agents with accumulated context, skills, tools, memory, orchestration, and verification capabilities.
By unifying data preparation, training, and evaluation under a single autonomous framework, the AI‑native R&D lifecycle dramatically reduces the time and compute overhead traditionally spent on manual coordination.
Analyzing AI Autonomy
We quantify AI agents’ autonomy across the R&D lifecycle and highlight task‑dependent limits.
Agents exhibit higher autonomy on operational tasks than on model‑design tasks.
Operational tasks (experimentation, deployment) are rated at level 4, while design tasks (architecture, learning algorithm) are rated at level 2–3.
**Table 4.** Binary Function Search results on 50 tasks. Valid submissions counts answers in the required format; Correct counts exact oracle matches; Accuracy is Correct/50. The dark blue cell in bold and the light blue cell indicate the best and second-best accuracy.
Capability Dynamics
Capability trends across token windows reveal diminishing returns and divergent improvements.