Gemma 4 Technical Report

Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, Mayank Chaturvedi, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhawaty, Utku Evci, Clément Farabet, Johan Ferret, Filippo Galgani, Sertan Girgin, Jean-Bastien Grill, Maarten Grootendorst, Jiaxian Guo, Cassidy Hardin, Yanzhang He, Steven M. Hernandez, Omri Homburger, Léonard Hussenot, Juyeong Ji, Armand Joulin, Aishwarya Kamath, Parnian Kassraie, Olivier Lacombe, Preethi Lahoti, Gaël Liu, Gus Martins, Luciano Martins, Tatiana Matejovicova, Ramona Merhej, Nikola Momchev, Sneha Mondal, Ryan Mullins, Sindhu Raghuram Panyam, Shreya Pathak, Sarah Perrin, André Susano Pinto, Etienne Pot, Angéline Pouget, Alexandre Ramé, Sabela Ramos, Douglas Reid, David Rim, Morgane Rivière, Karsten Roth, Louis Rouillard, Omar Sanseviero, Pier Giuseppe Sessa, Shane Settle

Gemma 4 introduces a multimodal, encoder-free architecture and a reasoning-focused "thinking" mode to the open-weight model family.

How does the Gemma 4 architecture integrate multimodal capabilities and compute-efficient reasoning into a unified, open-weight model suite?

Large language models often struggle with multimodal integration and reasoning efficiency, frequently relying on heavy, separate encoders that fragment memory and slow down inference. Gemma 4 replaces these separate encoders with a unified, encoder-free architecture that projects raw audio and image patches directly into the model's embedding space, while adding a "thinking" mode that generates reasoning traces before final responses. This generation achieves frontier-level performance on reasoning and multimodal benchmarks, with the 31B model ranking as the leading dense open-source model on the Arena leaderboard.

Paper Primer

The Gemma 4 suite addresses the trade-off between model capability and hardware constraints by optimizing the entire stack: from architectural design to inference-time efficiency. The core move is a shift toward unified, encoder-free processing for the 12B model and the integration of speculative decoding via multi-token prediction (MTP) drafters.

Gemma 4 31B is the top-performing dense open-weight model on the Arena text leaderboard.

Blind side-by-side human evaluations using the Elo rating system. Achieved an Elo score of 1451, rivaling significantly larger Mixture-of-Experts (MoE) models.

The encoder-free architecture and KV cache optimizations significantly reduce memory footprint.

Global KV cache footprint reduced by up to 37.5% through key-value reuse and local-to-global attention ratios. Audio encoder footprint reduced by 78% compared to previous generations.

Why move to an encoder-free architecture for the 12B model?

Separate vision and audio encoders create memory fragmentation and require additional parameters; projecting raw patches directly into the embedding space simplifies the architecture and improves efficiency.

What is the purpose of the "thinking" mode?

It enables the model to generate internal reasoning traces before providing a final response, which improves performance on complex tasks like mathematics and coding.

Introduction to Gemma 4

Introducing Gemma 4, a multimodal, efficient model suite built on dense and MoE designs.

Large language models have become increasingly capable but remain costly to run and often lack native multimodal processing, making on‑device deployment and reasoning‑intensive tasks difficult.

Gemma 4 addresses this gap by offering a family of open‑weight models that natively handle text, images, and audio while improving compute and reasoning efficiency through a hybrid dense/Mixture‑of‑Experts (MoE) architecture and integrated speculative decoding.

The architecture blends a standard dense transformer stack with sparsely activated expert sub‑networks, letting most tokens follow the cheap dense path and only a fraction invoke large expert modules; a unified encoder‑free front end projects raw visual and audio patches directly into the language‑model embedding space.

The suite spans from a 2.3 B‑parameter dense model up to a 31 B‑parameter dense model, and includes a 26 B‑parameter Mixture‑of‑Experts variant that activates only 3.8 B parameters per forward pass.

Embedding layers add per‑layer parameters, yielding total counts of 455 M (E2B), 975 M (E4B), 1.55 B (12 B dense), 4.016 B (31 B dense), and 1.41 B active + 1.87 B total for the MoE model.

Additional innovations include a multi‑token prediction (MTP) drafter head for speculative decoding, quantized variants trained with quantization‑aware training, a 5:1 (4:1 for the 2.3 B model) local‑window to global self‑attention ratio using $p$‑RoPE, and the thinking mode that improves performance on mathematics and coding tasks.

The shift toward natively multimodal, efficient reasoning models marks a new direction for open‑weight LLMs.

Core Model Architecture

Describes Gemma 4’s hybrid dense/MoE decoder and its long‑context efficiency tricks.

Gemma 4 builds on a decoder‑only Transformer backbone, but augments it with a hybrid dense/MoE design and several long‑context optimizations. The dense variants (E2B, E4B, 12 B, 31 B) use full‑width feed‑forward layers, while the MoE variant activates only a subset of expert sub‑networks, keeping compute modest despite a 26 B total parameter budget. These tricks address the core pain of scaling multimodal reasoning without exploding memory or latency.

Instead of a single monolithic feed‑forward block, the model holds many expert sub‑networks and activates only a few per token, so capacity grows while the per‑token compute stays low.

Global attention layers drop the separate value matrix and simply reuse the key matrix as values, cutting the KV cache size by 37.5 % while preserving the attention computation.

Compute keys $K$ for the 4 tokens (shape $4\times8$).

Set values $V$ equal to $K$ (no extra allocation).

Cache only the $K$ matrix: $4\times8 = 32$ slots, a 50 % reduction.

During attention, compute $QK^\top$ as usual, then multiply the resulting weights by $K$ (now serving as $V$).

The output vectors are identical to the standard case because $V$ and $K$ contain the same information.

Reusing keys as values halves the KV memory without altering the attention result, which is why Gemma 4 can afford longer context windows.

How does reusing keys as values differ from the usual separate value matrix?

In ordinary attention the model learns distinct representations for “what to match” (keys) and “what to return” (values). Gemma 4’s trick forces the returned content to be exactly the same representation used for matching, eliminating the extra $V$ storage while leaving the $QK^\top$ similarity scores unchanged.

**Table 1.** Parameter counts for the Gemma 4 models. The vocabulary we use has 262k entries. The model noted with a star is an MoE defined by its number of active parameters. Note that the extra embedder parameters in E2B and E4B are per-layer embeddings.

Multimodal Vision Integration

Gemma 4 12B replaces modality‑specific encoders with tiny projection modules.

The previous sections detailed modality‑specific encoders; the bottleneck they introduce is the extra compute and parameter budget for each sensor.

Instead of a deep vision or audio encoder, the model projects raw patches or audio chunks directly into the language‑model embedding space with a single linear layer, keeping spatial or temporal cues via lightweight positional embeddings.

Flatten each patch to a $48$‑dimensional vector $p_i$.

Compute $e_i = p_i W$, yielding an $8$‑dimensional embedding for each of the four patches.

Add 2‑D positional vectors $pos_i = (x_i, y_i)$ transformed by a $2\times8matrixP$, producing $e_i' = e_i + pos_i P$.

Apply LayerNorm to each $e_i'$ to obtain the final token representations fed to the LLM.

The entire spatial reasoning is captured by the cheap linear projection plus positional offsets; no deep attention over patches is required.

How does this projection differ from a conventional Vision Transformer encoder?

A Vision Transformer first runs multiple self‑attention layers over patch embeddings, learning complex interactions before any positional bias is added. The encoder‑free design skips those layers entirely, using only a single linear map and fixed 2‑D coordinates, so it cannot model higher‑order patch relationships but gains massive speed and parameter savings.

**Figure 2 | Image resizing.** Here we use `patch_size`=16, `pooling_kernel_size`=3, `max_soft_tokens`=10. The image is thus first resized to 2 × 4 pooled patches (each of size 48px^2), which is the closest match that results in a sequence length below the targeted 10. The 72 patches (each of size 16px^2) are then processed by the vision encoder, the vision encoder representations are pooled 3 × 3, and the resulting 8 soft tokens are processed by the LLM backbone.

**Table 10.** Vision encoder architecture.

By removing heavyweight encoders, Gemma 4 12B achieves multimodal capability with a dramatically smaller compute footprint, enabling faster inference and easier deployment.

Quantization and Efficiency

Quantization‑Aware Training trims memory and latency while keeping accuracy.

Deploying large multimodal models on commodity hardware demands a way to shrink weights and activations without breaking the learned representations.

During training the model pretends its weights and activations will be low‑bit at inference, so it learns scale factors that keep the quantized values faithful to the full‑precision behavior.

Channel 0 values = [0.12, ‑0.34]; min = ‑0.34, max = 0.12 → scale = (2‑1)/(max‑min) ≈ 2.94.

Quantize: round((value ‑ min) × scale) → [2, 0] (int2 representation).

Channel 1 values = [0.56, ‑0.78]; min = ‑0.78, max = 0.56 → scale ≈ 1.79.

Quantize to int4: round((value ‑ min) × scale) → [5, 0] (int4 representation).

During forward pass the dequantizer reconstructs each entry as min + quantized/scale, preserving the original dynamic range per channel.

Per‑channel low‑bit quantization adapts the numeric range to each channel’s statistics, yielding far lower reconstruction error than a single global scale.

How does QAT differ from post‑training quantization?

Post‑training quantization snaps a fixed full‑precision model to low‑bit after training, often causing a sudden accuracy drop. QAT, by contrast, incorporates the quantization step into every training iteration, allowing the optimizer to adjust both the underlying parameters and the scaling factors so the final low‑bit model behaves as if it had been trained in that regime from the start.

We expose two quantization families: “mobile quantization” mixes per‑channel int2 and int4 weights with int8 activations, while “`Q4_0`” applies blockwise quantization across the entire tensor.

**Table 3.** Text only, Gb memory footprint comparison between raw and quantized checkpoints for weights and int8 KV caching (+KV) at 32k context size. † is mobile quantization, ‡ is `Q4_0`.

Speculative Decoding and Infrastructure

We transform pre-trained models into instruction-tuned agents using curated data filtering and specialized control tokens.

Instruction tuning converts raw pre-trained models into helpful assistants by aligning them with human intent. Our approach mirrors the post-training pipeline of Gemma 3, with one critical addition: a "thinking mode" that allows the model to generate a reasoning trace before providing its final answer.

Post-training performance depends less on raw volume and more on the quality of the instruction-response pairs; we aggressively prune data that introduces noise or safety risks.

While pre-trained (PT) and instruction-tuned (IT) models share the same tokenizer, they diverge in their use of control tokens to signal the end of a generation. PT models terminate with an $<eos>$ token, whereas IT models use a $<turn|>$ token to delineate the end of a conversational turn.

Control tokens act as structural markers that tell the model when to stop generating and how to interpret the current turn in a multi-turn dialogue.

Safety and Provenance

Gemma 4 is a multimodal open model that blends dense and Mixture‑of‑Experts layers with speculative decoding for efficient reasoning.

Our governance framework builds on prior models but adds checks for the new multimodal capabilities, balancing openness with the risk of malicious use that could harm individuals or institutions.

Safety policies prohibit generation of child sexual abuse material, dangerous instructions, sexual content, hate speech, and harassment, and the training pipeline filters personal and sensitive data to protect privacy.

We evaluate safety with both automated metrics and human raters, running tests without any safety filters so the intrinsic behavior of Gemma 4 can be measured; across all sizes and modalities the model shows markedly fewer policy violations than its predecessors.

Ethical guidance emphasizes continuous bias monitoring, debiasing during fine‑tuning, developer education on misinformation risks, and strict adherence to privacy regulations when deploying the models.

We provide a Responsible Generative AI Toolkit and recommend that developers configure safeguards appropriate to their product policies, adopting mitigations proportional to the identified risks before releasing the models.

Performance Benchmarks

Key performance numbers and supporting details for Gemma 4.

Gemma 4 31B improves MMMU Pro accuracy by +2.0 percentage points over the next‑best open‑weight model.

Table 4 shows Gemma 4 31B at the top of the open‑weight leaderboard, with the runner‑up trailing by roughly two points on the same benchmark.

Gemma 3 is the predecessor dense model that operates without the “thinking” mode; it processes inputs in a single pass without speculative decoding.

The table lists various contexts and their corresponding formatting tags used in a discussion format. | Context | Formatting | | :--- | :--- | | Thinking toggle | `<|think|>` | | Function declaration | `<|tool>declaration:...<tool|>` | | Function call | `<|tool_call>call:...<tool_call|>` | | Thinking trace | `<|channel>thought ...<channel|>` | | System turn | `<|turn>system` | | User turn | `<|turn>user` | | Model turn | `<|turn>model` | | End of turn | `<|turn|>` |

**Table.** Performance comparison of different Gemma 4 model variants across various benchmarks.

**Table 4.** Leading open-weight models on Arena Text (Chiang et al., 2024) (as of June 19, 2026). Models are evaluated through blind side-by-side evaluations by human raters, and attributed scores based on the Elo rating system. The top closed model (gray) is included for scale. Gemma models rival much larger models, and Gemma 4 31B is the leading dense open model on the leaderboard.

**Table 5.** Performance comparison of Gemma 3 27B and Gemma 4 models on diverse benchmarks. All models are in thinking mode unless explicitly stated.

**Table 6.** Gemma 4 models performance on vision benchmarks at different resolutions (thinking). We use the maximal supported resolution (1120 vision tokens) and report results with 280 vision tokens in Table 12. Gemma 3 27B is non-thinking and uses Pan &amp; Scan.

**Table 7.** Audio performance for Gemma 4 and Gemma 3n models. Top: CoVoST (S2TT prompt: transcribe then translate). Bottom: FLEURS ASR (transcription). Compared to Gemma 3n of corresponding sizes, Gemma 4 achieves a 12% (E2B) / 10% (E4B) relative improvement on translation and a 17% (E2B) / 12% (E4B) relative improvement on transcription, despite a 78% reduction in on-disk audio encoder footprint (from 390 MB to 87 MB after quantization).

**Table 9.** Long context performance of Gemma 3 and Gemma 4 models (without thinking).

The algorithm rescales an image so that the resulting token grid fits a target token budget while keeping the original aspect ratio.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers