Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation
Qian Yu, Zongyang Qiu, Juncheng Yan, Hongliang Li, Qi Qin, Keqi Wang, Siqi Luo, Junkui Ju, Yapeng Mi, Tiantian Lu, Tian Liu, Pengyu Xie, Chenhao Ji, Minghui Zhang, Yiang Zhao, Siyu Yan, Haiying Xu, Yanbo Wang, Jianghan Shen, Yuandong Pu, Jiayi Lei, Borui Zhang, Jinxin Shi, Bo Zhang, Yuhang Zang, Shenglong Ye, Lixin Gu, Hongjie Zhang, Linghan Liu, Xinyuan Chen, Yuewen Cao, Lihe Zhang, Xiaohong Liu, Junjun He, Qiangwei Liu, Qi Zhang, Yu Cheng, Tianfan Xue, Hui Xiong, Chang Xu, Yu Qiao, Yihao Liu, Bin Fu
A unified diffusion language model using multi-codebook discrete visual tokens for omni-visual understanding and generation.
How can we unify visual understanding and generation in a single diffusion language model using multi-codebook visual tokens?
Unified multimodal models often struggle to balance high-capacity visual generation with the discrete, sequential nature of text modeling, leading to optimization conflicts between modalities. Intern Lumina U2 solves this by representing images, videos, and 3D assets as multi-codebook discrete tokens, allowing them to be processed by a single masked-diffusion backbone alongside text. This unified approach achieves leading performance across visual understanding and generation benchmarks while accelerating inference by 4.45× through a stability-aware sampling strategy.
Paper Primer
Existing unified models typically force a choice: either use continuous visual representations that clash with discrete text objectives, or use single-codebook discrete tokens that lack the capacity for high-fidelity generation. Intern Lumina U2 bridges this gap by factorizing each visual position into eight complementary codes, effectively increasing representational capacity without increasing the backbone's sequence length.
The core mechanism is a multi-codebook autoregressive head: the backbone processes all spatial positions in parallel, while the head predicts the eight codes at each position sequentially. This is like a high-resolution printer that renders the broad layout of a page in one pass, then fills in the fine-grained color and texture details for each pixel block in a rapid, localized sequence.
Intern Lumina U2 achieves state-of-the-art performance among unified models across diverse visual tasks.
The model ranks first on six of ten image-understanding benchmarks and outperforms the LLaDA2.0-Uni baseline on eight of ten, while also delivering strong results in video, 3D understanding, and text-to-image generation.
The stability-aware inference strategy significantly reduces generation latency.
By committing tokens at confidence plateaus and refreshing the cache only when hidden-state drift is detected, the system avoids redundant computation.
Why use a multi-codebook approach instead of simply increasing the number of tokens per image?
Increasing the number of tokens per image would linearly increase the sequence length, significantly slowing down the backbone. By merging the eight codes into a single backbone token at the input and using an autoregressive head at the output, the model maintains a constant sequence length regardless of the codebook depth.
How does this model handle the different generation requirements of text and images?
The model uses a task-aware attention mask that enforces modality-specific generation: text follows block-wise sequential diffusion, while visual tokens are refined in parallel with bidirectional access to local and global context.
Introduction and Motivation
Motivates multi‑codebook tokenization to unify visual understanding and generation.
Existing unified multimodal models either compress visual data into a single discrete code per patch—losing detail—or keep visual states continuous, which forces separate training objectives for text and images and creates optimization tension.
Single‑codebook tokenizers cannot capture enough visual detail, while heterogeneous continuous representations split the training objective; a high‑capacity, fully discrete visual tokenization would let a single diffusion language model learn both understanding and generation together.
**Figure 1.** Overview of Intern Lumina U2's versatile omni-visual understanding and generation capabilities.
**Figure 2.** Showcases of Intern Lumina U2 on image generation, editing, and scientific-diagram generation.
The key shift is moving from a single discrete code per visual patch to a set of complementary codes, dramatically expanding capacity while staying compatible with text tokenization.
Related Work
We position Intern Lumina U2 among prior multimodal and diffusion approaches.
We relate Intern Lumina U2 to four bodies of work: unified multimodal models, diffusion language models, multi‑codebook visual representations, and inference acceleration for masked diffusion.
Table 1 compares 14 multimodal systems across backbone paradigm and supported modalities; Intern Lumina U2 is the only model highlighted with full support for image, video, 3D, and masked‑diffusion decoding.
Unified multimodal understanding and generation splits into two lines: discrete visual‑token prediction (e.g., Chameleon, Emu3, MMaDA, Lumina‑DiMOO) and continuous diffusion or flow generators (e.g., Transfusion, UniFluid, BAGEL, InternVL‑U, SenseNova‑U1). Hybrid designs such as LLaDA2.0‑Uni combine discrete backbone tokens with a separate diffusion decoder.
Diffusion language models began with LLaDA, showing that masked diffusion can match strong autoregressive baselines. Extensions (LLaDA2.0, LLaDA‑V, LLaDA‑o) integrate vision features or pair discrete text diffusion with continuous image diffusion. Intern Lumina U2 builds on LLaDA2.0‑mini, extending the visual state to a multi‑codebook token while preserving the masked‑diffusion backbone.
Multi‑codebook representations increase capacity without lengthening the spatial grid. Prior work (RQ‑Transformer, VILA‑U, UniTok, TokenBridge) introduced residual or factorized quantization in autoregressive settings. AToken supplies the shared multi‑codebook backbone used here; Intern Lumina U2 makes the multi‑codebook token the visual state of a masked‑diffusion LM, keeping sequence length independent of the number of codebooks.
Accelerating masked‑diffusion decoding follows learning‑based (auxiliary model) and training‑free (sampler reshaping) routes. Training‑free methods such as MaskGIT, Fast‑dLLM, low‑discrepancy reveal orders, and cached sampler state reduce the number of refinement steps. Our sampler adopts the training‑free approach, adapting commit schedules and cache refresh based on visual‑token confidence.
**Table 1.** Comparison of multimodal models based on supported modalities and representation/decoding techniques.
Model Architecture
We detail how multi‑codebook tokenization and task‑aware masking enable unified diffusion across modalities.
Single‑codebook tokenizers either preserve fine detail or align with language, but never both, limiting a unified model’s capacity. Our trick is to split each visual position into eight independent codebooks, then merge them into a single backbone token, preserving detail while keeping sequence length short.
Think of a visual pixel as a stack of eight small “color” vocabularies; each stack is collapsed into one token for the backbone, giving eight times the representational capacity without lengthening the sequence.
Embedding lookup: e₁ = E₁(2), e₂ = E₂(1).
Concatenate embeddings → [e₁; e₂] (dimension 2d).
Apply linear projection `W_in` to obtain hidden vector $x$ of dimension d.
During generation, the AR head first predicts c₁, embeds it as e₁, then predicts c₂ conditioned on e₁.
The joint (c₁,c₂) carries more detail than a single code, yet the model processes only one hidden vector $x$, keeping the sequence length unchanged.
How does this differ from a naïve single‑codebook tokenizer?
A single code can only select one of 4096 entries, limiting expressive power. By using eight independent codebooks, we effectively have 4096⁸ possible combinations while still feeding a single vector to the backbone, so we gain capacity without blowing up sequence length.
Instead of generating tokens one after another, the model denoises a whole block of masked tokens in parallel, then repeats the process over successive blocks, unifying text and visual generation under the same diffusion objective.
Why not keep the original autoregressive generation for visual tokens?
Autoregressive generation would require a separate token stream for each of the K codes, exploding sequence length. The diffusion‑based block approach lets us predict all K codes for a position in a single forward pass, preserving efficiency while still modeling inter‑code dependencies.
The mask decides which tokens can see which others, enforcing the correct conditioning for each task (e.g., a visual answer may attend to its own context but not to future answers).
How does this mask differ from the standard block‑wise mask used for pure‑text diffusion?
The standard mask only restricts attention within a single modality (text). Our task‑aware mask adds cross‑modal rules: visual answers can attend to their own condition and to earlier clean copies, while text answers cannot see future visual answers, enforcing proper conditioning across modalities.
Autoregressive inference of the multi‑codebook AR head for a masked visual position.
All tasks share a packed dialogue sequence where each token carries five metadata fields (type, turn, noise state, block, position ID). Table 18 lists the structural tokens that delimit visual inputs. This unified format lets the same backbone and mask handle text‑only, image‑only, and multimodal examples.
**Figure 3.** Overall architecture of Intern Lumina U2. Visual inputs are encoded into visual tokens by AToken and text into text tokens; the MoE diffusion LLM recovers masked text tokens through the text head, with text answers generated by block diffusion, and masked visual tokens through the multi-codebook AR head, which predicts the $K$ codes of each visual token sequentially.
**Figure 4.** Three candidate input interfaces for multi-codebook visual tokens. We adopt the third design: independent embeddings for the eight codebooks, concatenation at each spatial position, and projection into one backbone token.
**Figure 5.** Training and inference of the multi-codebook AR head: one teacher-forced causal pass yields all $K$ logits at training; at inference the prefix grows for $K$ steps, each sampled code $\hat{c}_k$ is embedded into $e_k$ and appended. Positions are processed in parallel, codebooks autoregressively.
**Figure 6.** Task-aware attention mask at block level. (a) Image answer: one noisy visual answer $\tilde{V}$ after the clean context $C$, bidirectional over all positions. (b) Text answer: block diffusion over the noisy blocks $\tilde{b}_1, \tilde{b}_2, \tilde{b}_3$ and their clean copies $b_1, b_2, b_3$, with $M_{\text{BD}}$, $M_{\text{OBC}}$, and $M_{\text{BC}}$. (c) A text answer followed by an image answer, as in thinking before generation: the image reads the clean copy of the preceding text blocks. (d) Multi-turn text with conditions $C_1, C_2$, noisy answers $\tilde{x}_1, \tilde{x}_2$, and clean copies $x_1, x_2$: a later turn reads the clean copies of earlier answers.
Data Construction
We build a unified pipeline that turns six heterogeneous data streams into a dialogue format for staged training.
Training a multimodal model requires data that is both diverse—spanning images, video, 3D assets, and pure text—and internally consistent. Without a unified format, each new source would demand custom preprocessing, breaking scalability and introducing hidden biases.
All six modality categories are first filtered, then encoded into a single dialogue record whose fields (conditions, targets, task type) are fixed across categories, allowing the same training loop to consume any sample.
Validate & dedup confirms both samples are unique and passes quality thresholds.
Recaption generates a short caption for I₁ (“a cat on a sofa”) and a long caption for I₂ (“a photorealistic sunrise over mountains”).
Quality judge scores the captions; both exceed the aesthetic threshold, so they are kept.
Edit check sees no edit instruction for Sample A, but Sample B receives an edit tag “enhance colors”.
Reasoning trace adds a chain‑of‑thought tag to Sample A (“why is the cat sitting?”) and a simple prompt tag to Sample B.
Re‑weight assigns a weight of 1.0 to Sample A (high‑density text) and 0.6 to Sample B (synthetic image), balancing their contribution to the loss.
Even with only two modalities, the pipeline produces a uniform dialogue record that the model can ingest without any special‑case code.
Validate & dedup: remove duplicates and enforce basic quality thresholds.
Recaption: generate short and long textual descriptions for visual inputs.
Quality judge: score aesthetic and relevance; discard low‑scoring items.
Edit check: verify presence and correctness of any edit instruction.
Reasoning trace: optionally attach a chain‑of‑thought segment between tags.
Re‑weight: compute a per‑sample weight from domain density and supervised‑token count.
**Figure 7.** Data construction pipeline. Six data categories are built with category-specific rules, converted to one record format, encoded offline, and re-weighted at each training stage. Bars show backbone-position shares (Table 2).
Training Strategy
We train the model through five progressive stages that scale resolution, tasks, and modalities.
Training visual modules from scratch quickly destabilizes a model that already knows language; naïvely scaling resolution and data together caused severe degradation in early experiments.
The pipeline breaks a huge, heterogeneous learning problem into five manageable steps, each expanding one axis (resolution, task variety, or modality) while keeping the others fixed.
Why not simply increase resolution and add all new tasks at once?
Doing both together forces the model to relearn low‑level perception while also learning high‑level generation, which overwhelms the optimizer and erases the alignment built in earlier stages. The staged schedule lets the model first master the new pixel scale, then safely expand its task repertoire.
After Stage 1 the model has consumed 5 B tokens and learned basic vision‑language alignment.
Stage 2 adds 20 B tokens, bringing the total to 25 B and introducing fine‑grained perception tasks.
Stage 3 runs at higher resolution without new tasks, adding 25 B tokens (half of the combined budget) to reach 50 B total.
Stage 4 adds the remaining 25 B tokens, now covering editing, video, and 3D tasks while keeping resolution fixed.
Stage 5 fine‑tunes with 30 B tokens, yielding a total of 80 B tokens and a model ready for instruction following.
Splitting the token budget this way shows how each stage contributes a distinct capability; the total token count alone would hide the fact that resolution and modality expansions are deliberately separated.
**Table.** Training configurations for different stages of the model.
Inference Procedure
We implement block diffusion for text and spatial masked diffusion for images, accelerated by cache-aware reuse and plateau-based sampling.
Our inference procedure leverages the multi-codebook visual tokenizer to perform both text and image generation. While text generation proceeds block-by-block to allow variable-length outputs, image generation uses full spatial refinement to maintain global canvas consistency.
Block diffusion generates text by appending fixed-length blocks of mask tokens and denoising them in parallel, allowing the model to reuse the key-value cache of the completed prefix while maintaining autoregressive control.
Visual generation uses iterative parallel refinement where all spatial positions are decoded jointly, allowing the model to revise previously sampled codes throughout the generation process.
To mitigate the high computational cost of iterative visual sampling, we implement training-free acceleration through cache-aware reuse and adaptive remasking.
PHV accelerates inference by committing tokens that reach a temporal confidence plateau and using Halton ordering to distribute remaining uncertainty evenly across the image.
The table presents a comparison of various system optimization methods and step budget strategies, evaluating them based on Steps, BB fwd, Latency (s), Speedup, and GenEval metrics. The table is divided into two sections: "Cumulative system optimizations (fixed 96-step schedule)" and "Reduced and adaptive step budgets".
**Figure 9.** Offline temporal-stability test for PHV on 200 prompts. Each plateau-eligible position is paired with a still-masked *matched-control* token from the same image that does not satisfy the plateau criterion while exhibiting comparable confidence and EMA values. Both are held masked for 2, 4, or 8 additional refinement steps. (a) The difference in top-1 flip rate between the plateau token and its matched-control token grows with the hold duration. (b) Per-prompt flip rates at $H = 8$; 157 of 200 prompts fall below the diagonal. (c) Posterior max-probability trajectories for three representative plateau-matched-control pairs.
Performance Evaluation
Intern Lumina U2 dominates image, video, and 3D benchmarks, setting new bests across most tasks.
Intern Lumina U2 achieves the best score on six of ten image‑understanding benchmarks despite activating only 1.4 B parameters.
Table 5 shows Intern Lumina U2 leading on six benchmarks (CharXiv, HallusionBench, MathVision, DynaMath, and two others) and tying on MME‑C.
**Table.** Performance comparison of various models across different benchmarks, categorized by task type (Documents and charts, Perception and hallucination, Knowledge and mathematical reasoning). The table includes parameter counts for each model and highlights the best-performing results in bold.
**Table 6.** Video and 3D understanding.
**Table 1.** Performance comparison of various models across different metrics (Sty, Wor, Att, Act, Rel, Log, Gra, Com, Lay, Txt, Overall).
**Table 14.** Instruction-based image editing. ImgEdit (Ye et al., 2025b) overall score (1–5) and GEdit-Bench (Liu et al., 2025b) overall score $G_O$ (0–10) for English and Chinese prompts. Baseline numbers are taken from Bie et al. (2026) and Tian et al. (2026). **Bold** marks the best unified-model score in each column.
Scientific Diagram Generation
Intern Lumina U2 sets new state‑of‑the‑art scores on scientific diagram benchmarks.
Intern Lumina U2 matches the top GenEval score of 89, tying with LLaDA2.0‑Uni among unified models.
Table 8 shows Intern Lumina U2 achieving 89 on GenEval, the highest among unified‑model entries.
**Figure 10.** Input-interface ablation. (a) Final next-code top-1 accuracy for each of the eight codebooks, normalized by D3; the blue annotations give D3’s absolute accuracy at the first and last codebooks. (b) Image loss over the two training epochs, shown as a rolling median with an inset for the converged tail. Higher is better in (a), whereas lower is better in (b).
Ablation Studies
Ablation studies reveal the multi‑codebook AR head’s superiority and optimal inference sampling.
Recall that Intern Lumina U2 leverages a multi‑codebook visual tokenizer to enable a unified diffusion language model for both understanding and generation.
The multi‑codebook AR head beats the alternatives on every metric, confirming that modeling inter‑code dependencies, not just richer conditioning, drives the gain.
The quality‑first sampling setting ($\delta$p=0.18, $f_p$=0.005) attains 89.3 GenEval at 35.9 s per image, a 4.45× speedup.
Figure 11 shows this configuration on the speed‑quality frontier.
**Figure 11.** Inference-sampling ablation over the plateau change threshold $\delta_p$ and confidence floor $f_p$, 553 prompts per cell. (a) GenEval over the $6 \times 6$ grid; the black box marks the selected quality-first setting $\delta_p=0.18$, $f_p=0.005$. (b) Speed-quality frontier over all 36 cells, i.e. the configurations not improved on in both latency and quality by any other cell; the selected setting (red) reaches GenEval 89.3 at 35.9 s per image (4.45$\times$).
**Table 16.** Ablation of the visual output head: three training runs at 512-pixel generation resolution with comparable token budgets (about 197B–220B tokens each). All scores are on a 0–100 scale; TIIF is the mean accuracy and UniGenBench the EN ten-dimension average. Bold marks the best score in each column.
The table presents a comparison of performance metrics across different physics domains (E&M, Mechanics, Optics, Quantum mechanics, Thermodynamics, Acoustics) for various models (Seedream, Gemini, GPT-Image, SenseNova, Princigram-D, Princigram-B, and Intern Lumina U2) under two scoring categories: Local scoring and Global scoring.
**Table 12.** VeriphyT2IBench domain-level breakdown for representative models. Each panel reports question-level accuracy (percent; higher is better) for the local attribute bank or the global whole-diagram bank. The overall rows are omitted because the corresponding aggregate scores are already reported in Table 11.
Appendices
Appendix lists full model specs, token formats, data sources, hardware details, and references.
The conclusion restates that Intern Lumina U2 is the first unified diffusion language model handling multi‑codebook visual states, achieving strong results on generation and understanding tasks, while noting decoding overhead and missing video/3D generation as current limits.
The reference section enumerates all prior works cited throughout the paper, providing the scholarly context for the presented methods.
**Table 17.** Complete model configuration. Upper part: backbone settings. Lower part: visual interface modules.
Section A details the tokenization scheme: each visual codebook has its own embedding table, and special tokens (e.g., mask, delimiters) are replicated across all $K$ columns of the packed $[L, K]$ tensor.
**Table 19.** A packed image-understanding sample with per-position metadata. In this example, type 1 denotes text or structural tokens, type 2 a media-condition token, type 3 a text-answer token, and type 4 a visual-answer token. The clean user-provided condition and the assistant answer form one turn, and the clean copy of the answer belongs to the same turn. The `is_clean` flag is 1 for clean tokens and 0 for noisy tokens, the block id is -1 for conditions and 0 for this answer block, and a clean copy reuses the position ids of its noisy counterpart. Here $v_{ij}$ is the visual token ($K$ codes) at grid row $i$, column $j$; $\lrcorner$ is the image-row delimiter (code 5001); $q_i$ and $a_i$ are question and answer tokens; and M is <|mask|>. Labels are -100 at all positions that do not participate in supervision; only masked answer positions contribute to the loss.
Section B describes how AToken converts each modality into a grid of $K=8$ codes, flattens the grid in a prescribed order, and inserts delimiter tokens to mark modality boundaries.
**Table 20.** Visual token layouts. Patch size is $(t, h, w) = (4, 16, 16)$; every grid position carries $K=8$ codes.
Section C enumerates data sources per category, the encoding resolution applied to each, and the training stages (1–5) in which they are consumed.
**Table 21.** Sources by data category, with the public and private sources of each category listed separately. Encoding resolution is the long side (images and editing) or the frame area budget (video); “native” means the original resolution clamped to a [256, 1024] long side. Stages are numbered 1–5 as in Section 5.
Section D outlines the adaptations required to run the model on Ascend NPUs, including kernel replacements for flash‑attention, fused MoE, and attention semantics.
**Table 22.** Quality filtering of the curated text-to-image sources (counts in millions).
Section E provides additional qualitative examples of generation, omni‑visual understanding, and editing, complementing the quantitative results presented earlier.