Uncovering Understanding–Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

Penghao Wu, Haiwen Diao, Weichen Fan, Lewei Lu, Dahua Lin, Ziwei Liu

Unified multimodal models require task-specific architectural specialization to turn representational competition into mutual synergy.

Does training a single model for both visual understanding and generation create a positive synergy, or do the objectives compete for capacity?

Unified multimodal models (UMMs) combine visual understanding and generation in one interface, but this functional integration often forces the two objectives to compete for the same computational capacity, leading to asymmetric performance degradation. The authors introduce a task-decoupled architecture that routes understanding and generation tokens through specialized visual branches while maintaining shared semantic interaction via a common language backbone. This design enables mutual reinforcement: generation enriches visual features for understanding, while understanding improves the alignment of generative representations with language semantics.

Paper Primer

In a dense shared model, understanding and generation compete for the same parameters, causing one objective to dominate while the other degrades. The authors' task-decoupled architecture acts like a specialized workshop: it assigns distinct tools to the understanding and generation tasks while keeping them in the same building so they can still share the same blueprints (textual semantics).

Task-decoupled routing enables mutual improvement between understanding and generation.

Compared to dense or modality-decoupled baselines, the task-decoupled model avoids the asymmetric performance trade-offs where one capability improves at the expense of the other. The model achieves balanced performance across both understanding and generation benchmarks, whereas dense models suffer significant generation degradation and modality-decoupled models lose understanding accuracy.

End-to-end unification outperforms modular pipelines for complex reasoning-intensive tasks.

In reasoning-intensive image editing, an end-to-end UMM was compared against a matched planner-executor pipeline that separated instruction generation from image execution. The end-to-end UMM consistently outperformed the modular pipeline on both RISEBench and KRIS-Bench.

Why does simply putting understanding and generation in the same model not automatically create synergy?

The objectives often compete for the same representational and computational capacity; without architectural specialization, one objective tends to dominate the shared feature space, effectively reducing the other to a mere regularizer.

When is joint training most effective for improving performance?

Synergy emerges when the understanding and generation tasks rely on shared domain knowledge or capabilities, such as in geometry reasoning, vector graphics, or 3D spatial intelligence.

For builders of multimodal systems, unification is not a free lunch; performance gains require architectural specialization to resolve objective conflicts, while end-to-end training is best reserved for tasks requiring tight, sequential interaction between reasoning and generation.

Introduction and Motivation

Unified multimodal models unite visual understanding and generation, yet synergy is not guaranteed.

Unified multimodal models (UMMs) bring visual understanding and generation together, but this unification does not automatically yield synergy; the two objectives can reinforce each other, compete for capacity, or simply coexist.

A single model that can both interpret images (e.g., classification, VQA) and produce images (e.g., synthesis, editing) without switching architectures.

The joint training of understanding and generation objectives can pull the model in opposite directions, leading to trade‑offs where one task dominates the shared computation.

The paper’s core thesis is that synergy between visual understanding and generation is not automatic; it emerges only when architectural choices prevent one objective from overwhelming the other, turning competition into positive transfer.

**Figure 1.** Overview of our three-level study of understanding–generation synergy in native UMMs. At the representation level, we study the intrinsic relationship between understanding and generation under different architectures. At the task level, we test whether shared task knowledge enables bidirectional transfer. At the system level, we compare an end-to-end UMM with a composed pipeline of comparable understanding and generation capability.

The paper’s core thesis: synergy is not automatic but architecturally dependent.

Architectural Foundations

We compare dense sharing and modality‑decoupled routing to reveal how architecture shapes multimodal synergy.

To understand how architectural choices affect multimodal synergy we isolate two extreme routing designs: a fully shared path for all tokens and a split path that isolates visual processing from the pretrained language backbone.

All token types—text, clean visual tokens for understanding, and noised visual tokens for generation—are processed by the same LLM decoder layers, giving maximal parameter sharing and direct access to pretrained language semantics.

Text tokens stay in the pretrained language branch, while a freshly‑initialized vision branch handles all visual tokens, keeping the language backbone untouched by visual learning.

Step 1: The language branch embeds $t$ and forwards it unchanged to the shared attention layer.

Step 2: The vision branch processes $v_i$ and $g_i$ through two convolutional layers, producing token embeddings of dimension 8.

Step 3: Global attention computes interactions: each $v_i$ attends to all $v_j$ (bidirectional) and to $t$, while $g_i$ attends only to $g_j$ and $t$.

Step 4: The attention outputs are concatenated and fed back to their respective branches for the next transformer layer.

Step 5: After three layers, the language branch’s hidden state $h_t$ now encodes visual context via the attention links, while the vision branch’s tokens have been refined by the language token.

This toy walk‑through shows how the split architecture still enables cross‑modal information flow without mixing the underlying parameter sets.

**Figure 2.** Illustration of different architectures in our study. (a) Dense sharing sends all tokens through the same LLM decoder. (b) Modality-decoupled MoT keeps text in the LLM and sends all visual tokens through a scratch-trained branch. (c) Task-decoupled MoT keeps UND-V language-anchored while specializing GEN-V. For simplicity, we omit the pre-buffer layers here.

Representational Synergy

Joint training reshapes visual representations, revealing when sharing yields synergy.

We probe how joint training of understanding and generation reshapes the frozen visual backbone. The goal is to see whether shared parameters foster representational synergy or cause interference.

When a unified model learns both visual understanding and image generation, the two tasks can reinforce each other, yielding feature maps that are simultaneously discriminative for classification and coherent for synthesis.

The task‑decoupled MoT achieves a $0.08$ higher linear CKA between text and image features than generation‑only training.

Figure 5 shows CKA values of $0.20$ for MoT‑Und+Gen versus $0.12$ for MoT‑Gen at layer 40.

**Table 1.** Understanding and generation results across routing architectures. Understanding columns report category averages. Atom-level score is reported for GenEval2. Higher is better for all metrics.

**Figure 3.** Layer-wise probing experiments of visual representations. Higher is better for classification and mIoU; lower is better for depth R-MSE. Joint training strengthens visual representations, with the clearest gains in early and middle layers across semantic and geometric probing tasks.

**Figure 4.** **PCA visualization of dense visual features.** Patch-level features are projected onto a shared three-dimensional PCA basis and visualized as RGB maps. Compared with understanding-only training, joint training produces more coherent object regions and clearer spatial structure, consistent with the improvements observed by the frozen probes.

**Figure 5.** Joint learning improves vision–language alignment for generation.

Joint training produces more coherent object‑level features.

Task-Level Synergy

Joint training boosts both understanding and generation across diverse tasks.

Joint training (UND+GEN) improves both visual understanding and generation, raising the average 3D‑SI benchmark from 57.15 to 59.01.

Table 4 shows higher scores on eight of nine 3D‑SI benchmarks; Table 5 shows lower errors on Spatial‑Edit and higher PSNR/SSIM on VSC.

**Figure 9.** Examples of 3D spatial-intelligence generation tasks (I). Ego-motion transition requires applying explicit camera motion to a source view, while multi-view reconstruction infers an unobserved view from the provided object views.

System-Level Interaction

End‑to‑end unified models outperform modular pipelines on system‑level tasks.

End‑to‑end Unified Multimodal Model outperforms a planner‑executor pipeline on system‑level tasks.

Table 6 shows higher scores on both RISEBench and KRIS‑Bench across all categories.

Table 6 confirms that the end‑to‑end UMM consistently beats the planner‑executor pipeline on RISEBench and KRIS‑Bench, covering temporal, causal, spatial, logical, factual, conceptual, and procedural metrics.

System‑Level Interaction describes how perception and generation stages influence each other when executed sequentially inside a single model.

How does system‑level interaction differ from simply chaining separate models?

In a unified model the two stages are trained jointly, so gradients from the generation loss adjust the perception module, creating a feedback loop absent in a modular pipeline where each model is optimized in isolation.

**Figure 6.** Joint training improves direct visualization from SVG code. Left: Joint training achieves higher accuracy on code-conditioned VQA, which requires inferring the rendered appearance of an SVG program without seeing the image. Right: The predicted clean image after only one generation step visualizes how easily the model forms global shape and layout from the same code.

Sequential interaction tasks benefit most from unified training.

Related Work

We review prior work on unified multimodal models and their reported synergies and conflicts.

Unified multimodal models (UMMs) have rapidly emerged, combining visual understanding and generation to enable capabilities such as interleaved multimodal documents and joint textual‑visual reasoning. Recent native approaches train on multimodal data from the outset and process pixel‑level inputs and outputs directly, avoiding external encoders and allowing the interaction between understanding and generation to be examined without additional visual priors. Prior work reports mixed findings: some studies observe interference between understanding and generation, while others introduce components like Reconstruction Alignment (RecA) or UNO to promote synergy, though these results often depend on specialized modules or task designs.

Representation-Level Details

Component‑wise ablations reveal how each design choice impacts performance.

We evaluate the effect of each architectural and data‑handling choice by removing it and measuring the resulting drop.

The task‑decoupled MoT routes generation‑specific inputs through a separate branch, leaving the shared backbone untouched for understanding tasks.

Removing the task‑decoupled MoT branch hurts overall GenEval2 performance.

Task‑dec. MoT‑U+G scores 82.97 versus 84.60 for Mod‑dec. MoT‑U+G.

Removing the task‑decoupled MoT also degrades DPG‑Bench performance.

Task‑dec. MoT‑U+G achieves 87.30 versus 92.14 for Mod‑dec. MoT‑U+G.

Discarding the task‑decoupled MoT in favor of the modality‑decoupled variant yields a large gain on the GenEval2 “Object” skill.

Mod‑dec. MoT‑U+G scores 84.60 overall, while Task‑dec. MoT‑U+G scores 82.97; the “Object” column in Table 9 shows 84.60 versus 63.96.

Task-Level Details

Ablation diagnostics probe the contribution of each task component.

To understand which task‑level components truly matter, the authors construct a suite of diagnostics that isolate each capability.

Filtering the SVG Blind VQA set reduces its size by more than half.

From the original 3,000 generated examples, only 1,370 survive the “base model answers incorrectly” filter.

The SVG Blind VQA diagnostic asks the model to mentally render SVG code and answer multiple‑choice questions about the resulting image.

This component tests whether the model can predict the visual outcome of a short camera motion described by six primitive transforms.

This component asks the model to infer a missing viewpoint given orthogonal front and left images.

Here the model receives a pair of views (right and front) and must predict the opposite side, testing its understanding of view symmetry.

This component evaluates whether the model can turn a structured 3‑D scene description into a realistic 2‑D rendering.

The 3D object annotations list ten objects with precise centers and dimensions, providing a full scene graph for the layout‑to‑image task.

The accompanying kitchen scene description supplies a realistic visual target for the layout‑to‑image generation component.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers