What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models
Tengju Ru, Weitong Lian, Zecong Tang, Zhiheng Liu, Lingyi Meng, Haoran Li, Kaixuan Wang, Bohao Li, Xihui Liu
GAFU unlocks fine-grained visual understanding in multimodal models by injecting native generation-side representations into the understanding pathway.
How can we extract and inject the fine-grained visual information hidden in a multimodal model's generation-side representations to improve its understanding capabilities?
Unified multimodal models (UMMs) struggle with fine-grained visual tasks like counting or spatial reasoning because their understanding-side encoders compress away the local details needed for these tasks. The authors show that generation-side representations naturally preserve these details and propose GAFU, a framework that compresses this generation-side information into compact tokens and injects them into the understanding pathway via task-conditioned retrieval. This approach consistently outperforms standard supervised fine-tuning across diverse architectures, with relative improvements of up to 20% on fine-grained benchmarks.
Paper Primer
The core move is to treat the generation pathway as a source of high-fidelity visual evidence. GAFU compresses these native representations into "G tokens" using an encoder-decoder architecture that enforces spatial binding and relational alignment, ensuring the understanding pathway can selectively retrieve relevant visual cues based on the current text context.
GAFU provides consistent performance gains over standard supervised fine-tuning (SFT) across all tested UMM architectures and benchmarks.
Evaluation on 15 model-benchmark combinations, including general, spatial, and domain-specific tasks. Average gains of 3.56–5.98 percentage points, with maximum relative improvements of 20.00% on VStar-Bench and 17.94% on OmniSpatial.
The framework relies on a pairwise preference learning objective (DPO) in the cross-entropy space to force the model to prioritize sample-specific generation evidence over generic visual tokens, while a KL-divergence constraint prevents the model from degrading when mismatched generation tokens are provided.
Why does the generation pathway contain information that the understanding pathway misses?
Generation pathways require richer structural and local detail to reconstruct images, whereas understanding pathways are typically optimized for high-level semantic compression, which often discards the fine-grained spatial and attribute-level evidence needed for tasks like counting or relation reasoning.
Is this method just a more complex way of doing standard visual encoding?
No; GAFU specifically exploits the native generation-side representations that already exist within UMMs. It avoids the bottleneck of re-encoding visual inputs by using a post-training framework to align and inject existing generation-side features into the understanding pathway.
Researchers can now leverage the generation pathways of UMMs as a "visual memory" to resolve fine-grained understanding failures without needing to retrain the primary visual encoders.
Motivation and Problem Framing
Unified multimodal models hide fine‑grained visual cues in generation outputs that understanding fails to use.
Unified multimodal models (UMMs) encode fine‑grained visual cues in the representations produced by their generation pathway ($G$), but the understanding pathway ($U$) fails to exploit this latent information. This underutilization creates a bottleneck for fine‑grained visual tasks.
RG is the visual embedding that the model produces while generating text or images, preserving detailed spatial layout and attribute information.
RU is the visual embedding consumed by the understanding pathway, optimized for high‑level semantic reasoning rather than pixel‑level detail.
**Figure 1.** Control-corrected generation–understanding sensitivity gaps and strong-response complementarity across six models and four domains. (a) Solid bars show the unpermuted $G - U$ gap, while hatched bars show the gap after spatial shuffle. (b) Stacked bars show the fractions of samples with generation-side-only and understanding-side-only strong responses.
The gap between generation and understanding representations is the primary bottleneck for fine‑grained tasks.
The GAFU Framework
We compress generation representations into G tokens, align them spatially, and inject them conditionally into the understanding pathway.
Generation‑side representations contain fine‑grained visual cues that the understanding pathway rarely exploits, creating a hidden performance gap on detailed vision tasks.
The framework extracts the latent fine‑grained visual signal stored in generation‑side tokens, aligns it with the understanding side, and makes it queryable during downstream tasks.
How does GAFU differ from a standard retrieval‑augmented generation (RAG) system?
RAG fetches external documents and inserts them as additional context, whereas GAFU reuses the model’s own generation‑side latent image features, aligns them with the internal understanding representation, and queries them directly without any external memory.
We compress the generation representation into a fixed‑size token set $G$ that still knows where each visual fragment originated, then align $G$ with the understanding side through reconstruction and relational losses.
Encoder attends each query to all 16 patches; query 1 aggregates patches {1,2,3,4}, query 2 aggregates patches {5,6,7,8}.
Each query output is projected to a $G$ token of dimension 64.
Decoder receives the two $G$ tokens and reconstructs a $4\times4$ feature map; the $L_{\text{rec}}$ loss penalizes the difference to the original map.
Spatial binding head $h_{\text{spa}}$ maps $G_1$ to the average of its source patches (positions (0,0)–(0,1)) and $G_2$ to positions (1,0)–(1,1).
Relation alignment pools the $4\times4$ map to $2\times2$ grids, computes cosine similarity matrices for $G$ and $U$, and minimizes $1-\cos(\cdot)$ between them.
Even with only two queries, the $G$ tokens preserve enough spatial and relational cues to reconstruct the original layout, demonstrating that compression does not discard fine‑grained evidence.
Why not simply use a vanilla auto‑encoder bottleneck instead of the query‑based compression?
The query mechanism forces each $G$ token to attend to a specific spatial region, preserving locality; a vanilla bottleneck would mix all patches indiscriminately, erasing the spatial bindings that later alignment relies on.
During understanding, each transformer layer queries the $G$ tokens; the cross‑attention selectively injects the fine‑grained visual evidence that the current textual context needs.
How is this injection different from ordinary cross‑attention between text and image tokens?
Ordinary cross‑attention attends to the full set of frozen image tokens $U$, which the model already sees; task‑conditioned injection attends only to the compact $G$ tokens that have been explicitly aligned and are therefore guaranteed to carry the fine‑grained evidence the model needs for the current prompt.
**Figure 2.** Overview of GAFU, comprising generation representation compression and alignment (Stage 1) and task-conditioned injection (Stage 2).
Empirical Evaluation
GAFU delivers consistent gains on fine‑grained visual tasks across models.
GAFU improves fine‑grained visual understanding by up to 7.9 percentage points over SFT across three benchmarks.
Table 1 shows improvements of 5.23–7.85 pp on VStar‑Bench, 2.61–4.44 pp on OmniSpatial, and 3.67–5.67 pp on MMVP.
A suite of tasks that probe a model’s ability to perceive tiny visual details, attribute variations, and local spatial relations.
A task family that asks models to answer relational questions about object positions and contacts within an image.
**Figure 3.** GAFU enhances fine-grained visual understanding. Examples show better recognition of fine-grained details, spatial relations, and domain-specific cues over the base model and SFT.
**Figure 6.** Fine-grained local spatial-relation example from VSR.
**Figure 7.** Fine-grained pathological example from PathVQA.
GAFU consistently outperforms base models on fine‑grained spatial and attribute‑based tasks.
Representation Sensitivity Analysis
We revisit the premise that generation‑side representations contain untapped fine‑grained visual cues.
We extract the understanding‑side representation $R_U$ and the generation‑side representation $R_G$ from each UMM using model‑specific pipelines, then align both to a common $16\times16$ grid via adaptive average pooling.
BAGEL: $R_U$ = SigLIP visual tokens (16×16); $R_G$ = KL‑VAE posterior mean (64×64) → 2×2 patchification → 32×32 grid. Janus‑Pro: $R_U$ = SigLIP‑L/16 tokens; $R_G$ = quantized VQ‑16 embeddings on 24×24 grids. Lance: $R_U$ = merged Qwen tokens (16×16); $R_G$ = frozen Wan2.2‑VAE posterior mean (32×32). Show‑o2‑7B: both representations are adaptively average‑pooled to 16×16 (channels 1152 for $R_U$, 3584 for $R_G$). BLIP3‑o‑4B: $R_U$ = merged Qwen2.5‑VL tokens (16×16); $R_G$ = EVA‑CLIP‑E/14‑plus generation tower features (32×32). DreamLLM‑7B: $R_U$ = CLIP ViT‑L/14 penultimate features (16×16, CLS excluded); $R_G$ = VAE posterior mean with native scaling (64×64).
For control‑corrected construction we first compute per‑pixel mean‑squared RGB difference $e(p)=\frac{1}{3}\sum_{c=1}^{3}\bigl(I_v(p,c)-I(p,c)\bigr)^{2}$, select the top 25 % non‑zero pixels, dilate twice to obtain mask $M$, and enforce equal RMS intensity within $M$ between the semantic edit $I_v$ and the control image $I_n$ via $X_n = X + \frac{\text{RMS}_M(X_v-X)}{\text{RMS}_M(D)}\,D$.
Generation‑side representations yield substantially more strong responses than understanding‑side representations on average.
Table 3 reports $P_{G\text{-only}} = 28.18\%$ versus $P_{U\text{-only}} = 2.48\%$ across all model–domain pairs.
Lance exhibits the largest attribute‑domain advantage.
In Table 3, Lance’s attribute gap $\Delta = 56.80$ pp, exceeding all other models.
Lance also leads in the object domain.
Table 3 shows Lance’s object gap $\Delta = 52.77$ pp, the highest among the six models.
Resolution‑stability experiments (Table 4) confirm that the control‑corrected generation–understanding gap stays positive for every model, domain, and input resolution (128, 256, 512), indicating the advantage is not resolution‑dependent.
Spatial‑dependence tests permute blocks on a large canvas before encoding; the gap persists, showing that the generation‑side advantage does not rely on the original spatial layout.
**Table.** Performance comparison across different models and domains, showing $P_{G\text{-only}}^{(d)}$, $P_{U\text{-only}}^{(d)}$, and the difference $\Delta_{\text{comp}}^{(d)}$.
**Table 4.** Resolution stability of the control-corrected generation–understanding response gap.
Injection and Fine-Tuning Details
Ablation study shows how each injection component and training objective shapes performance.
The paper showed that generation‑side representations contain fine‑grained visual cues that the understanding side under‑utilises; here we test how each injection component and objective contributes.
The injection pathway feeds generation‑side tokens into the text encoder so the model can exploit visual details while answering.
**Figure 4.** Stage 1 reconstruction visualization. Each case compares the original image, the VAE upper bound obtained by decoding the native generation representation, and the reconstruction from the compressed $G$ tokens.
**Figure 8.** Stage 2 validation dynamics under different fine-tuning objectives. The three columns correspond to CE only, CE + DPO, and CE + DPO + KL, respectively. The first row reports the answer-token CE of the real and shuffle-G branches, while the second row reports their CE gap, defined as $CE_{shuffle-G} - CE_{real}$. The real branch uses matched G tokens, whereas the shuffle-G branch uses mismatched G tokens from another sample. A positive CE gap indicates that the model assigns lower answer-token cross-entropy to matched G tokens.
Experimental Data and Setup
Dataset, training, evaluation details, and limitations for the GAFU framework.
D.1 DATA lists the image collections used for each training stage. The compression‑and‑alignment stage draws 1 M stratified images from the Cambrian Dataset. Task‑conditioned fine‑tuning samples 1,344,016 high‑relevance items from FineVision, 7,680 training and 2,195 test items from VSR‑Bench, and 19,654 training plus 6,719 test items from PathVQA. Representative examples illustrate fine‑grained multi‑object reasoning, local spatial‑relation checks, and pathological visual‑question answering.
A large, stratified collection of one million images used to learn a generic visual compression and alignment that the GAFU framework can later inject into downstream tasks.
D.2 TRAINING details the two stages. Stage 1 uses AdamW with a peak learning rate of $10^{-4}$, linear warmup for 200 steps, then cosine decay to $10^{-5}$, weight decay $0.01$, betas $(0.9, 0.95)$, $\epsilon=10^{-8}$, and gradient clipping at norm 1.0, running on 8 GPUs with a global batch size of 32 for 8 000 optimizer steps. Stage 2 keeps the same batch size for 5 000 steps; the injection projector and internal attention modules use a peak learning rate of $2\times10^{-5}$, while the understanding‑pathway LoRA uses $10^{-4}$. Both are warmed up for 500 steps and decay to $2\times10^{-6}$ and $10^{-5}$ respectively; LoRA rank and $\alpha$ are set to 16, and AdamW weight decay is zero.
D.3 EVALUATION describes the inference protocols. VStar‑Bench evaluates 191 multiple‑choice items by shuffling option positions and selecting the answer with the lowest average answer‑token $CE$. OmniSpatial uses 1 533 spatial‑reasoning samples and a forced‑choice rule based on average candidate‑token $CE$. MMVP evaluates 300 questions (150 complementary pairs) similarly. VSR‑Bench applies a greedy True/False generation on its 2 195 test samples. PathVQA runs greedy decoding on all 6 719 test items and reports normalized exact‑match accuracy after lowercasing and punctuation stripping.
E LIMITATIONS notes three constraints. First, GAFU currently supports only UMMs with mainstream dual‑pathway visual encoders, so models with alternative architectures need extra adaptation. Second, the gains depend on the quality of the native generation‑side representations, leading to variable improvements across models and tasks. Third, extracting, projecting, and injecting generation tokens adds extra inference computation and memory overhead compared with using the native understanding pathway alone.
Conclusion and Broader Impact
We wrap up GAFU results, disclose AI assistance, and outline ethical and reproducibility commitments.
GAFU leverages native generation‑side representations to boost fine‑grained visual understanding across UMM architectures. By compressing, spatially binding, and aligning these representations, the framework extracts complementary visual evidence that the understanding pathway can directly consume. Empirically, GAFU achieves up to 20.00 % relative improvement on VStar‑Bench, 17.94 % on OmniSpatial, and 8.62 % on VSR‑Bench.
Generative AI tools were employed solely for translation and editorial polishing of the manuscript. They were not used to generate synthetic data, develop models, formulate hypotheses, write code, or perform any core research activities. All AI‑assisted text was reviewed and approved by the authors, who retain full responsibility for the final content.
This work utilizes existing benchmark suites—including Cambrian, FineVision, VStar‑Bench, OmniSpatial, MMVP, VSR‑Bench, VisMin, and PathVQA—under their respective citation and data‑use policies. The medical results on PathVQA are intended only for research benchmarking and must not inform clinical decisions. No privacy‑sensitive data, human subjects, new data collection, or harmful insights are involved, and the authors declare no conflicts of interest.
For reproducibility, the systematic analysis can be recreated using the procedures described in Section 3.1 and Appendix A, which detail representation extraction and sensitivity analyses. GAFU’s training and evaluation pipelines are documented in Section 3.2, Section 4.1, and Appendices B‑D, covering architecture, objectives, and evaluation protocols.
Implementation Details
Details on the grid layout, Stage 1 compression, and spatial‑permutation effects.
The canvas is divided into a fixed $64\times64$ grid of image blocks, which yields $8\times8$, $6\times6$, and $7\times7$‑pixel blocks for the three resolutions used in the experiments.
Both permuted and unpermuted conditions share identical representation extraction, common‑grid alignment, and control‑correction pipelines, ensuring that any observed differences stem from spatial permutation alone.
Table 4 (not reproduced here) quantifies how the control‑corrected generation–understanding response gap varies with resolution, confirming that the gap persists across scales.
Applying spatial permutation consistently narrows the generation–understanding gap across all 24 model–domain pairs, supporting the hypothesis that the generation‑side advantage relies on spatial organization.
Stage 1, labeled B COMPRESSION AND ALIGNMENT, freezes the native understanding and generation pathways of the unified multimodal model and trains only the GToken Encoder, Decoder, and their prediction heads.
The GToken Encoder–Decoder follows a symmetric six‑layer Transformer (hidden size $1024$, $8$ attention heads, $4096$‑dimensional feed‑forward network) and incorporates 2D RoPE for spatial position encoding.
The Encoder first projects the generation representation to $1024$ dimensions, appends $256$ learnable queries, and runs joint self‑attention with the spatial positional encoding to aggregate generation information.
The resulting query outputs are projected into the hidden dimension of the understanding‑pathway tokens, yielding a fixed‑length generation memory $G$; its dimensionality is $3584$ for BAGEL, $4096$ for Janus‑Pro, and $2048$ for Lance.
The Decoder maps $G$ back to $1024$ dimensions, concatenates reconstruction queries placed according to the source grid, and reconstructs the original generation representation via joint self‑attention followed by a linear prediction head.
For spatial binding, $zbind$ is constructed on a $16\times16$ grid of the $256$ $G$ tokens. BAGEL and Lance form $zbind$ by concatenating the matching $2\times2$ region of the $32\times32$ native generation latent, whereas Janus‑Pro obtains it by adaptively average‑pooling its native $24\times24$ grid to $16\times16$.
Each $G_i$ predicts the $zbind$ target‑energy‑normalized mean squared error, providing the supervision signal for the compression‑alignment stage.
**Table 5.** Spatial dependence of control-corrected representation sensitivity.