VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs
Yuan Feng, Qize Yang, Ruizhe Chen, Sibo Song, Haolin He, Muzhi Zhu, Zihan Liu, Yunfei Chu, Xize Cheng, Yuxuan Wang, Jin Xu, Xike Xie
VisionWeave enables MLLMs to dynamically weave fine- and coarse-grained visual tokens, reducing sequence length by 43%.
How can MLLMs dynamically adjust visual token density based on image content rather than using fixed-size patch grids?
Multimodal Large Language Models (MLLMs) process visual inputs as dense, fixed-size patch tokens, forcing a trade-off between high-resolution detail and computational efficiency. Existing pruning methods often discard task-critical information and struggle to integrate with modern serving engines. VisionWeave introduces a native capability where the model learns to allocate either fine-grained or coarse-grained representations to each spatial block based on content. It uses a gated spatial pooler to construct coarse tokens and a granularity router to decide the optimal resolution for every region. On Qwen3.8-27B, this approach saves 43% of visual tokens on average while retaining 98.9% of native performance, yielding a 2.30× throughput gain in SGLang.
Paper Primer
VisionWeave treats visual granularity as a learnable degree of freedom rather than a fixed preprocessing step. The core move is a granularity router that probes ViT features at multiple depths to decide, per spatial block, whether to retain four native fine-grained tokens or replace them with a single coarse-grained token constructed by a gated spatial pooler.
VisionWeave achieves content-adaptive token savings with minimal quality degradation.
Across eight benchmarks, Qwen3.8-27B saves an average of 43.0% of visual tokens while preserving 98.9% of native performance. Outperforms heuristic pruning baselines (FastV, VisionZip), which retain only ~88% performance at similar savings targets.
Native integration enables significant real-world serving speedups.
Deployment on the SGLang engine achieves a 2.30× end-to-end throughput gain. Reduces mean Time to First Token (TTFT) by 54.4% and mean Time Per Output Token (TPOT) by 60.6%.
Why is this approach more robust than simple token pruning?
Simple pruning often discards tokens based on heuristic importance, which can leave gaps in spatial coverage or remove task-critical details. VisionWeave maintains full spatial coverage by replacing fine-grained tokens with coarse-grained ones, ensuring the model still "sees" every region of the input.
Does this method require retraining the model for different efficiency needs?
No. The model supports inference-time control via a routing threshold. Lowering the threshold increases the use of coarse-grained representations, allowing users to trade off quality for higher token savings without retraining.
VisionWeave demonstrates that representational granularity can be a native, learnable capability of MLLMs, effectively shifting the burden of efficiency from manual heuristics to the model's own internal routing logic.
Motivation and Core Concept
Fixed‑size patch tokens waste compute in MLLMs, prompting the need for elastic visual weaving.
Multimodal Large Language Models (MLLMs) have become the dominant paradigm for visual understanding, supporting tasks such as visual question answering, document analysis, and video comprehension. As these tasks demand higher spatial resolution and longer video sequences, MLLMs process ever‑larger images and token streams, but their visual inputs are still partitioned into fixed‑size patches that inflate sequence length and thus computational and memory cost.
Instead of forcing every region of an image into the same patch granularity, the model dynamically decides where to keep fine‑grained tokens and where a coarser representation suffices, weaving the two together into a single token stream.
FastV applies a uniform token‑pruning schedule that drops a fixed proportion of visual tokens regardless of image content.
VisionZip merges neighboring visual tokens into a single token based on a learned similarity metric, aiming to reduce token count while preserving salient structures.
These existing token‑pruning approaches illustrate the core inefficiency: a one‑size‑fits‑all patch grid cannot adapt to the heterogeneous visual complexity present in real‑world inputs, causing unnecessary compute waste or loss of critical detail.
The fixed‑size patch tokenization of current MLLMs wastes compute by processing visual detail that many regions do not require.
The VisionWeave Architecture
We detail how VisionWeave dynamically weaves coarse and fine visual tokens to cut compute.
Processing every fine‑grained visual token wastes compute because many patches are already confidently predicted. VisionWeave therefore introduces a dynamic routing mechanism that decides, per spatial block, whether to keep the four fine tokens or replace them with a single pooled token.
The pooler learns to compress a $2\times2$ patch region into one token by weighting each patch’s features according to its content, rather than naïvely averaging.
Layer‑normalize each vector (trivial for this toy example) → $u_1=(1,2)$, $u_2=(2,1)$, $u_3=(0,0)$, $u_4=(1,1)$.
Compute the global context $g_k$ by concatenating $u_1\!-\!u_4$ and passing through a zero‑weight MLP → $g_k$ is a zero vector.
Gate values $\alpha_k$ become uniform $0.25$ for each channel because the softmax of zeros is equal.
Refine each $u_k$ with $f_{\text{val}}$ (zero) → $v_k = u_k$.
Aggregate: $P = 0.25\cdot(u_1+u_2+u_3+u_4) = (1.0,1.0)$.
When the gate collapses to uniform weights, the pooler behaves exactly like mean pooling, confirming the design’s backward compatibility.
How does the Gated Spatial Pooler differ from simple average pooling?
Average pooling treats every channel equally and ignores the surrounding context. The gated pooler first builds a global context from the four patches, then computes channel‑wise gates that can amplify or suppress specific dimensions before summing, allowing the pooled token to retain salient details that uniform averaging would wash out.
The router learns, for each spatial block, whether the coarse token is sufficient or whether the four fine tokens should be kept, by probing ViT features at multiple depths with a shared query.
Compute softmax over logits: $\exp(0.3)=1.35$, $\exp(-0.1)=0.90$, sum $=2.25$.
Coarse probability $p_{00}=1.35/2.25 \approx 0.60$.
Since $p_{00}>t=0.5$, the router selects the coarse token for block $(0,0)$.
The coarse token is placed at the block‑center coordinate $(2\cdot0+0.5, 2\cdot0+0.5) = (0.5,0.5)$.
Even with modest logit differences, the softmax can push the probability over the threshold, illustrating how the router can make decisive coarse selections without needing a large margin.
Why not simply use a fixed rule (e.g., always keep fine tokens) instead of learning $p_{IJ}$?
Fixed rules cannot adapt to image content; some regions are homogeneous and can be safely compressed, while others contain critical details. The learned $p_{IJ}$ leverages both low‑level texture and high‑level semantics, enabling the model to keep fine tokens only where they matter, thereby saving compute without sacrificing accuracy.
The training pipeline gradually introduces the routing mechanism: first the pooler learns a coarse representation, then the router learns a soft mixing of coarse and fine tokens, and finally the hard routing used at inference is distilled.
Why is a multi‑stage distillation needed instead of training the full system end‑to‑end?
Training end‑to‑end would require the router’s hard decisions to be differentiable, which is impossible. The staged approach first teaches the pooler, then lets the router learn a smooth proxy, and finally aligns the proxy with the hard routing used at test time, ensuring stable training and high final performance.
**Figure 3.** Overall architecture of VisionWeave. The gated spatial pooler complements native fine-grained tokens $V_{fine}$ with coarse-grained representations $V_{coarse}$ within the same MRoPE coordinate, while the granularity router adaptively weaves them into a mixed-granularity sequence for the LLM.
**Algorithm 1** VisionWeave inference with elastic visual representations **Require:** An $H \times W$ patch grid for an image; textual context $c$; routing threshold $t$ (default 0.5) 1: Obtain $X^{(0)}$, $X^{(L/2)}$, $X^{(L)}$ in a single ViT forward pass 2: $V_{\text{fine}} \leftarrow \mathcal{M}(X^{(L)})$; $V_{\text{coarse}} \leftarrow \mathcal{M}(\mathcal{P}(X^{(L)}))$ 3: Initialize $h_{IJ}$ from the shared query with block-center RoPE coordinates 4: **for** $\ell \in \{0, L//2, L\}$ **do** 5: $\quad h_{IJ} \leftarrow \text{LocalCrossAttn}^{(\ell)}(h_{IJ}, X^{(\ell)})$ 6: $\quad h_{IJ} \leftarrow \text{GlobalCrossAttn}^{(\ell)}(h_{IJ}, X^{(\ell)})$ 7: **end for** 8: Compute $p_{IJ}$ using the calibrated router head 9: **for** each block $(I, J)$ **do** 10: **if** $p_{IJ} > t$ **then** 11: Select $V_{\text{coarse},IJ}$ with its block-center coordinate $\pi_{IJ}$ from Eq. (3) 12: **else** 13: Select $\{V_{\text{fine},2I+a,2J+b}\}_{a,b \in \{0,1\}}$ with their native coordinates from Section 2.1 14: **end if** 15: **end for** 16: Gather selected tokens into $V_{\text{routed}}$ in native raster order 17: **return** $p_{\theta}(y \mid c, V_{\text{routed}})$
Empirical Performance and Efficiency
VisionWeave delivers large serving gains while keeping accuracy near native levels.
We evaluate VisionWeave against fixed‑budget baselines across eight diverse benchmarks, measuring token savings, quality retention, and serving efficiency. The experiments follow the three requirements defined earlier: adaptive token reduction, robustness to resolution and frame budgets, and practical deployment gains.
VisionWeave yields a 2.30× end‑to‑end throughput speedup on SGLang compared with the native Qwen 3.8‑27B model.
Table 5 shows throughput increasing from 0.90 req/min to 2.07 req/min while latency metrics improve.
**Figure 2.** VisionWeave achieves content-adaptive token savings with near-native quality, whereas fix-budget baselines degrade sharply on challenging tasks. Visual case studies on ScreenSpotV2 and DocVQA illustrate this difference (Appendix B). Our VisionWeave still retains higher quality even when baselines are granted its adaptive saving ratios (Appendix E.3).
**Figure 4.** Training dynamics of three-stage self-distillation.
**Figure 5.** VisionWeave achieves consistently favorable trade-offs across tasks and input resolutions
**Figure 6.** VisionWeave maintains favorable trade-offs across frame budgets on LongVideoBench.
**Table 4.** Content-adaptive token savings with near-native quality.
**Table 5.** VisionWeave accelerates Qwen3.8-27B serving on SGLang.
**Figure 8.** VisionWeave enables flexible inference-time control of token savings while closely matching native-model performance when savings are disabled.
VisionWeave maintains near‑native accuracy while dramatically reducing visual token counts, delivering large serving speedups.
Ablation Studies and Component Analysis
We isolate each component’s impact, showing how gating and learned granularity drive performance.
The central premise is that VisionWeave dynamically weaves coarse and fine visual tokens to cut compute while preserving accuracy. This section quantifies how each ablation contributes to that goal.
Gated spatial pooler outperforms a pixel‑unshuffle projector across all eight benchmarks.
Higher scores on every benchmark, e.g., +2.64 points on DocVQA and +2.77 points on VideoOCR, at a fixed 75 % token‑saving budget.
Shuffling block‑wise routing scores (i.e., removing learned granularity allocation) degrades performance.
Mean scores drop by 2.31 points on Qwen3.5‑4B and 5.36 points on Qwen3.8‑27B; average drops of 3.18 and 6.54 points respectively.
**Figure 9.** VisionWeave maintains spatial coverage through mixed granularity, whereas FastV$^\dagger$ drops tokens covering task-relevant content.
**Figure.** Comparison of Pixel-unshuffle and Gated spatial pooler across various benchmarks.
**Table 6.** Learned granularity allocation improves performance over shuffled routing.
Limitations and Supplementary Data
Limits, case studies, and extra data for VisionWeave.
Our ablations reveal that the high computational cost of training limits a deeper search over router designs and pooler variants, leaving many promising alternatives unexplored.
We anticipate that extending the current two‑level design to richer multi‑scale representations could unlock further gains, and that elastic visual weaving might eventually be introduced during base‑model pre‑training rather than only via post‑training self‑distillation.
Case studies illustrate why VisionWeave’s interleaved fine‑ and coarse‑grained tokens outperform token‑pruning baselines on grounding and document‑VQA tasks.
In GUI grounding, VisionWeave keeps every region within a shared spatial frame, whereas FastV drops tokens covering critical click targets such as the star button, creating gaps that hurt grounding accuracy.
For DocVQA, at the default routing threshold $t=0.5$, VisionWeave saves 45.2 % of tokens while preserving all fine‑grained tokens for dense tables; FastV removes 50 % of tokens, including essential text characters.
Additional visualizations on Qwen3.5‑4B and Qwen3.8‑27B show how fine‑ and coarse‑grained representations are woven according to visual content, confirming the method’s consistency across model scales.
Related work on visual token pruning (e.g., FastV, VisionZip) often relies on fixed compression ratios and lacks implementations in modern serving engines, limiting practical adoption.
Content‑adaptive visual representations that vary patch size by low‑level statistics target standalone vision tasks, while approaches like Matryoshka, ViCO, and PARCEL require external budget specifications or tile‑based designs that do not map cleanly onto native‑resolution MLLMs.
KV‑cache compression reduces memory during inference but does not shorten the input token sequence; combining it with elastic visual weaving could further lower both compute and cache footprints.
Additional experimental details include ablations of mixed‑granularity self‑distillation (Stage 3) and distillation policy, as well as matched‑savings performance and the SGLang serving configuration.
**Table 7.** Stage 3 improves Qwen3.8-27B performance on seven of eight benchmarks.
**Figure 17.** On- and off-policy self-distillation yield comparable performance on Qwen3.5-4B.
**Table.** Performance comparison of Qwen3.5-4B and Qwen3.8-27B models across various benchmarks using different methods (FastV, VisionZip, VisionWeave) compared to an Upper bound (512 tokens).
The SGLang serving configuration uses Python 3.12.3, PyTorch 2.13.0+cu129, Transformers 5.12.1, and Triton 3.7.1 on two A100‑SXM4‑80GB GPUs with BF16 and custom all‑reduce disabled, enabling a context limit of 147,456 tokens and 4,096‑token prefill chunks.