WeVisOne-VL: Unified Visual Perception from Objects to Pixels
Tianming Liang, Junyi Pan, Yuxi Li, Canmiao Fu, Zhipeng Huang, Chong Sun, Chen Li, Jing Lyu
(not extracted)
How can a single vision-language model unify object-level detection and pixel-level segmentation tasks using a shared language-based interface?
(primer unavailable — see sections below)
Paper Primer
Primer unavailable — open the sections below for the full paper.
Introduction
WeVisOne-VL unifies object‑level and pixel‑level perception via a single language‑driven interface.
Current VLMs struggle with fine‑grained visual tasks: they often hallucinate nonexistent objects, misinterpret user intent, or miss small, non‑salient instances, which hampers real‑world deployment.
A single model that can name objects, locate them with boxes or points, and also output pixel‑accurate masks or alpha mattes, all driven by the same natural‑language prompt.
**Figure 1.** Overview of WeVisOne-VL's unified object- and pixel-level perception capabilities.
The shift from a zoo of task‑specific models to a unified VLM‑based perception system simplifies deployment and opens the door to truly reliable visual assistants.
The WeVisOne-VL Data Engine
The WeVisOne-VL Data Engine builds a unified dataset linking categories, expressions, and masks for visual perception.
Unified visual perception requires data that simultaneously supports category recognition and instance‑level mask supervision. Existing pipelines treat these tasks separately, leading to mismatched annotations and wasted effort.
It treats each image–query pair as a single data unit, aligning category concepts, referring expressions, and mask annotations through a shared knowledge graph.
How does the WDE differ from a conventional data‑collection pipeline that simply aggregates images and labels?
Instead of treating detection and referring data as independent label sets, WDE ties them together via a knowledge graph, so each query—whether a category name or a referring expression—maps to the same underlying image region, preserving instance identity across tasks.
Link “cat” to a node and add the alias “feline”.
Associate the expression “the sleeping cat” with the “cat” node and record the attribute “sleeping”.
Retrieve images containing “cat” or “dog”: I₁ contains a cat, I₂ a dog, I₃ both.
For each image, generate proposals and run Qwen3‑VL verification; keep proposals that satisfy both the category and any attached expression constraints.
Resulting annotations: I₁ receives a mask for the cat, I₂ a mask for the dog, I₃ masks for both the cat and the dog.
The graph ensures that the same visual instance is reachable both by its category label and by any referring expression that describes it, enabling unified supervision across tasks.
Build a shared knowledge graph from Wikipedia, WordNet, and visual evidence.
Sample image–query pairs using graph terms as textual retrieval queries, prioritizing under‑represented concepts and hard visual conditions.
Generate detection proposals in parallel from SAM3 and an open‑vocabulary detector; merge them into a candidate pool.
Verify each candidate against the category query with Qwen3‑VL; retain bounding boxes for the verified instances.
Generate referring proposals with SAM3 Agent and Rex‑Omni; verify each against the referring expression with Qwen3‑VL, producing binary masks.
Optionally run the matting pipeline to convert binary masks into continuous alpha supervision.
After each training round, review model errors; feed the error signals back to adjust acquisition priorities and annotation refinements.
**Figure 2.** Overview of the WeVisOne-VL Data Engine (WDE).
Mask Annotation and Processing
We detail how SAM2 masks, hard examples, and matting pipelines enrich unified visual perception data.
Binary instance masks provide only coarse silhouettes, which hampers learning fine‑grained boundaries needed for high‑quality matting. The pipeline therefore requires a mechanism that can turn a box query into a precise pixel‑level mask while staying consistent with the original instance description.
SAM2 takes a prompt (e.g., a bounding box) and instantly produces a dense binary mask that aligns with the prompted region, enabling per‑instance segmentation without extra training.
How does SAM2 differ from classic region‑based segmentation methods like GrabCut?
GrabCut iteratively refines a binary mask using graph cuts and requires an initial foreground/background scribble. SAM2, by contrast, produces the mask in a single forward pass from a box or text prompt, without any iterative optimization, and its mask head is trained to respect the prompt directly.
SAM2 extracts a feature map for the whole 8×8 image.
The box prompt is encoded and injected into the mask head, which outputs a 4×4 binary mask covering the car.
The resulting mask aligns perfectly with the original box, demonstrating prompt‑consistent segmentation.
Even with a minimal prompt, SAM2 can generate a mask that respects the spatial extent of the queried object, enabling downstream tasks to reuse the same query for both box and mask supervision.
Mask Annotation (2.4) uses SAM2 to predict a binary mask for each localized instance. The mask inherits the query of its originating box, so every instance possesses both a box and a mask under the same prompt, supporting both detection and segmentation supervision.
Hard‑Example Construction (2.5) creates challenging positives and negatives by sampling confusable concepts, visually similar categories, and model failure cases. Negatives are only labeled when no object satisfies all query conditions, and hard negatives arise when the current model still predicts a target region for such a query.
Expert‑assisted Annotation: for each binary mask $S$, obtain two alpha candidates—soft foreground $\alpha_B$ from BiRefNet and alpha matte $\alpha_M$ from MatAnyone2.
Validate BiRefNet candidate $B = \mathbf{1}[\alpha_B \ge \tau_\alpha]$ using criteria: $|B| \ge 8$, $\text{IoU}(B, S) \ge 0.5$, coverage $|B \cap S|/|S| \ge 0.5$, precision $|B \cap S|/|B| \ge 0.8$, area ratio $|B|/|S| \in [0.5, 2]$, box IoU $\ge 0.7$, centroid offset $\le 0.15$ × diag($S$).
If BiRefNet passes, keep connected components of $\alpha_B \ge 0.01$ with at least $10\%$ overlap with a dilated $S$; otherwise fall back to $\alpha_M$ if $\text{IoU}(\mathbf{1}[\alpha_M \ge \tau_\alpha], S) \ge 0.9$, else retain $S$.
Matting Annotator Training: train a lightweight alpha refinement head on a balanced mix of initial labels and high‑quality alpha mattes, using multi‑scale L1, Laplacian, and prompt‑consistency losses.
Corpus Re‑annotation: apply the trained annotator to all instances, replacing initial masks with the predicted alphas while preserving the original description and bounding box linkage.
Training Feedback and Data Refinement (2.7) analyzes model errors after each training round, categorizes them, and directs annotation effort toward persistent failure modes. Errors that stem from annotation defects are corrected before the next data release, while model‑driven hard examples are incorporated to sharpen the learner.
Unified Task Formulation
Defines a compact token scheme that unifies detection, segmentation, and other visual tasks.
Earlier multimodal models emitted separate streams for text and geometry, inflating sequence length and complicating training. Our trick is to encode geometry directly as a compact series of special tokens, letting a single language decoder handle everything.
We treat every visual request as a plain‑language prompt and let the model answer by writing a short token script that mixes text tags with tiny coordinate tokens—like a recipe where each ingredient amount is a token.
Model writes the reference tag: <ref>person</ref>.
It opens a box field: <box>.
It emits the four coordinate tokens in order: <82> <105> <301> <947>.
It closes the box field: </box>.
This walk‑through shows that a full bounding‑box description fits in just six tokens, dramatically shrinking the output compared to verbose coordinate strings.
How does this token scheme differ from using separate decoder heads for boxes and masks?
Separate heads require additional parameters and a second forward pass, breaking the unified sequence generation. The token scheme keeps everything in the language model’s autoregressive stream, so geometry and text share the same parameters and inference path.
**Figure 3.** Representative input–output examples for the unified task formats used by WeVisOne-VL.
Model Architecture
We integrate a lightweight pixel decoder with Qwen3-VL to enable unified object-level and pixel-level perception.
To achieve unified perception, WeVisOne-VL extends the Qwen3-VL vision-language model with a specialized pixel-level decoder. Instead of relying on external segmentation models like SAM, the architecture generates dense spatial masks directly from the model's internal object tokens and box predictions.
Qwen3-VL serves as the backbone, providing both the visual features and the autoregressive reasoning needed to identify objects and their spatial extents.
The pixel decoder is a compact, 9.6M parameter module that transforms abstract object tokens and box priors into high-resolution masks or alpha mattes.
How does this decoder differ from standard segmentation models like SAM?
Standard models like SAM are standalone, large-scale architectures that require separate inference passes. Our decoder is instance-conditioned: it uses the specific <OBJ> token's hidden state and the model's own predicted box as spatial guidance, allowing it to be significantly smaller (9.6M parameters) while remaining tightly integrated with the VLM's semantic reasoning.
**Figure 4.** Overview of the lightweight pixel decoder. Qwen3-VL provides multi-level ViT features, the hidden state of each generated <OBJ> token, and its predicted box. The decoder builds shared visual features, updates a box-guided target query, and converts the two representations into a full-resolution mask or alpha matte.
Feature Extraction and Decoding
Adapts multi-level visual features and query decoding to enable fine-grained dense predictions.
Dense prediction suffers when only a single‑scale feature is used; fine‑grained details are lost. Our solution extracts and adapts features at several depths so that both semantics and spatial precision survive to the decoder.
We pull feature maps from several intermediate ViT layers, then project each map into a common decoder dimension and inject both spatial and level cues.
Apply $W_1\in\mathbb{R}^{2\times4}$ to $F_1$, yielding $X^{(0)}_1$ of shape $2\times2\times2$.
Add $P_{2D}$ (same shape) and $e_1\in\mathbb{R}^2$ broadcast over the spatial grid to obtain $X^{(0)}_1$.
Repeat for level 2 with its own $W_2$ and $e_2$, producing $X^{(0)}_2$.
The two tensors now share the decoder dimension $d=2$ while retaining distinct spatial patterns.
Projecting each level to the same $d$ lets the later adapter treat them uniformly, yet the level embeddings preserve the original depth information.
How does this differ from simply concatenating the raw ViT features?
Concatenation would keep the original channel size $C$, forcing the decoder to handle a much larger tensor and mixing semantics with geometry. The projection to $d$ reduces compute, and the explicit level embedding lets the decoder know which depth each token came from.
The projected tensors are then processed by a stack of $T$ adapter blocks that blend spatial context and channel information before being flattened into a visual memory.
After $T$ blocks, each level’s output $X^{(T)}_l$ is flattened along the spatial axes and concatenated, forming a visual memory matrix $M\in\mathbb{R}^{L H_p W_p\times d}$ that stores a unified representation of all scales.
Instance queries are not static embeddings; they are derived on‑the‑fly from the language model’s hidden state and the predicted bounding box, ensuring the decoder attends to exactly the region it must segment.
Compute $q_1 = W_h h_1 = [0.2, -0.1]$, $q_2 = W_h h_2 = [-0.3, 0.4]$.
Fourier‑encode $b_1$ and $b_2$ (e.g., using sine/cosine of each coordinate) to obtain vectors $\phi(b_1)$ and $\phi(b_2)$.
Apply the MLP (identity) to get $p_1 = \phi(b_1)$, $p_2 = \phi(b_2)$.
The pair $(q_i, p_i)$ now fully describes each instance for the subsequent deformable attention.
Because $q_i$ comes from the language token, the query carries semantic cues (e.g., “cat”), while $p_i$ anchors the attention to the exact region, eliminating the need for a separate learned query bank.
How is this different from the static learned queries used in the original Deformable DETR?
Static queries are fixed vectors learned during training and cannot adapt to the content of a particular image. Here the queries are generated per instance from the VLM’s hidden state and the predicted box, so they are dynamically tailored to each detection.
The decoder stacks six deformable blocks; each block first lets the $N$ instance queries self‑attend, then performs multi‑scale cross‑attention against the visual memory $M$, finally applying a feed‑forward network.
After the queries have gathered multi‑scale visual cues, we fuse features from all levels to produce a high‑resolution mask for each instance.
Upsample $G_2$ to $2\times4\times4$ (e.g., bilinear interpolation).
Apply $\text{Conv}_{1\times1}$ to $X^{(T)}_1$, producing a $2\times4\times4$ tensor.
Add the upsampled $G_2$ to the $1\times1$‑convolved $X^{(T)}_1$.
Pass the sum through $\text{Conv}_{3\times3}$, then GroupNorm and GELU, yielding $G_1$.
$G_1$ now carries both the high‑level semantics from level 2 and the fine spatial layout from level 1, ready for mask head prediction.
The recursive fusion avoids a naïve concatenation that would explode the channel dimension; instead, each step integrates information with a constant‑size tensor.
Why not simply concatenate all level features instead of the top‑down fusion?
Concatenation would increase the channel dimension proportionally to $L$, leading to higher memory and compute costs in the mask head. The top‑down fusion keeps the channel size fixed while still delivering the combined semantic‑spatial signal.
Dense Prediction Mechanisms
Dense prediction adds a shallow CNN and dynamic mask head with multi‑stage refinement.
Fusing high‑level visual features often discards fine‑grained color, texture, and edge cues needed for pixel‑accurate masks, so we re‑introduce a lightweight CNN and a dynamic mask head that refines predictions in three progressive stages.
We run a tiny two‑layer CNN on the full‑resolution image to recover low‑level details that the high‑level encoder may have blurred away.
How does this two‑layer CNN differ from the deep visual encoder used earlier?
The deep encoder extracts abstract semantics and discards high‑frequency signals; the shallow CNN is shallow enough to retain raw color and edge information, which the mask head needs for precise pixel‑level output.
Upsample $G_1$: $U = \text{Up}(G_1)$ copies each coarse pixel into a $2\times2$ block, producing a $4\times4$ map.
Compute shallow CNN output: $C = \text{CNN}(I)$ averages each $2\times2$ region, resulting in a $4\times4$ map of averaged colors.
Combine: $P = U + C$ adds the two $4\times4$ maps element‑wise, preserving both semantic layout (from $U$) and color detail (from $C$).
This simple addition fuses high‑level structure with low‑level texture without extra parameters, enabling the mask head to access both kinds of information.
For each detected instance we synthesize a small convolution kernel from its query embedding, making the mask generation instance‑specific.
How does this dynamic kernel differ from a fixed convolution used in traditional mask heads?
A fixed kernel is shared across all instances, forcing the mask to be generic; the dynamic kernel is generated per instance, allowing the mask to specialize to the object's appearance and geometry.
Generate kernel: $k_i = 0.8$, bias $\beta_i = 0.1$.
Compute logits: $M_i = k_i \cdot P + \beta_i = 0.8 \times \begin{bmatrix}0.2 & 0.5\\0.3 & 0.1\end{bmatrix} + 0.1 = \begin{bmatrix}0.26 & 0.50\\0.34 & 0.18\end{bmatrix}$.
Apply sigmoid later to obtain mask probabilities (not shown here).
The mask logits are directly modulated by the instance query, so changing $e_{qi}$ would reshape the entire mask without touching the backbone.
We iteratively upsample and refine the mask so that thin structures and translucent boundaries can be recovered gradually.
Why not simply upsample the initial mask once instead of using three progressive stages?
Thin structures and semi‑transparent boundaries often require high‑frequency corrections that a single upsample cannot provide; each stage can incorporate fresh image context and residual adjustments, yielding sharper edges.
Stage 1 ($j=1$): Upsample $F_i^{(0)}$ to $4\times4$, concatenate with bilinearly interpolated $I$ and $P$ (both $4\times4$) and $C_i^{(1)}$ (the upsampled mask). Apply $1\times1$ conv → $U_i^{(1)}$.
Generate scale/shift: $\gamma_i^{(1)}=0.5$, $\delta_i^{(1)}=0.1$ (from the MLP). Compute $F_i^{(1)} = f_{rc}^{(1)}(1+\tanh(0.5))\odot U_i^{(1)} + 0.1$.
Update mask: $M_i^{(1)} = \text{Interp}_1(M_i^{(0)}) + \text{Conv}_{1\times1}(F_i^{(1)})$, yielding a $4\times4$ mask with finer detail.
Repeat analogous operations for $j=2$ and $j=3$, each time doubling resolution to $8\times8$ and $16\times16$, respectively, and incorporating fresh image context.
Final $M_i^{(3)}$ captures thin edges that were invisible in the original $2\times2$ mask.
Progressive refinement lets the network correct errors locally at higher resolutions, which a single upsample cannot achieve because it lacks the high‑frequency cues introduced at each stage.
Training Pipeline
How the training pipeline balances diverse tasks and produces pixel masks.
Training a unified vision‑language model requires juggling many tasks—detection, referring, counting, and more—while avoiding over‑fitting to any single objective.
The raw mask logits $M_i$ are passed through a sigmoid “dimmer switch” so the model outputs a probability $A_i$ for each pixel belonging to the target.
Compute the sigmoid: $A_i = \frac{1}{1 + e^{-0.8}} \approx 0.69$.
During training the binary‑cross‑entropy loss compares $A_i$ (≈ 0.69) to the ground‑truth label (0 or 1).
At inference, applying a threshold of 0.5 yields a positive mask prediction for this pixel because $0.69 > 0.5$.
The sigmoid turns any raw score into a well‑behaved probability, ensuring stable gradients and a simple thresholding rule for mask extraction.
The pipeline proceeds through six stages, each emphasizing different tasks and data sources to gradually shape the model’s capabilities.
SFT Stage 1 consumes 90.2 M samples, dominated by detection (67.2 %) and referring (20.1 %).
SFT Stage 2 uses 40.5 M samples, again heavy on detection (66.4 %) but with a higher segmentation share (5.7 %).
Segmentation Warm‑up adds 250 M samples, sourced 62.7 % from the WeVisOne‑VL Data Engine (WDE) and 37.3 % from public data.
SFT Stage 3 (41.8 M samples) shifts focus toward segmentation (52.4 %) while reducing detection to 37.3 %.
Matting Fine‑tuning (130 M samples) is almost entirely WDE‑derived (99.8 %).
The final RL stage (178 K samples) blends detection (44.0 %), dense detection (20.8 %), and referring (17.5 %) among other tasks.
**Figure 4.** Training Pipeline
Curriculum SFT
We train WeVisOne-VL with a three-stage Supervised Fine-Tuning (SFT) curriculum followed by mixed-task Reinforcement Learning (RL). The curriculum first establishes a shared output interface, then improves high‑resolution and dense perception, and finally connects autoregressive localization with pixel‑level prediction. After SFT, RL directly optimizes the quality of the complete prediction rather than the likelihood of individual output tokens. The data composition across the training pipeline is summarized in Table 1.
Section: 4.1 Curriculum Supervised Fine‑Tuning Stage 1: Task Alignment. This stage adapts the model to the new tokens and task‑specific structured outputs. Starting from Qwen3‑VL with an extended vocabulary, we freeze the vision encoder and visual‑language projector and train only the language model on 90 M examples covering all target tasks for 13 K optimization steps. The model learns to follow different task instructions, produce the structured outputs, and return None when the target is absent. For pixel‑task examples, we impose only next‑token prediction supervision (NTP) without activating the pixel decoder, thereby establishing an initial correspondence between each predicted instance and its <OBJ> token. In this stage, we use 16–512 visual tokens per image and limit the context length to 4 096 tokens. Notably, early training can be dominated by extremely dense samples (e.g., containing hundreds of boxes in a single response), making the model highly prone to repetitive output collapse. To mitigate this issue, we primarily use samples with at most 50 boxes or points per category during this stage.
Stage 2: Resolution and Density Scaling. Building on Stage 1, we unfreeze the vision encoder, projector, and language model. This stage serves two purposes. First, we raise the maximum input resolution to 1 960 visual tokens, preserving the visual details for recognizing small objects. Second, we increase the maximum box/point number within one response to 500, exposing the model to crowded scenes for dense detection. The maximum context length is extended to 8 192 tokens. We train on approximately 40 M examples for 9 K optimization steps. During this stage, pixel‑level examples remain supervised solely by NTP.
Stage 3: Alternating Token‑Pixel Optimization. This stage aims to equip the VLM with pixel‑level perception by establishing the <OBJ> token as the bridge between the VLM and the lightweight pixel decoder. To initialize the pixel decoder before alternating token‑pixel training, we first jointly train the SFT‑S2 VLM and the randomly initialized decoder on 250 M segmentation samples for 100 K steps with the following segmentation loss function: Lseg = 20Lfocal + Ldice, (9) where the focal term emphasizes hard pixels under foreground‑background imbalance, while the Dice term directly optimizes mask overlap. After warm‑up, we retain only the decoder weights and the learned <OBJ> token embedding. We attach them to the original Stage 2 checkpoint and jointly train the VLM and decoder for 9 000 steps. To preserve localization ability, we alternate NTP and pixel‑level tasks at a 1:1 ratio, which can be expressed as the following loss function: Ljoint = LNTP + 10Lseg.
(10) After the alternating training, we specialize the pixel decoder for continuous alpha prediction while keeping the VLM frozen. We use 130 M matting samples to further finetune the decoder for 100 K steps. Following standard practice in image matting [29], we combine L1, gradient, and Laplacian losses to supervise alpha estimation, and additionally employ Lhard to emphasize difficult foreground and background pixels: Lmat = 200Ll1 + 5Lgrad + 50Llap + Lhard. (11)
Section: 4.2 Mixed‑Task Reinforcement Learning Although SFT provides a strong initialization, its token‑level likelihood objective cannot directly optimize task rewards. We therefore perform mixed‑task reinforcement learning on an uncertainty‑filtered corpus.
Uncertainty‑Aware Sample Selection. GRPO requires relative advantages to provide learning signals. Before RL training, we therefore use the SFT model for data sampling. For each candidate prompt x, we sample a group of responses and compute their corresponding task rewards. Let µx and $\sigma$x denote the mean and standard deviation of the group rewards, respectively. The uncertainty score for each sample is defined as follows: U(x) = 4µx(1 − µx)$\sigma$x. (12) This score favors moderately difficult samples with high reward variance, on which the model succeeds in some rollouts but fails in others. Such samples have high uncertainty and yield stronger relative‑advantage signals, thereby improving the efficiency of RL training. Using this score, we select approximately 140 K samples for RL training. We supplement this mixture with targeted box‑supervised examples and pointing.
Object Detection Results
Key detection gains and qualitative comparisons for unified vision‑language models.
Qwen3‑VL‑4B improves LVIS Mean F1 by +17.2 over Qwen3‑VL‑8B.
LVIS Mean F1 31.0 vs 13.8 as shown in Table 2.
**Table 2.** Results on general object detection benchmarks COCO and LVIS, and dense object detection benchmarks Dense200 and VisDrone. COCO and LVIS are evaluated in a zero-shot setting. We report F1 at IoU=0.5, IoU=0.95, and averaged over IoU thresholds from 0.5 to 0.95 (Mean). **Bold** and <u>underline</u> denote the best and second-best results among VLMs.
**Figure 5.** Qualitative comparison on general and dense object detection.
Referring Detection Results
Referring detection results show WeVisOne‑VL surpasses prior models on overall mAP.