Vision as Unified Multimodal Generation

Xiaoyang Han, Jianhua Li, Kewang Deng, Zukai Chen, Xuanke Shi, Sihan Wang, Boxuan Li, Linyan Wang, Siyi Xie, Xin You, Jinsheng Quan, Zhongang Cai, Haiwen Diao, Ziwei Liu, Lei Yang, Dahua Lin, Quan Wang

SenseNova-Vision casts heterogeneous computer vision tasks as unified text and image generation.

Can we unify diverse computer vision tasks (detection, segmentation, geometry) into a single multimodal generation framework without task-specific heads?

Computer vision currently relies on a fragmented ecosystem of task-specific architectures, where detection, segmentation, and 3D geometry each require custom heads, losses, and decoding rules that cannot easily share supervision. SenseNova-Vision replaces these specialized modules by formulating all vision tasks as native multimodal generation: natural language instructions specify the task, and the model generates either text for symbolic records or images for dense spatial targets. This unified approach matches leading task-specialized systems across structured understanding, dense prediction, and multi-view geometry while enabling language-defined task variants not explicitly present in the training data.

Paper Primer

The core mechanism hinges on a unified data protocol that converts heterogeneous annotations into a shared instruction-response format. Symbolic outputs like bounding boxes or camera parameters are serialized as text tokens, while dense spatial outputs like depth maps or segmentation masks are rendered as images in the model's native latent space.

Unified multimodal generation achieves performance parity with task-specialized systems.

The model matches or approaches leading baselines across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. The model leads on structured visual understanding benchmarks and remains competitive with specialized models in dense prediction and 3D reconstruction.

By training on a mixture of these converted tasks, the model learns to interpolate between domains. For example, it can perform "referring-style interactive segmentation" by taking a text-encoded coordinate as a prompt to generate a mask, effectively recombining localization knowledge from detection tasks with mask-generation capabilities.

Why is this approach more scalable than previous unified vision models?

Earlier models often relied on shared sequence formats that struggled with dense spatial outputs, or they attached task-specific decoders to MLLMs. By using the model's native image generation space for dense maps and text generation for symbolic records, this method eliminates the need for custom architectural heads.

Does this unified training degrade the model's general-purpose capabilities?

No; evaluations on multimodal understanding and text-to-image generation benchmarks show that the model preserves its core capabilities while gaining the ability to perform a wide range of perception tasks.

Vision as Unified Multimodal Generation

We frame computer vision as unified multimodal generation, removing task‑specific heads.

Computer vision today relies on a patchwork of specialist architectures, each with its own output head and training pipeline. This fragmentation hampers supervision sharing, composability, and scaling to new tasks.

Each vision task is equipped with a dedicated output layer that maps shared backbone features to a task‑specific format such as bounding boxes, masks, or depth maps.

All vision tasks are expressed as generation of either text or images using a single unified multimodal model (UMM).

Building on this formulation, we train SenseNova‑Vision, a UMM that learns from the SenseNova‑Vision Corpus—a large instruction‑response dataset covering text, image, and mixed targets. The model matches leading task‑specific systems across detection, OCR, segmentation, depth, and 3‑D geometry without any task‑specific heads.

**Figure 1.** Benchmark overview across representative computer vision task families. Despite using a single unified multimodal generation interface and no task-specific heads, SenseNova-Vision achieves competitive performance across heterogeneous output formats, including text-serialized records, dense image outputs, mixed text-mask responses, and multi-view geometric predictions.

**Figure 2.** SenseNova-Vision integrates diverse computer vision tasks into a single UMM, producing outputs for structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry through unified multimodal generation.

Shifting from task‑specific heads to a generative interface unifies vision tasks under a single model.

Evolution of Unified Vision Models

Survey of prior approaches to unifying vision tasks and their limitations.

Early unified vision models cast diverse visual predictions as token sequences, starting with Pix2Seq for object detection and extending through Pix2Seq v2 and UniTAB to broader task families.

Subsequent systems such as OFA, Uni‑Perceiver, Unified‑IO (and its successor) and Florence‑2 broadened this paradigm by representing heterogeneous inputs and outputs in a shared token space, yet they still rely on task‑specific encodings that make dense maps awkward.

Visual foundation models (MAE, DINOv2) provide reusable features for recognition and dense prediction, while promptable segmentation (SAM) and in‑context image editing (Painter, SegGPT) extend this to a wide range of mask and geometry tasks, but they lack a unified output space.

Diffusion‑based image‑generative models (Marigold, Lotus, FE2E, DICEPTION, Visual Bridge, Vision Banana) reinterpret dense perception as conditional image generation, aligning spatial targets with the native image space but leaving symbolic and mixed outputs unsupported.

Multimodal large language models (Kosmos‑2, Shikra, Ferret, Rex‑Omni, Text4Seg, DenseMLLM, LISA, PixelLM, X‑SAM, G2VLM, VisionLLM v2) attach task‑specific modules to a language backbone, preserving useful inductive biases but keeping output spaces fragmented.

Recent Unified Multimodal Models (UMMs) demonstrate that a single model can natively generate both text and images, motivating the present work to map heterogeneous vision annotations into the shared text‑and‑image generation space of a UMM.

Constructing the Unified Vision Corpus

We unify diverse vision annotations by converting them into instruction‑response samples and assembling the SN‑VC corpus.

Heterogeneous vision annotations—boxes, masks, depth maps, camera poses—prevent a single model from learning across tasks, because each format requires a different output head.

The protocol works like a postal service: the instruction is the address label that tells the model what to deliver, and the response is the package that arrives at the correct mailbox.

How does this differ from ordinary supervised learning where inputs map directly to labels?

Standard supervised learning couples each input with a fixed label type (e.g., a class ID). The protocol adds an explicit natural‑language instruction that tells the model *what* to produce and *how* to format it, turning every task into a unified generation problem.

Instruction is generated: “Detect all objects and output their class and bounding box in

Response text is assembled: “

The model receives (I, instruction) and is trained to autoregressively emit the response string.

At inference, the same instruction yields a text string that can be parsed back into a detection record.

This example shows that a single textual schema can represent both classification and localization, eliminating the need for separate detection heads.

SN‑VC is a massive library built by applying the instruction‑response protocol to public vision datasets, turning every annotation into a unified training example.

How are supervision gaps filled when public annotations are missing?

For structured tasks we use the Rex‑Omni pipeline to synthesize detections and OCR text; for dense geometry we run MoGe‑2 to densify depth/normal maps; for multi‑view geometry we employ LingBot‑Depth to generate missing depth and filter out inconsistent camera metadata.

Gather public image datasets and categorize them into the four protocol families.

Apply the instruction‑response conversion to every annotation that already matches a target schema.

Identify gaps (missing depth, incomplete OCR, absent camera poses) and invoke the appropriate generation/curation module (Rex‑Omni, MoGe‑2, LingBot‑Depth, etc.).

Validate generated targets with consistency checks (e.g., coordinate ranges, mask integrity).

Assemble all converted and generated samples into the unified SN‑VC collection.

Sample from the full pool to obtain the released SN‑VC‑50M subset, preserving the family‑wise proportions shown in Figure 5b.

**Figure 5.** SN-VC source composition and training mixture. Left: configured source examples by converted family and subtype; right: realized sample-type proportions during training.

**Figure 4.** Representative SN-VC examples under the data protocol. Source annotations from structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry are converted into instruction-conditioned text, image, or mixed text-image targets.

**Table 7.** Datasets covering referring detection, OCR localization, keypoint detection, document layout analysis and GUI understanding. “RGDE” and “OCRE” refer to outputs from our referring grounding and OCR data engines.

**Table 18.** Instruction templates for multi-view visual geometry.

Training the Unified Model

Fine‑tune the pretrained multimodal model on heterogeneous vision data without adding task‑specific heads.

Directly training a vision‑capable model from scratch would discard the broad language and generation abilities of the pretrained multimodal foundation. The challenge is to inject benchmark‑level vision skills while preserving those open‑ended capabilities.

Instead of adding separate heads for each vision task, the model receives a single stream where text and image targets are interleaved, letting one shared decoder learn all formats together.

Cross‑entropy loss on the text tokens: $-\log p(\text{cat}) - \log p(\text{on})$.

Rectified‑flow loss on the visual token: $\|z - \hat{z}\|^{2}$ where $\hat{z}$ is the model’s prediction.

Combine with $\lambda=0.5$: $\mathcal{L}= (\text{CE}) + 0.5\,(\text{RF})$.

Back‑propagate through the shared decoder; both text and visual parameters receive gradients.

The interleaved loss forces the decoder to allocate capacity jointly, so improvements on one modality can benefit the other.

How does this differ from classic multi‑task learning that uses separate heads for each vision task?

Classic multi‑task setups keep each task’s output layer isolated, so the model never shares decoder weights across modalities. Here the same decoder predicts both tokenized text and VAE‑decoded images, so the parameters are truly joint rather than merely co‑trained.

Load the Bagel‑7B‑MoT checkpoint and freeze the VAE visual encoder.

Sample a mini‑batch using the weighted joint sampler (vision + auxiliary data).

Tokenize text targets, encode visual targets into VAE latents.

Compute $\mathcal{L}_{\text{CE}}$ for text and $\mathcal{L}_{\text{RF}}$ for visuals, combine with $\lambda$.

Apply AdamW ($\text{lr}=2.5\times10^{-5}$, no weight decay) to update all unfrozen parameters.

Every 500 steps apply EMA ($\alpha=0.995$) to maintain a stable evaluation checkpoint.

Joint sampling and loss computation for a mixed‑task batch.

High‑resolution conditioning and multi‑view geometry are essential for fine‑grained perception, yet they strain memory and require consistent pose handling.

Keep the image encoder (SigLIP2) at near‑full resolution (up to 980 px) while sampling a limited set of views, then align them to a common reference so the model sees a coherent 3‑D scene.

Each view’s pixel coordinates are shifted so that the camera centre aligns with view A’s centre.

Pixels outside the common field of view are replaced by a sky‑box color (e.g., bright blue).

The aligned images are fed to SigLIP2 at 980 px, producing high‑resolution embeddings.

Camera pose for view B is quantized into tokens $t_{1},t_{2},\dots,t_{2001}$ and inserted into the token stream.

The decoder receives the combined token sequence and learns to predict both the visual output and the pose tokens jointly.

Limiting to ten views and using a sky‑box prevents sparse‑view artifacts while still exposing the model to enough viewpoint diversity to learn 3‑D reasoning.

Why introduce a special pose‑token vocabulary instead of encoding pose as plain text numbers?

Plain text would require variable‑length sequences and lose the tight coupling to the model’s discrete token space. Fixed‑size pose tokens fit naturally into the existing transformer vocabulary, letting the decoder emit them with the same softmax machinery used for words.

**Figure 3.** Overview of SenseNova-Vision: heterogeneous computer vision annotations are converted into native text, image, and mixed text-image generation targets for joint training in a single UMM, without task-specific heads.

Empirical Evaluation and Results

SenseNova‑Vision matches specialized systems across vision task families while keeping its multimodal abilities.

The paper’s core claim is that a single unified multimodal model can replace the fragmented, task‑specific vision pipelines by casting every annotation as a generation problem.

SenseNova‑Vision attains performance within ≈ 2 % of the best specialized models across all four vision task families.

Tables 2‑4 show the model ranking first or second on most metrics, and Table 5 confirms competitive scores against leading generalist baselines.

**Table 2.** Quantitative comparison of dense geometric prediction. The upper block reports geometry-specialized models, while the lower block compares generation-based methods. Methods denoted with an asterisk (*) have been re-evaluated to ensure a direct and consistent comparison with our method.

**Table 3.** Quantitative comparison of segmentation. For Gen. Seg., we report PQ for panoptic segmentation (Pan.) and mIoU for semantic segmentation (Sem.). For Ref. Seg., we report cIoU, defined as the ratio of total true positives to total union. For Rea. Seg., we report gIoU; for GCG Seg. and Inter. Seg., we report mIoU. Higher values indicate better performance for all metrics.

**Table 4.** Quantitative comparison of multi-view point map reconstruction and camera pose estimation. We evaluate feed-forward geometric models (top) alongside generalist geometric approaches (bottom). MapAnything is classified within the latter category, as it accepts images with optional geometric inputs and fuses the encoded features with a multi-view transformer.

**Table (b)** Comparison with Vision Banana

**Figure 6** Normalized convergence curves across representative tasks. Each metric is normalized by its final-step value to compare relative convergence trends during training.

**Figure 7.** Qualitative results of SenseNova-Vision across representative computer vision tasks. All examples are generated through the same language-conditioned multimodal generation interface.

**Figure 8** Qualitative examples of referring-style interactive segmentation with text-encoded point cues. SenseNova-Vision interprets normalized coordinates embedded in the referring span and generates binary masks for the indicated targets.

**Figure a.** Crowded scenes with compact objects from Dense200.

**Figure 9.** Dense instance segmentation with adapted detection-task instructions. (b) Aerial scenes with tiny objects from VisDrone.

**Figure 10.** Visual Grounded (VGD) segmentation with text-specified reference cues. The prompt specifies a reference instance with a point or box written as textual coordinates and asks the model to segment other same-class instances. (a) The point cue selects one bottle, and SenseNova-Vision segments matching bottles across different shapes and sizes. (b) With a box cue on one elephant, the model separates nearby same-class instances despite overlap and occlusion.

SenseNova‑Vision delivers performance on par with task‑specific models while preserving the pretrained multimodal foundation.

Corpus Construction Details

Appendix details the SenseNova‑Vision corpus construction, processing pipelines, prompts, and released resources.

A.1 Structured visual understanding aggregates bounding‑box, point, and visual‑prompt detection into a single pipeline, covering eight vision tasks such as OCR and keypoint detection.

The table compares the performance of two methods, "Vision Banana" and "SenseNova-Vision," across various computer vision tasks including Semantic Segmentation (mIoU), Referring Segmentation (cIoU), Reasoning Segmentation (gIoU), Depth estimation ($\delta_1$), and Normal estimation (mean error).

All detection coordinates are expressed as relative image fractions, rounded to three decimals and clipped to the [0.000, 0.999] range, which eliminates resolution‑dependent discrepancies.

Prompt templates follow the InternVL style: a JSON dictionary with an placeholder, a natural‑language instruction, and a target field that may contain text, an image path, or both.

**Table 8.** Representative delimiters used in structured instructions and target responses.

Because the source datasets are public, we release only the generated training examples, preserving the original image files.

A.2 Dense geometric prediction gathers depth and normal data from synthetic and LiDAR sources, supplementing sparse LiDAR with MoGe‑2 pseudo‑labels.

**Table 9.** Instruction templates for structured visual understanding tasks.

Depth maps are clipped to 0.1–80 m; frames with >1 % invalid pixels are dropped, while normal maps discard images with >10 % sky coverage.

Depth supervision uses inverse depth encoded as three‑channel RGB; normals are stored as RGB where each channel encodes X, Y, Z components.

**Table 10.** Summary of datasets used for depth and normal estimation tasks.

Only the MoGe‑2‑processed examples are released, keeping the original raw datasets private.

A.3 Segmentation splits into binary‑mask tasks (referring, reasoning, interactive) and panoptic tasks (generic, GCG).

**Table 11.** Instruction templates for dense geometric prediction.

**Table 12.** Binary-mask segmentation datasets used for training.

Binary masks use (255,255,255) for foreground and (0,0,0) for background; panoptic masks assign each instance a unique RGB color drawn from a 200‑anchor palette.

**Table 13.** Generic and GCG segmentation datasets used for training.

**Table 14.** Representative delimiters used in segmentation tasks.

We release the converted binary‑mask and panoptic examples, allowing reconstruction of the full masks from the provided paths.

A.4 Multi‑view visual geometry compiles reconstruction and camera‑pose datasets, all rendered with dense depth via LingBot‑Depth.

**Table 15.** Instruction templates for segmentation tasks.

Views with > 66 % invalid depth are filtered; remaining depth maps are completed, and depth values beyond dataset‑specific caps are marked invalid.

Camera poses are encoded as quaternions (scaled by 1000) and a direction‑plus‑scale representation, using reserved tokens <‑1000>…<1000> for all scalar fields.

Scene‑level JSON prompts embed image, depth, and trajectory sequences, enabling online generation of multi‑view examples.

All processed multi‑view examples are released, with raw RGB files referenced by path.

A.5 SN‑VC‑50M Release Summary reports 73 dataset‑task entries across 10 task types, totaling 50 M frames of training data.

**Figure 12.** Text segmentation from free-form language prompts. (a) A word-level prompt asks the model to segment the queried word “coke” as a text-shaped region. (b) A letter-level prompt asks the model to segment all visible letters, requiring finer-grained localization and decomposition of text regions. In both cases, the generated masks align with the requested text targets.

Generation Protocol and Formatting

Appendix B details data tags, dataset composition, and extra qualitative examples.

Translation scale values are encoded as `value` where is a plain number; no surrounding whitespace or line breaks are permitted. Model outputs that denote a target frame are wrapped in `…` tags, again with strict no‑extra‑character formatting.

Multi‑view reconstruction asks the model to ingest several RGB images of a scene and emit a dense 3‑D coordinate map for each view, all expressed in the coordinate system of the first camera. The output maps are aligned so that any point can be cross‑referenced across views without additional transformation.

**Table 17.** Reserved tokens for camera pose encoding.

Structured visual understanding draws from a wide range of segmentation datasets—APTv2 (14.7 K frames), BDD100K (68 K), DOTAv2 (1.7 K), DeepFashion (112.6 K), EgoObjects (49.9 K), FAIR1M (16 K), FSC147 (1.8 K/1.1 K), GroceryStore (1.8 K/1.1 K), HumanParts (7 K), ImageNetPart (10.2 K), NuImages (55 K), Objects365 (1 077.1 K/3 589.0 K), PACO‑LVIS (26.9 K), PixMo‑Points (0.1 K), and SA‑1B (3 119.0 K/1 949.4 K).

Point, bounding‑box, and visual‑task coverage includes V3Det‑OVD (31.6 K), VisDrone (60.8 K), OpenImages (6.4 K), and BLIP3‑OCR (4 034.0 K) frames, providing diverse object‑level supervision.

Generic datasets such as COCONut‑XL (1 582.0 K), Cityscapes (5.0 K/3.5 K), Hypersim (0.4 K/0.3 K), EntityV2 (1.2 K), Trashcan (2.5 K/1.6 K), Pidray (66.8 K/59.5 K), ZeroWaste‑f (1.0 K), LVIS (19.3 K), IDD‑1/2 (27.8 K), and IDDAv3 (148.2 K) round out the corpus with varied scene contexts.

Dense geometric prediction draws on depth/normal pairs from Taskonomy (2 000.0 K each), ScanNet++ (813.0 K each), COCO2017 (78.0 K/79.0 K), SA‑1B (4 442.0 K/4 462.0 K), and Objects365 (1 317.0 K each), enabling the model to learn fine‑grained 3‑D structure.

Reconstruction datasets include ScanNetV2 (241.0 K frames), ScanNet++ (813.0 K), DL3DV (3 462.0 K), and WildRGB‑D (8 026.0 K), providing full‑scene point‑cloud supervision for multi‑view alignment.

Table 19 summarizes the SN‑VC‑50M open‑source release, grouping the above sources by task family and reporting the exact frame counts made publicly available.

The “B Additional Qualitative Results” subsection showcases example outputs across text, image, and mixed modalities, illustrating how the same model follows distinct task instructions while remaining parsable under the tag protocols defined earlier.

The “Exploratory Results on Challenging and Language‑Defined Vision Tasks” subsection probes the model on edge cases: composed depth‑normal prediction under varying classifier‑free guidance, hexadecimal‑color segmentation that obeys language‑specified palettes, dense instance segmentation with enumerated color‑coded masks, fine‑grained Visual Grounded (VGD) segmentation via object queries, and dense geometry on ambiguous or transparent surfaces.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers