Video Generation Models are General-Purpose Vision Learners

Letian Wang, Chuhan Zhang, Rishabh Kabra, Jasper Uijlings, Steven Waslander, Andrew Zisserman, Joao Carreira, Kaiming He, Misha Andriluka, Eduard Gabriel Bazavan, Andrei Zanfir, Cristian Sminchisescu

GenCeption repurposes text-to-video diffusion models into unified, feed-forward perception engines.

Can large-scale video generation models serve as universal visual representation learners for downstream perception tasks?

Computer vision remains fragmented into specialized models, each requiring custom architectures and loss functions for tasks like depth estimation or segmentation. This lack of a unified foundation prevents the field from achieving the generalist capabilities seen in large language models. GenCeption treats a pre-trained video diffusion backbone as a universal feature extractor, reformulating perception as a sequence-to-sequence task steered by text prompts. By fixing the diffusion timestep to zero and negating the velocity output, the model performs inference in a single feed-forward pass rather than slow, iterative sampling. This approach achieves state-of-the-art performance across diverse benchmarks, often matching or exceeding specialized models while using up to 500× less training data.

Paper Primer

The core move is to treat the video diffusion model as a frozen, high-fidelity feature extractor. By mapping all dense perception tasks into a shared 3-channel RGB space and using learnable tokens for sparse tasks, the authors unify disparate vision problems under a single $L_2$ loss function.

Video generative pre-training provides superior visual priors compared to traditional self-supervised methods.

GenCeption outperforms V-JEPA and VideoMAE V2 on depth estimation benchmarks under comparable training settings. The model achieves state-of-the-art results on multiple tasks, including depth, surface normal, and camera pose estimation, while requiring 7× to 500× less training data than leading specialized models like D4RT and VGGT-$\Omega$.

The model exhibits emergent generalization: despite being trained exclusively on synthetic human-centric videos, it performs robustly on real-world footage, out-of-distribution object categories like animals, and scenes containing multiple objects.

Why use a video generation model for perception instead of a standard vision-only foundation model?

Video generation forces the model to internalize 4D temporal causality, 3D geometry, and physical interactions, while native text-conditioning provides inherent vision-language alignment that vision-only models lack.

Does this unified approach sacrifice performance on specific tasks?

While joint training can degrade performance on sparse tasks like 3D keypoint prediction due to architectural divergence, the model generally matches or surpasses specialized state-of-the-art models on dense perception tasks.

The Case for Generalist Vision

We argue that large‑scale video generation is the catalyst for a unified vision foundation model.

Natural‑language processing progressed from a zoo of task‑specific models to a single, large‑scale next‑token predictor that now powers thousands of language tasks. In contrast, computer vision still depends on separate architectures for depth, segmentation, pose, and other perception problems. This mismatch reveals a gap: vision lacks a universal pre‑training objective that can serve as a foundation for all visual tasks.

Video generation forces a model to learn the world’s 4‑D dynamics, align visual features with language, and scale to massive data—exactly the ingredients needed for a generalist visual intelligence.

GenCeption instantiates this paradigm by repurposing a pre‑trained video diffusion backbone as a feed‑forward perception engine, then fine‑tuning it on a wide spectrum of vision tasks using primarily synthetic data. The resulting single architecture can perform depth estimation, surface‑normal prediction, camera‑pose recovery, segmentation, and 3‑D keypoint detection—all steered by a textual instruction.

The essential insight is that shifting from task‑specific pipelines to a unified, video‑generation‑pre‑trained backbone paves the way for generalist vision intelligence.

Perception Foundation Models

We situate GenCeption among prior vision foundations, representation learning, and diffusion repurposing.

Perception foundation models such as the Segment Anything and Depth Anything series have shown strong image‑level performance by training on massive datasets, yet they lack temporal reasoning and rely on rigid, task‑specific encoder/decoder heads.

DepthAnything3 predicts dense per‑pixel depth from a single RGB frame and generalizes across diverse scenes without task‑specific finetuning.

SAM3 extends the Segment Anything paradigm to video, producing consistent object masks across frames given arbitrary prompts.

Self‑supervised representation learning has progressed from masked autoencoders (MAE) for images to VideoMAE and RVM for video, while feature‑distillation methods such as the DINO series and contrastive vision‑language models like CLIP and SigLip provide multimodal grounding.

Video MAE v2 masks spatio‑temporal patches in a video clip and learns to reconstruct them, yielding a representation that captures motion and appearance jointly.

V‑JEPA adapts the Joint Embedding Predictive Architecture to video by predicting future latent tokens from past ones, thereby learning a predictive visual prior.

Recent work repurposes diffusion models for perception: Marigold fine‑tunes Stable Diffusion into a single‑step depth estimator; GenPercept identifies feature‑injection and decoding tricks for dense tasks; Diception steers perception via text prompts; BufferAnytime adds a temporal layer to image diffusion for video depth; DepthCrafter and NormalCrafter align video diffusion with CLIP/DINO for depth and normal estimation; Geo4D, DiffusionRenderer, and ReferEverything extend video diffusion to 4‑D reconstruction, rendering, and open‑world segmentation; THFM and Vision Banana explore unified video‑to‑text generation; concurrent work [69] demonstrates that large‑scale video generators already encode useful visual priors.

The GenCeption Framework

GenCeption turns a video diffusion model into a single‑step, unified perception engine.

Video diffusion models generate high‑fidelity footage but require dozens of denoising steps, making them too slow for perception tasks that need instant, accurate predictions. The latency and memory overhead of the iterative sampling loop become a hard bottleneck. GenCeption eliminates this bottleneck by converting the generative backbone into a single‑pass predictor.

GenCeption repurposes a pretrained video diffusion model as a universal perception backbone, freezing its learned spatio‑temporal priors and attaching a lightweight feed‑forward head that maps video tokens directly to task outputs.

How is GenCeption different from simply running the diffusion model with fewer denoising steps?

Running fewer steps leaves the latent state noisy and still requires the stochastic denoising schedule; GenCeption discards the denoising altogether and trains a deterministic head on top of the frozen latent features, so the output is produced in a single forward pass without diffusion dynamics.

VAE encoder → video tokens: $(v_{1,1}, v_{1,2}, v_{2,1}, v_{2,2})$.

Frozen DiT forward pass (no diffusion steps) → latent tokens $(\ell_{1,1}, \ell_{1,2}, \ell_{2,1}, \ell_{2,2})$.

Linear depth head $W_{\text{depth}}$ multiplies each $\ell$ to produce depth scalars $(d_{1,1}, d_{1,2}, d_{2,1}, d_{2,2})$.

Reshape the scalars into a $2\times2$ depth map matching the original frame layout.

The example shows that the latent tokens already contain enough geometric information to predict depth directly, so the iterative diffusion refinement is unnecessary for perception.

Video Generative Pre‑training trains a text‑to‑video diffusion model to predict clean video frames from noisy ones conditioned on a textual description, forcing the network to internalize spatio‑temporal dynamics, 3‑D geometry, and visual‑language alignment.

Why does a generative objective produce features useful for perception?

The generative loss forces the model to reconstruct realistic motion and appearance; the gradients therefore shape the encoder to capture cues like depth, surface normals, and object identity—exactly the signals needed for perception tasks.

Spatio‑temporal world priors are the implicit knowledge about how objects move, occlude, and interact over time that the model acquires during video generation pre‑training.

How are these priors accessed when the model is used for a perception task?

The perception head simply reads the frozen latent tokens; because the tokens already embed motion and geometry, a linear or shallow MLP decoder can translate them into task‑specific outputs without additional temporal modeling.

**Figure 1.** **Methodology** (Left): GenCeption treats a video generative diffusion model as a pre-training base to capture rich spatio-temporal world priors and native vision-language alignment at scale. During multi-task post-training, the model is adapted to feed-forward model fine-tuned on predominantly synthetic data to handle diverse perception tasks. GenCeption shows strong performance with intriguing Emerging Behaviors, enabling seamless sim-to-real transfer and generalization to out-of-distribution object categories. **Paradigm Shift** (Right): This highlights a paradigm shift from specialized task-specific computer vision models, to fully unified generalist vision models.

**Figure 4 |** Architecture overview of GenCeption, a simple yet powerful architecture adapted from text-to-video diffusion models. Given an input video and a text prompt specifying the desired output, our unified model, trained majorly on synthetic data, is capable of performing a wide range of dense and sparse perception tasks, with a single forward-pass of the model. The dense vision tasks are unified in the RGB ambient space where supervision can be applied in latent space efficiently, and the sparse vision tasks are realized by adding learnable tokens as additional inputs to the diffusion transformer (DiT).

Adapting Generative Models for Perception

We turn the diffusion backbone into a single‑pass feature extractor and fine‑tune it on multiple perception tasks.

Diffusion video models excel at generation but require dozens of denoising steps, making them impractical as perception backbones.

We freeze the diffusion transformer, feed it a clean latent video, and run it once—essentially repurposing the generative model as a high‑capacity feature extractor.

We feed $x_0$ directly to DiT (no Gaussian noise added).

The timestep embedding is set to $t=0$, so the model receives a zero‑vector for the time token.

DiT outputs a velocity matrix $v$ (e.g., all entries ≈ 0.12).

Negating yields $-v$, which we add to $x_0$ to obtain $\hat{x}=x_0 - v$ (e.g., each entry reduced by 0.12).

The resulting $\hat{x}$ is fed to the VAE decoder, producing a feature map that the downstream task heads consume.

Negating the velocity turns a “what‑to‑remove” prediction into a “what‑to‑keep” estimate, aligning the model’s output with the clean latent without any extra training.

Why not simply train the transformer to predict $x_0$ directly instead of predicting velocity and negating it?

During pre‑training the model learned to predict velocity because that objective stabilizes training across many diffusion steps. Re‑using the same head preserves the pretrained weights; retraining a new $x_0$ head would discard that knowledge and require costly fine‑tuning.

With a fast, feed‑forward backbone in place, we can attach multiple perception heads and train them jointly.

All dense vision outputs (depth, segmentation, normals, etc.) are rendered as three‑channel RGB images, letting a single decoder handle every task.

Compute the median depth $=4.0$ m and divide each value by the median → $[0.5, 1.25, 0.875, 1.0]$.

Apply the logarithmic mapping $d'=\text{clip}(\alpha\log(d+1),0,1)$ with $\alpha=0.6$ → $[0.31, 0.58, 0.45, 0.51]$.

Replicate the scalar $d'$ across R, G, B channels, producing a $2\times2\times3$ RGB tensor.

Even though depth is inherently 1‑D, the three‑channel replication lets the same decoder treat it exactly like a color image.

Why not keep depth in its native single‑channel format and add a tiny depth‑specific head?

Introducing a separate head would break the “single‑decoder” design, requiring extra parameters and losing the benefit of sharing pretrained visual priors across tasks.

We attach a small set of learnable tokens—one per video frame—to the latent sequence, letting the model output structured sparse predictions (e.g., 2D/3D keypoints) without extra attention layers.

Three learnable tokens $t_1,t_2,t_3$ are concatenated to the $T'=2$ latent tokens (after temporal down‑sampling).

DiT processes the combined sequence, producing three token embeddings $e_1,e_2,e_3$.

An MLP maps each $e_i$ to a 2‑D coordinate, e.g., $e_1arrow(0.12,0.78)$, $e_2arrow(0.45,0.33)$, $e_3arrow(0.89,0.11)$.

These coordinates are interpreted as keypoint locations in the original video frame resolution.

The learnable tokens act like “query anchors” that the backbone can attend to, enabling precise sparse predictions without extra attention passes.

How does this token‑based approach differ from adding a separate attention head for sparse outputs?

Instead of allocating a full attention sub‑layer, the tokens are processed alongside the existing latent sequence, reusing the same attention matrix and thus incurring almost no extra cost.

**Figure 5.** The "Rothko" Raymap as an example of adapting high-dimensional modal data into standard 3 RGB channels. This representation effectively compresses the camera's multi-channel ray data by assembling rotation and translation components into a single three-channel map.

Training data are generated synthetically at scale to cover all modalities required for the unified tasks.

We adopt a single $L_2$ loss applied in latent space for dense outputs and in output space for sparse outputs, eliminating the need for task‑specific loss engineering.

Before feeding depth maps into the RGB‑packed decoder we normalize each frame by its median depth and apply a logarithmic compression, turning raw meters into a stable $[0,1]$ signal.

Empirical Performance and Versatility

GenCeption delivers universal perception performance, rivaling task‑specific SOTA.

Large‑scale video generation models double as universal visual learners; after multi‑task finetuning they excel on perception benchmarks.

GenCeption Generalist‑L attains $\mathbf{97.1\%}$ $\delta_1$ on depth estimation while achieving $0.048$ AbsRel, surpassing all specialized baselines.

Table 2 reports $\delta_1=97.1\%$ and AbsRel $=0.048$ for the WAN 2.1 L variant, beating the next best specialist (VGGT‑$\Omega$, $\delta_1=70.2\%$).

**Figure 2.** **SOTA Generalist Capability (Left):** GenCeption achieves universally competitive performance on a wide range of vision tasks, matching or outperforming state-of-the-art models dedicated to individual tasks (e.g. DepthAnything3 [41], SAM3 [10], D4RT [82], VGGT-$\Omega$ [65], Sapiens [33], David [56], Genmo [38], Lotus-2 [22]). Our specialist denotes a model trained on each task individually, whereas the generalist represents a single model trained jointly across multiple tasks. **Data Efficiency in Finetuning (Right):** Validated on depth estimation, the video generative pretrained backbone (i) outperforms the largest available variants alternative pretraining paradigms (e.g., V-JEPA, and VideoMAE V2) under the same finetuning data. (ii) exhibits preliminary scaling properties, where the performance improves with more data and large model size; (iii) shows exceptional data efficiency, achieving comparable performance with leading models like D4RT [82] and VGGT-$\Omega$ [65] with 7× to 500× less training data.

**Figure 3.** Capabilities Illustrations. GenCeption is a versatile video perception model with SOTA performance on a multitude of output modalities including surface normal estimation, depth estimation, foreground segmentation, expression-referring open-world segmentation, dense human pose estimation, 2D/3D keypoints estimation, and camera pose estimation. Our approach has been trained on human-centric synthetic videos, yet it generalizes to real videos both for people as well as other categories such as animals and anthropomorphic characters.

**Figure 6.** Demonstration of GenCeption's depth and surface normal estimation capabilities.

**Figure 7 |** Qualitative results of our approach on referring expression segmentation. Our model accurately recognizes objects, colors, spatial relationships, and motion. Furthermore, it demonstrates strong generalization by identifying unseen objects, effectively leveraging the knowledge acquired during text-to-video pretraining.

**Figure 8.** **3D Pose Estimation.** Example 3D pose estimation results on challenging skiing (top) and snowboarding (bottom) video. We show every 10-th frame for a subsequence of the input video. Compared to the existing state-of-the-art 3D human pose estimation methods GenCeption takes a full video frame as input and does not require pre-processing steps such as person detection or 2d keypoint estimation.

GenCeption matches or exceeds specialized SOTA models across diverse modalities.

Benchmarks and Evaluation

Defines the benchmark datasets and metrics used to evaluate GenCeption on multiple perception tasks.

We evaluate GenCeption on a suite of video and image benchmarks covering surface normals, depth, soft foreground segmentation, camera pose, 3D keypoints, and expression‑referring segmentation.

Surface normal estimation uses Hi4D (sequences 28, 32, 37 captured by camera 4, totaling 1,195 frames) and the SINTEL dataset. Depth estimation is tested on KITTI, SINTEL, ETH3D, and Goliath, where Goliath follows DAVID’s split of 12 cameras, 16 frames, 4 subjects, amounting to roughly 2.2 k frames across face, upper‑body, and full‑body categories.

Soft foreground segmentation results are reported on VideoMatte (using the static composite split from [43]), PhotoMatte 85, and PPM‑100. Expression‑referring open‑world segmentation follows VoCap’s protocol on Ref VOS‑DAVIS and MeViS.

Camera pose estimation is evaluated on SINTEL, while 3D keypoint estimation uses the EMDB dataset.

For surface normals we report mean and median angular error and the percentage of pixels within thresholds $t \in \{11.25, 22.5, 30\}$ degrees. Depth performance is measured by absolute relative error (AbsRel) and root‑mean‑square error (RMSE). Soft foreground segmentation uses mean squared error (MSE). Expression‑referring segmentation is assessed with the J&F score (average of IoU and contour accuracy). 3D keypoint accuracy is reported as mean per‑joint position error (MPJPE), and camera pose uses average translation error (ATE) together with relative pose error for translation (RPE‑T) and rotation (RPE‑R).

Ablations and Generalization

We dissect how each component influences performance and reveal surprising generalization traits.

The central premise—that large‑scale video generation furnishes a universal visual representation—still guides this analysis. Here we ask whether each design choice truly matters.

Joint training versus task‑specific finetuning yields mixed outcomes: dense tasks such as depth and camera pose remain stable, foreground segmentation improves, but surface‑normal estimation diverges and 3D human keypoint estimation collapses.

We attribute the collapse to two architectural factors. First, the token‑based coordinate regression departs from the continuous pixel‑space learned during diffusion pre‑training; second, the extra learnable tokens introduced for each task disturb the native attention patterns of the pretrained DiT layers.

Consequently, the safest post‑training strategy is to alter the backbone as little as possible, or to redesign the pre‑training objective so that it natively accommodates highly diverse downstream heads.

**Table 2.** Validated on depth estimation, the generative backbone outperform other visual pretraining (V-JEPA and Video MAE v2), and show preliminary data and model scaling properties, where performance improves with data volume and model size Note that our method was trained exclusively on synthetic data.

**Figure 9.** Effect of transferring increasing number of layers from the pre-trained text-to-video model.

**Figure 10.** **Generalization capabilities.** Left: Trained on synthetic data with one single object within each video, our method generalizes in zero-shot to real videos with multiple objects. Right: Our approach has been trained only on videos of a single class (humans), but generalizes to a variety of other classes of articulated objects.

Beyond raw performance, the model exhibits striking emergent behaviors: it reproduces fine‑grained details absent from the Blender‑generated training set (e.g., cat whiskers) and generalizes zero‑shot to multi‑object and cross‑class scenarios.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers