Beyond Pixels: From Video Priors to 4D Worlds

Zihao Liu, Xiaolong Shen, Zhenglin Zhou, Ruijie Quan, Yi Yang

Latent-to-4D bypasses RGB decoding to generate 4D scenes directly from video model latents.

How can we generate dynamic 3D scenes (4D) directly from video model latents instead of reconstructing RGB video frames first?

Existing 4D generation methods either rely on unstable RGB-to-geometry reconstruction or require expensive, model-specific retraining to adapt to new video generators. Latent-to-4D treats the final denoised latent of a video model as a reusable interface, aligning these latents directly to a 4D decoder without ever generating RGB frames. This single checkpoint transfers across multiple compatible video models, outperforming standard RGB-based cascades in geometric completeness and temporal stability.

Paper Primer

The core move is the Latent-to-4D Alignment and Refinement (L4AR) network, which acts as a bridge between the video model's latent space and the 4D decoder's token grid. It is like a translator: it takes the raw, compressed latent representation from a video model and reshapes it through 3D convolutions and spatiotemporal attention to match the specific geometric structure required for 4D scene prediction.

Latent-to-4D significantly improves geometric fidelity and completeness over traditional generate-then-reconstruct pipelines.

In controlled same-latent comparisons on the I4D-200 benchmark, the method achieved a 5.81-point gain in projection-based DINO-F1 scores compared to matched RGB-based cascades. Across both text- and image-conditioned tasks, human raters consistently preferred the method's outputs for geometric plausibility and temporal stability, with all preference intervals exceeding 50%.

Why is bypassing RGB decoding necessary for 4D generation?

RGB decoding introduces an intermediate representation boundary where generation artifacts—such as holes or fragmented surfaces—propagate into the geometry, leading to unstable 4D outputs.

What defines the "compatibility" required for this method to work across different video models?

Compatible models must share the same Variational Autoencoder (VAE) checkpoint, latent normalization, tensor layout, and compression convention, allowing them to produce latents in a common representation space.

Introduction and Motivation

We expose why decoding video latents to RGB before 4D reconstruction wastes information.

Current 4D pipelines first turn video latents into RGB frames and then reconstruct geometry, which discards the latent’s structural cues and leads to brittle motion. This indirect route creates a fundamental information bottleneck that the paper seeks to eliminate.

Decoding video latents to RGB before geometry reconstruction throws away the latent’s spatial‑temporal structure, making motion inconsistent and geometry coarse.

**Figure 1.** Latent-to-4D enables unified text- and image-conditioned 4D generation. It turns video-model latents directly into dynamic geometry and motion through a shared pathway, without task-specific adaptation.

**Figure 2.** Comparison of video-to-4D interfaces. Previous methods decode video latents to RGB before reconstruction, whereas ours maps them directly to 4D through L4AR.

The shift from RGB‑based reconstruction to direct latent‑space generation preserves structural fidelity and yields more stable motion.

Related Work

Related work surveys video generation, 4D reconstruction, and 4D generation approaches.

Recent video generators excel at producing realistic appearance and motion, while 4D reconstruction pipelines extract geometry from RGB streams. This section positions our latent‑to‑4D approach among these families.

These baselines first decode a video VAE latent into RGB frames, then feed the frames to a conventional RGB encoder that predicts 3D structure.

Connects a video VAE to a static 3D reconstructor, using the VAE’s latent as input to a pretrained geometry model.

Diffusion Transformers conditioned on text, images, or trajectories; used here as the frozen video generator whose latent is repurposed.

Query‑based model that recovers camera poses and dynamic geometry from RGB video streams.

Recovers cameras and geometry across viewpoints and time using a feed‑forward network.

Estimates cameras and dense point maps from unordered image sets, extending the reconstruction pipeline to multi‑view inputs.

Optimises a NeRF per scene guided by text or images, producing dynamic geometry after a separate optimisation phase.

Uses a set of 3D Gaussians to model dynamic scenes, optimized per output.

Generates multiview RGB observations with a diffusion model, then reconstructs geometry from the rendered frames.

Similar to Diffusion4D: synthesises temporal RGB sequences before applying a 4D decoder.

Combines a diffusion generator with a 4D reconstruction module, feeding generated frames into the decoder.

Adapts a video generator to predict dynamic point clouds directly from latent video representations.

Predicts dynamic Gaussians from a video generator’s latent, enabling fast rendering of moving scenes.

Jointly predicts RGB frames and geometry, coupling generation and reconstruction in a single feed‑forward network.

Extends WorldReel with inference‑time camera control, allowing user‑specified viewpoints after generation.

Connects a frozen video model directly to a pretrained 4D decoder in latent space, bypassing RGB reconstruction.

The Latent-to-4D Framework

Directly map video VAE latents to 4D geometry, bypassing RGB reconstruction.

Existing 4D generation pipelines first decode video latents into RGB frames, then reconstruct geometry, which discards structural detail and introduces motion artifacts. The RGB boundary becomes a bottleneck that limits fidelity and slows training. Our method eliminates this step by mapping the VAE latent directly into the 4D token space.

Instead of sending a painted picture (RGB) to a builder, we hand the raw blueprint—the video VAE latent—directly to the 4D reconstruction hierarchy, preserving every spatial and temporal cue.

Encode an observed video $V$ with the frozen VAE encoder $E_v$ to obtain $z_{\text{obs}} = \mu(E_v(V))$.

Pass $z_{\text{obs}}$ (or a generated latent $z_{\text{gen}}$) through the Alignment Module $A_{\phi}$ to produce the initial token grid $Q^{(0)}$.

Refine $Q^{(0)}$ with the hierarchical Spatiotemporal Refinement Module $H_{\psi,\Delta\psi}$, alternating frame‑wise and global self‑attention.

Decode the refined tokens with the 4D Decoder $D_{\omega}$ to obtain per‑frame cameras $\mathbf{C}_t$ and dense point maps $\mathbf{P}_t$.

**Figure 3.** **Latent-to-4D training pipeline.** A frozen video VAE encodes an observed video into a VAE-space latent. L4AR aligns the latent grid through a learned 3D convolution, reuses frozen camera and time tokens, and refines the representation through alternating frame-wise and global attention. The 4D decoder predicts cameras and dynamic world-space geometry.

Alignment reshapes a mismatched wooden block to fit a pre‑drilled socket, then refinement sands the whole piece so the fit is smooth across every grain.

Resampling $R$ doubles each spatial dimension, yielding $z' \in \mathbb{R}^{2\times4\times4\times4}$.

The 3D convolution $S_{\phi}$ with kernel size $3$ and output channels $d=8$ produces $S_{\phi}(z') \in \mathbb{R}^{2\times4\times4\times8}$.

Flattening the spatial axes ($4\times4=16$) gives $Q^{(0)} \in \mathbb{R}^{2\times16\times8}$ (2 frames, 16 tokens per frame, 8‑dim features).

Frame‑wise attention treats each frame as a batch of 16 tokens, updating local geometry.

Global attention reshapes to $(1, 32, 8)$ and mixes information across both frames, producing the refined grid.

The example shows how a tiny $2\times2\times2$ latent expands to a $2\times16$ token grid, and why the 3D convolution must precede attention: it supplies a coherent spatiotemporal feature map for the subsequent self‑attention stages.

How does L4AR differ from simply upsampling the VAE latent and applying a 2‑D convolution?

Upsampling + 2‑D convolution can only mix information within each spatial slice; it cannot aggregate across time. L4AR’s 3‑D convolution $S_{\phi}$ jointly processes the temporal dimension, and the subsequent frame‑wise/global attention hierarchy explicitly exchanges information between frames, which is essential for consistent motion and camera prediction.

Training and Architecture

Training splits supervision into three focused losses and activates modules stage‑wise.

Training must reconcile latent video representations with 4D geometry while preserving the pretrained geometric prior. Naïve end‑to‑end loss quickly destabilizes the frozen VAE and transformer weights.

We split supervision into three orthogonal terms—uncertainty, camera, and geometry—so each tackles a distinct failure mode, much like a chef separates seasoning, cooking, and plating to keep flavors from interfering.

Confidence‑weighted depth error for point 1: 0.9 × (5.0 − 4.8)² = 0.036.

Confidence‑weighted depth error for point 2: 0.7 × (6.0 − 6.2)² = 0.028.

L₍unc₎ = 0.036 + 0.028 = 0.064 (average over points).

Assume a simple camera‑pose error of 0.05 m translation and 2° rotation; L₍cam₎ = 0.05² + (2·$\pi$/180)² ≈ 0.0039.

Geometry loss combines a metric depth error of 0.1 m (0.01) and a surface‑normal deviation of 5° (0.0076), yielding L₍geom₎ ≈ 0.0176.

Total loss L = 0.064 + 0.0039 + 0.0176 ≈ 0.0855.

The confidence weighting prevents a low‑confidence but large error from dominating the loss, while the separate camera term isolates pose errors that would otherwise be absorbed into depth.

How does the uncertainty‑aware term differ from a standard depth regression loss?

Standard depth loss treats every prediction equally; L₍unc₎ multiplies the squared error by a confidence score predicted by the network, so over‑confident wrong predictions are penalised more heavily while uncertain predictions contribute less.

Stage 0: Freeze all trainable modules (Alignment, Refinement, geometry/camera heads) and train only the static VAE and transformer to stabilise the latent prior.

Stage 1: Unfreeze the Alignment Module; train it together with the frozen components until L stops decreasing for 5 k steps.

Stage 2: Activate the lightweight Refinement updates; continue joint optimisation while monitoring L₍geom₎.

Stage 3: Unfreeze geometry and camera heads; fine‑tune the full pipeline on metric 4D annotations.

Stage 4 (optional): Unfreeze the video generator and motion decoder for a final end‑to‑end sweep if additional capacity is needed.

By the end of the schedule the Alignment Module, Refinement updates, and both heads are fully adapted, while the pretrained VAE and transformer remain stable, yielding consistent 4D reconstructions.

Experimental Results

Our method consistently outperforms baselines on user preference and geometry metrics across text and image prompts.

The core idea is to map video latents directly to 4D geometry, avoiding intermediate RGB decoding. This preserves structural fidelity and improves motion consistency.

Users prefer our method over baselines on overall quality, reaching 70.6% in the Image-to-4D benchmark.

Table 2 reports 70.6 [66.2–74.9] % overall quality preference for our approach versus all baselines.

**Figure 4.** **Text-to-4D comparison.** Texts are abbreviated for display. RGB baselines reconstruct the decoded video, whereas Ours consumes its terminal latent directly.

**Figure 5.** **Image-to-4D comparison.** Each row shows the input condition and 4D results. RGB baselines reconstruct the decoded video, whereas Ours consumes its terminal latent directly.

Ablations and Discussion

Key qualitative, human, and ablation findings demonstrate the impact of each L4AR component.

We first present qualitative 4D generation results, then human evaluation, diagnostic analyses, ablation studies, and finally broader applications.

**Table 1.** 4D generation on Text4D-200 and I4D-200. Two-view projection scores ($\times 100$; higher is better); Ours and matched Wan cascades share the same generated latent.

Fifty participants performed pairwise comparisons on 50 sampled cases per benchmark, rating fidelity, geometry, temporal stability, and overall quality. Each case received ten ratings; Table 2 reports average preferences with 95 % bootstrap intervals, showing a clear preference for our method.

**Figure 6.** **Grid-Align-null DiT-residual sensitivity.** Geometry and camera drift on 7-Scenes and NRGBD.

Table 3 reports ablations where individual L4AR components are removed. Every removal degrades performance on ground‑truth benchmarks, with the largest drops observed when the 3D convolution or either attention scope is omitted, confirming their critical role.

**Figure 7.** Additional controls. Motion, appearance, pose, and trajectory inputs.

**Figure 8.** Action-conditioned 4D. Manipulation and navigation latents from a compatible backbone.

Each element of the L4AR module—3D convolution, attention scopes, and alignment refinement—is essential for achieving high‑quality 4D generation.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers