Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

Kang Liao, Yihang Luo, Xiao-Ming Wu, Linyi Jin, Size Wu, Chunyu Lin, Yao Zhao, Fei Wang, Wei Li, Chen Change Loy

Puffin-World unifies 3D perception, generation, and reconstruction by anchoring visual models to absolute physical gravity and geometry.

How can we unify 3D world perception, spatial simulation, and generation into a single multimodal model that natively understands physical world states?

Generative world models typically treat images as 2D appearance, lacking explicit knowledge of camera orientation or scene geometry. This leads to orientation drift and physically inconsistent generation when moving across viewpoints. Puffin-World introduces the Omni-Camera representation, which fuses absolute gravity-anchored orientation with relative ray-based geometry. It propagates this physical state across future frames to ensure stable, gravity-consistent 3D world modeling. The model achieves state-of-the-art camera-to-world understanding across four public benchmarks and enables high-fidelity, physically grounded 3D scene generation.

Paper Primer

The core challenge is that relative camera motion—the standard for multi-view generation—lacks a global physical anchor. Puffin-World solves this by formulating three native world states: physics (gravity and latitude), geometry (depth), and appearance (RGB), jointly modeled within a single multimodal framework.

The method hinges on the Omni-Camera representation: a dense action signal that concatenates an absolute gravity-aligned field with a relative ray map. Physics propagation then anchors this absolute spatial knowledge, allowing the model to maintain a coherent gravity-aligned world frame across long-horizon trajectories.

Puffin-World achieves state-of-the-art camera-to-world understanding.

Outperforms existing learning-based and geometric methods on median error and AUC at 5° across four public benchmarks (MegaDepth, TartanAir, LaMAR, and Stanford2D3D). Consistently superior to prior specialized models like Puffin and GeoCalib.

To scale these capabilities, the authors constructed Puffin-16M, a dataset of 15 million vision-language-camera triplets and 1 million trajectories featuring diverse, challenging camera motions (e.g., 360° pans, look-up/down) that exceed the rotational diversity of existing datasets.

Why is a unified model necessary instead of just combining existing perception and generation modules?

Existing modular approaches suffer from compounding errors and lack a persistent physical reference frame. Puffin-World’s unified paradigm allows appearance and geometry to share a latent space, ensuring that generated views remain geometrically consistent and gravity-aligned.

How does the model handle geometry without requiring a dedicated depth decoder?

It encodes depth as an RGB image using an invertible Hilbert curve mapping, allowing it to be processed by the same frozen VAE as appearance. Asymmetric attention ensures geometry features inform the model without corrupting the appearance stream.

The model’s ability to perform "self-calibrated world exploration"—automatically detecting and correcting gravity misalignment—suggests a path toward more robust embodied AI that understands its physical orientation without external sensors.

Puffin-World demonstrates that grounding generative models in absolute physical states like gravity is essential for consistent 3D world modeling, effectively moving the field beyond 2D appearance-only generation.

Introduction: Unified 3D World Modeling

Puffin-World unifies perception, generation, and reconstruction via native 3D world states.

Current generative world models operate on raw pixel sequences and therefore miss the physical and geometric structure that underlies real scenes.

Instead of representing a scene only as an image, we keep three tightly coupled components: a physics field (gravity & latitude), a depth map that encodes geometry, and the RGB appearance that the camera actually sees.

The Omni‑Camera representation packs a dense action signal: an absolute gravity‑anchored field supplies a global orientation, while a relative ray field encodes view‑specific geometry.

Physics propagation then takes the absolute spatial knowledge inferred from observed frames and carries it forward through the relative control signals of future frames, ensuring that generated views stay physically stable even under complex motions.

**Figure 1.** Puffin-World is a unified multimodal model using native 3D world states (Appearance-Geometry-Physics) for spatial intelligence and physical AI. It unifies camera physics understanding, free-viewpoint spatial simulation, and 3D world generation and reconstruction within a single framework.

**Figure 2.** Illustration of the versatile capabilities of our model. It shows the best performance (AUC > 5°) on camera-to-world understanding for physical world perception; flexible camera-controllable abilities for free-viewpoint spatial simulation; 3D world generation and reconstruction with flexible-DoF actions and native world states. The red box marks the input to the model.

The key insight is a unified multimodal architecture that directly models native 3 D world states, enabling perception, simulation, and generation to share a single, physics‑grounded representation.

Native 3D World States

We introduce a unified camera representation that grounds each pixel in both absolute physics and relative geometry.

Existing generative world models treat scenes as flat RGB sequences, which discards the physical and geometric structure needed for coherent multi‑view synthesis. By embedding both absolute physics and relative geometry into each pixel, our method enables consistent world generation across viewpoints.

Why does modeling physics, geometry, and appearance together improve generation stability?

Because the physics layer provides a global reference (gravity and latitude) that constrains geometry and appearance, preventing implausible configurations such as floating objects or inconsistent depth.

Physics state: assign the same gravity vector $g$ to all four pixels.

Geometry state: store the depth values $Z$ as the geometry layer.

Appearance state: keep the given RGB colors as the appearance layer.

When generating a new view, the physics layer forces the depth map to respect gravity, so pixels with higher depth cannot appear above lower ones.

Joint modeling ensures that depth changes induced by camera motion remain consistent with the fixed gravity direction, eliminating floating artifacts.

Each pixel carries an absolute orientation tied to gravity and a relative ray that tells where the pixel looks in 3D, unifying global physical grounding with continuous spatial modeling.

How does Omni‑Camera differ from standard relative camera representations?

Standard relative representations only store the ray origin and direction, lacking any notion of absolute orientation. Omni‑Camera adds the up‑vector and latitude angle, grounding each pixel to the gravity field and thus providing a global physical reference.

Compute ray direction $d_x = \frac{K^{-1}\tilde{x}}{\|K^{-1}\tilde{x}\|_2} = ( -0.447, -0.671, 0.593 )$.

Camera center $o = -R^\top t = (0,0,0)$, so the relative component $p_x = (o, d_x)$.

Project a nearby 3D point $X = (0.1,0.1,2)$ along $-g$ to obtain $\Pi(X) = (100,150)$. A tiny shift $\epsilon = 0.01$ yields $\Pi(X - \epsilon g) = (100,150.01)$. The up‑vector $u_x$ is approximately $(0,1,0)$.

Latitude angle $\phi_x = \arcsin\!\big((-d_x)^\top g\big) = \arcsin( -0.593 ) \approx -36.3^\circ$.

Concatenate $a_x = (u_x, \phi_x)$ and $p_x$ to obtain $c_x \in \mathbb{R}^9$.

The absolute component ties the pixel’s orientation to gravity, while the relative component tells where the pixel looks, enabling both global physical grounding and local geometric reasoning.

**Figure 3.** The network architecture of our Puffin-World. It mainly comprises 3D world understanding, generation, and reconstruction formulating native 3D world states within one framework. Both Omni-Camera representation and role mask are conditioned on the reference view and the target views. For clarity, we omit the camera-controllable text-to-image generation and the mathematical mapping from the understanding results to the physics world state.

Related Work

We survey prior work on camera pose estimation, multimodal models, and 3D world modeling.

Camera-to-World Understanding has long been tackled by calibrating cameras and estimating poses from images, with early learning methods regressing parameters directly and newer approaches leveraging dense geometric cues.

It is the process of inferring a camera’s intrinsic and extrinsic parameters from visual input so that image pixels can be mapped into a consistent 3‑D coordinate frame.

Unified multimodal models aim to combine visual perception and generation in a single network, typically by tokenizing images and training autoregressive or diffusion decoders.

Most of these systems still assume a simple pinhole camera and ignore explicit geometry or physics, limiting their ability to reason about 3‑D structure across views.

Puffin introduced a language‑modeling view of camera parameters but remains a single‑view, non‑persistent approach, motivating extensions toward full 3‑D world state modeling.

3‑D world models generate future observations conditioned on camera information, yet they often model only appearance, rely on relative motion, and treat generation as a separate task.

These three limitations—appearance‑only representation, lack of an absolute physical frame, and decoupled generation—drive the need for a unified model that jointly captures physics, geometry, and appearance.

Physics Perception and Propagation

Physics Propagation injects camera geometry into diffusion latents for consistent 3D perception and control.

Existing generative world models predict pixels without respecting physical constraints, which leads to implausible camera poses and unstable spatial control.

PP treats the camera pose and intrinsic parameters as a first‑class signal that is added directly to the diffusion latent at multiple depths, so the generator never forgets where it is looking.

Patch embedding $P$ splits the $4\times4$ image into four $2\times2$ patches and flattens each to a vector.

The concatenated pair $[C; m]$ is projected by $F$ into a $d$‑dimensional vector $f_{\text{cam}}$ (here $d=8$ for illustration).

At the first diffusion step the noisy latent $z$ (also $8$‑dimensional per patch) receives the camera bias: $h_0 = P(z) + f_{\text{cam}}$.

During denoising, layers $l\in\{2,5\}$ apply the re‑injection: $h_l \leftarrow h_l + W_l\,f_{\text{cam}}$, where $W_l$ is an identity matrix for simplicity.

After the final step the decoder reconstructs an image whose viewpoint exactly matches the original $f$ and $r$ values.

Adding the same camera vector at multiple depths forces every attention head to see the same geometric cue, preventing drift that would otherwise accumulate in a deep diffusion process.

How does Physics Propagation differ from the “Perspective Field” injection used in the original Puffin model?

Perspective Field packs camera information into three channels and injects it via cross‑attention on a single latent branch. PP instead projects the full Omni‑Camera map (including role mask) into the patch space and adds it additively at several transformer layers, preserving exact geometry for every token and avoiding the bottleneck of a low‑dimensional channel.

With PP in place, the system can perform free‑viewpoint spatial simulation: given a text prompt and a desired Omni‑Camera map $C$, the diffusion generator produces an image that exactly matches the specified viewpoint and intrinsics.

Encode the text prompt into LLM hidden states.

Form the conditioning vector by passing $[C; m]$ through $F$ and pooling the result.

Combine the conditioning vector with the noisy diffusion latent $z$ using $h_0 = P(z) + F([C; m])$.

Run the MMDiT denoiser, re‑injecting $F([C; m])$ at layers $L$ as described above.

Decode the final latent with the VAE decoder to obtain the rendered image.

Trajectory Anchoring

Anchors multi‑view trajectories to a gravity‑aligned world frame via physics propagation.

The lack of an absolute world orientation makes multi‑view generation drift when cameras follow arbitrary trajectories. We resolve this by propagating a gravity‑aligned direction from a single reference view to every future view.

We fix the world’s “north” by propagating the gravity direction from a reference view to every future view—like a compass needle that all maps align to—so the whole trajectory stays glued to a single, physically consistent frame.

Is Trajectory Anchoring simply re‑using the known camera poses to align views?

No. Camera poses give only relative motion; Trajectory Anchoring injects an absolute gravity direction derived from the reference view, establishing a shared “up” that camera poses alone cannot provide.

Apply $R_{rel_{1 \leftarrow 0}}$ to $g_0$: $g_1 = R_{rel_{1 \leftarrow 0}} g_0 = (0,\,0,\,-1)$.

Apply $R_{rel_{2 \leftarrow 0}}$ to $g_0$: $g_2 = R_{rel_{2 \leftarrow 0}} g_0 = (0,\,1,\,0)$.

Render each view’s absolute camera representation from $g_1$ and $g_2$, then concatenate with the relative ray map to obtain the Omni‑Camera conditions for views 1 and 2.

Even though the cameras rotate arbitrarily, the propagated gravity vectors keep all views anchored to the same world “down” direction, preventing drift.

Extract roll $ϕ_0$, pitch $θ_0$, and vertical FoV $α_0$ from the reference view via the Physics Perception pathway.

Compute the reference gravity direction $g_0$ from $(ϕ_0,θ_0,α_0)$.

For each target view $t$, retrieve the relative rotation $R_{rel_{t \leftarrow 0}}$ from the known camera poses.

Propagate the gravity direction: $g_t = R_{rel_{t \leftarrow 0}}\,g_0$.

Render the absolute camera representation for view $t$ using $g_t$ (Eq. 4).

Concatenate this representation with the relative ray map to form the full Omni‑Camera condition for view $t$.

Joint Appearance-Geometry Modeling

Jointly predicts appearance and geometry by sharing a latent space.

Separate appearance‑only generators cannot guarantee that the rendered views share a coherent 3‑D structure, leading to jittery depth and inconsistent geometry.

Instead of training two separate networks for color and depth, the model packs both signals into the same latent token stream, letting a single denoising pass produce a coherent image and its depth map.

How does this joint modeling differ from simply concatenating a depth channel to the RGB image?

Concatenation would force the VAE to learn a new input distribution and would mix depth gradients into the appearance pathway from the start. By encoding depth as an invertible RGB map and appending a separate geometry token block, the model keeps the appearance stream untouched while still sharing the latent space, and the geometry weight ramps up only after the appearance backbone has stabilized.

Depth 0 maps to RGB (0,0,0); depth 3 maps to RGB (0,1,1); depth 5 maps to RGB (1,0,1); depth 7 maps to RGB (1,1,1).

The four RGB triples are stacked as a 4 × 3 image and fed to the frozen VAE encoder, producing four appearance latents and four geometry latents.

During denoising, the geometry token block receives the same noisy diffusion step $t$ as its RGB counterpart but is masked as “geometry” for the asymmetric attention layer.

After the final denoising step, the geometry latents are decoded by the VAE decoder, yielding the RGB‑encoded depth image, which is then inverse‑mapped back to scalar depths [0, 3, 5, 7].

The invertible mapping lets the model reuse the exact same VAE weights for both appearance and depth, avoiding a separate depth encoder while preserving precise geometric detail for near surfaces.

Asymmetric attention ensures that geometry tokens can gather context from appearance tokens, but appearance tokens never see geometry, preventing early‑stage depth noise from corrupting visual quality.

A zero‑initialized geometry‑modality embedding is added after patch embedding, so the model behaves exactly like the original multi‑view generator at initialization and gradually learns geometry‑specific features.

**Table 1.** Training recipe of Puffin-World. We report the per-stage hyperparameters, the trainable/frozen status of each module, and the data sampling ratio across tasks. *Und.*, *Gen.*, and *Recon.* abbreviate understanding, generation, and reconstruction, respectively.

Long-Horizon Exploration

Linking latent chunks avoids drift and preserves physics over arbitrarily long trajectories.

Long‑horizon generation suffers from cumulative VAE encode‑decode errors: each chunk’s output is re‑encoded as pixels, decoded again, and the small reconstruction bias compounds, eventually warping geometry and breaking physics consistency.

Instead of re‑encoding the last generated view into pixels, the method hands the denoised latent of that view directly to the next chunk, so the latent stream flows uninterrupted like a baton passed between runners.

Chunk 1: latent sequence = [L₁, L₂, L₃].

Slide the window: discard L₁, keep [L₂, L₃].

Condition Chunk 2 on L₃ (the retained latent) and generate new latents L₄, L₅, L₆.

Slide again: keep [L₄, L₅, L₆] for Chunk 3, and so on.

Because each chunk conditions on the denoised latent from the previous chunk, no pixel reconstruction occurs at the boundary, so the error that would have grown with each re‑encode is eliminated.

How does this differ from a recurrent VAE that feeds the previous pixel output back into the encoder?

In a recurrent VAE the previous output is first rendered as pixels, then passed through the encoder, which introduces a reconstruction bias at every step. Our sliding window bypasses the pixel stage entirely; the latent vector is handed off directly, so the only transformation is the learned decoder, not a second encoder‑decoder cycle.

Training proceeds in four stages: Stage I uses a 512‑dimensional latent with cosine schedule (0.05, 0.9‑0.95) and freezes all visual encoders; Stage II keeps the same latent size but unfreezes them; Stages III and IV progressively shrink the latent to 256 then 128 while keeping the visual encoders frozen, allowing the model to consolidate long‑range dynamics in a compact representation.

Data sampling ratios are heavily skewed toward image‑to‑text‑camera generation (1 × 10⁻⁴) and cross‑view image‑to‑3D generation (5 × 10⁻⁵), reflecting the need for abundant visual supervision while keeping cross‑modal samples sparse enough to avoid over‑fitting.

Training Recipe

A four‑stage curriculum progressively aligns, fine‑tunes, and expands the model while freezing components to preserve learned knowledge.

Training a unified multimodal model that handles vision, language, and camera inputs risks overwriting the rich representations learned in each modality. The authors therefore design a curriculum that first stitches the modalities together, then gradually opens the frozen components while adding new tasks.

The recipe starts with a thin “glue” layer that learns to translate between the vision encoder and the language model, then slowly loosens the glue and adds new heads so the system can generate full 3‑D worlds without losing what it already knows.

Encode R₁ and R₂ with the frozen vision encoder → obtain feature maps F₁, F₂.

Project F₁, F₂ through the learned connector → conditioning vectors C₁, C₂.

Feed C₁, C₂ and a random noise tensor into the diffusion backbone; the model predicts a denoised latent for T₁.

Decode the latent with the frozen VAE → synthetic RGB image S₁ for the target view.

Compute the flow‑matching loss between S₁ and the ground‑truth target image; back‑propagate only through connector, diffusion backbone, and condition‑fusion.

Even with only two reference views, the model learns to synthesize a consistent third view because the diffusion backbone is conditioned on a shared latent space that ties all views together.

How does this staged curriculum differ from simply fine‑tuning the whole model end‑to‑end on all tasks?

End‑to‑end fine‑tuning would update every weight simultaneously, quickly erasing the pretrained vision and language features that give the model its strong perception abilities. The staged approach isolates new objectives to a small subset of parameters at each phase, preserving the original representations while still allowing the model to acquire the additional 3‑D generation skills.

Puffin-16M Dataset

We build Puffin‑16M, a large, diverse 3‑D dataset that resolves prior scale and view limits.

Prior work such as Puffin‑4M suffers from three concrete drawbacks: its scale (4 M samples) is too small for today’s large models; all images are fixed at 512 × 512, forcing central crops that discard useful content; and the collection is dominated by single‑view pairs, offering only limited cross‑view or long‑range camera trajectories.

The dataset unifies vision, language, and explicit camera geometry at massive scale, providing both static images and long‑range camera trajectories.

Pick roll = $-30^\circ$, pitch = $0^\circ$, yaw = $180^\circ$, FoV = $60^\circ$.

Place a virtual pinhole camera with these extrinsics and intrinsics onto the panorama.

Render the perspective image, obtaining a 640 × 480 output with the specified aspect ratio.

Run the LLM captioner, which produces a sentence describing “a rooftop view looking down at a street, captured from a high‑position shot.”

Assign the camera‑height label “high‑position shot” based on the yaw‑adjusted horizon line.

This toy example shows how each sampled parameter directly determines the visual content and its semantic annotation, illustrating the pipeline’s deterministic yet diverse generation.

How does Puffin‑16M differ from conventional 2‑D image datasets?

Unlike ordinary image collections, every sample in Puffin‑16M carries explicit intrinsic (focal length, distortion) and extrinsic (roll, pitch, yaw, height) camera parameters, plus a structured 3‑D world state, enabling models to reason about geometry rather than just pixel statistics.

Puffin‑Cam‑15M expands the source panorama pool to 900 K images covering indoor, outdoor, synthetic, and real‑world scenes, then corrects geometric distortions via line‑segmentation and vanishing‑point estimation before uniformly sampling roll, pitch, and vertical FoV to produce ~15 M perspective views with diverse aspect ratios.

Puffin‑Traj‑1M adds full yaw sampling (0°–360°) to the extrinsics, generating one‑million long‑horizon trajectories that include continuous look‑down, look‑up, clockwise/counter‑clockwise rotation, and full 360° pans, using both single‑pass (straight‑line exploration) and recursive‑pass (memory‑preserving) generation strategies.

**Figure 4. An overview of the constructed Puffin-16M dataset.** It comprises 15 million vision-language-camera triplets with diverse resolutions and aspect ratios (left), along with 1 million trajectories featuring diverse and challenging rotational camera motions (right).

Table 2 compares Puffin‑16M against prior single‑task and unified‑task datasets, showing it uniquely provides full intrinsics (f, $\xi$) and extrinsics (ϕ, $\theta$, $\psi$, h), the largest frame count (16 M), and complete coverage of all seven detail categories (camera, text, reasoning, single‑view, cross‑view, trajectory, multi‑ratio).

Ablation Studies

Ablation studies quantify how each component affects physical and geometric performance.

Puffin‑World treats a scene as a native 3D world state—explicit geometry, appearance, and physics—rather than a raw pixel sequence.

We augment dozens of public image collections with absolute camera rotations, gravity fields, and dense depth maps, turning them into physically grounded training sources.

**Figure 12.** Ablation study of the proposed physics propagation (PP) strategy. For each sample, we show the generation results marked by the error up-vector and the error value with and without PP.

The PP drop‑off demonstrates that without explicit physical consistency the model drifts, producing mis‑aligned orientations and larger gravity‑field errors.

**Table 3.** Evaluation results on camera-to-world understanding. The comparison methods are evaluated on the public datasets: Stanford2D3D [4], MegaDepth [56], TartanAir [101], and LaMAR [83]. Our Puffin-World outperforms the previous methods on all median error metrics and most AUC metrics. Note that AnyCalib$^\dagger$ [94] is specifically designed for intrinsic parameter estimation (i.e., FoV estimation) and is included for reference. We color the best and second best results.

These results confirm that the combined physical‑geometric representation and the PP mechanism together deliver state‑of‑the‑art camera‑to‑world understanding.

Generation and Modeling Results

Puffin-World delivers superior camera‑controllable generation and 3D modeling across benchmarks.

Recall that Puffin‑World treats a scene as a native 3D world state—jointly modeling appearance, geometry, and physics—rather than a flat pixel sequence.

We evaluate camera‑controllable generation by estimating each generated image’s camera parameters with Puffin‑World, deriving pixel‑wise perspective fields, and comparing them to ground‑truth fields.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers