MetaView: Monocular Novel View Synthesis with Scale-Aware Implicit Geometry Priors

Yufei Cai, Xuesong Niu, Hao Lu, Kun Gai, Kai Wu, Guosheng Lin

MetaView synthesizes novel views from a single image by injecting implicit geometry priors and metric scale cues into a diffusion transformer.

How can we synthesize novel views from a single image while maintaining precise camera control and spatial consistency without relying on brittle explicit 3D geometry?

Monocular novel view synthesis models struggle to balance geometric consistency with generalization: explicit reconstruction methods are brittle under large viewpoint changes, while purely implicit models suffer from scale drift and imprecise camera control. MetaView addresses this by injecting implicit geometry priors and metric depth cues into a pretrained diffusion transformer via non-invasive parallel attention layers. This approach enables precise camera control and high-fidelity synthesis across large viewpoint changes, significantly outperforming both reconstruction-based and purely implicit baselines.

Paper Primer

The core mechanism hinges on two non-invasive injections into the frozen Multi-Modal Diffusion Transformer (MM-DiT) backbone. First, the model uses hierarchical features from a feed-forward geometry network (DepthAnything3) as implicit geometry tokens to regularize spatial structure. Second, it anchors the generation to a metric scale by injecting depth cues into a modified Rotary Positional Encoding (RoPE) subspace, effectively aligning the model's internal spatial perception with the target camera geometry.

MetaView achieves superior geometric consistency and generalization under large viewpoint changes compared to existing methods.

Quantitative evaluation on the DL3DV dataset shows MetaView consistently outperforming baselines across easy, medium, and hard difficulty levels, particularly in the hard subset where traditional metrics fail to capture perceptual quality. MetaView achieves a Dense Matching Distance (DMD) of 20.74 on the DL3DV-Hard subset, significantly lower than reconstruction-based baselines like Voyager (12.21) and PE-Field (8.91), while maintaining superior performance on easier subsets.

Why does this approach use parallel attention layers instead of fine-tuning the base model?

The authors freeze the pretrained MM-DiT backbone to preserve its rich semantic knowledge and prevent catastrophic forgetting, using parallel attention layers as a non-invasive mechanism to inject 3D awareness without altering the original model's learned representations.

What is the purpose of the Dense Matching Distance (DMD) metric?

Traditional metrics like PSNR and SSIM are dominated by low-frequency pixel alignment and fail to evaluate extrapolative synthesis under large viewpoint changes; DMD measures optical flow displacement between matched points to provide a more accurate assessment of spatial alignment and human-perceived quality.

MetaView demonstrates that high-fidelity novel view synthesis does not require explicit 3D reconstruction pipelines; instead, injecting sparse metric depth and implicit geometry priors into a frozen generative backbone is sufficient to achieve robust, scale-aware camera control.

Introduction and Motivation

We expose the trade‑off between explicit 3D geometry and implicit generation that MetaView resolves.

Current generative models produce high‑quality images but lack coherent spatial awareness. Explicit 3D priors enforce consistency yet become brittle under large viewpoint changes, while purely implicit approaches remain flexible but lose precise camera control and suffer scale drift.

**Fig. 1:** Monocular novel view synthesis results. **Upper:** Visual comparisons across different methods. **Lower:** Generalization of our MetaView in diverse scenes.

Diffusion‑based pipelines further illustrate the dilemma: reconstruction‑heavy approaches (panel a) suffer visual degradation, whereas fully implicit schemes (panel b) cannot enforce accurate camera control.

**Fig. 2:** Comparison of diffusion-based methods. Heavy reliance on explicit 3D inductive biases leads to severe visual degradation, while fully implicit methods fail to achieve effective camera control. Our method effectively addresses both issues.

Explicit 3D priors keep scene structure intact but become brittle when the camera moves far, whereas implicit generation stays flexible but drifts without reliable geometry; bridging this gap yields both consistency and controllable synthesis.

The core trade‑off is between brittle explicit 3D geometry and uncontrolled implicit generation; MetaView resolves it by injecting minimal, scale‑aware geometry priors into a frozen diffusion model.

Method Overview

We inject scale‑aware geometry priors into a frozen MM‑DiT via modified RoPE and parallel attention.

Explicit 3‑D reconstruction often introduces artifacts that degrade generated images. The challenge is to give a frozen diffusion model a sense of spatial layout without rebuilding its entire geometry pipeline.

We treat geometry as an extra token stream, encode its 3‑D coordinates with a modified RoPE that respects scale, and let a parallel attention branch fuse those cues with the image tokens.

RoPE rotates each token embedding by angle $\theta = s\cdot\sqrt{i^{2}+j^{2}}$ (radians).

Token (0,0) gets $\theta=0$ → no rotation; token (1,1) gets $\theta=0.5\cdot\sqrt{2}\approx0.71$ rad.

After rotation, the four embeddings are fed to the geometry‑only attention branch, producing geometry $Q,K,V$.

Parallel attention concatenates these with the image $Q,K,V$, allowing the image token at (1,1) to attend more strongly to geometry tokens that share a similar rotated orientation.

Scaling the rotation angle makes distant pixels rotate less, so their positional encoding stays distinguishable even when the camera zooms out.

How does this RoPE differ from the standard 1‑D RoPE used for text tokens?

Standard RoPE encodes only a single sequence index, rotating embeddings by a fixed frequency per position. Our modified RoPE incorporates the 2‑D grid coordinates $(i,j)$ and a view‑specific scale $s$, producing a rotation that reflects actual spatial distance rather than just token order.

**Fig. 3:** Overview of MetaView. **Left:** Our method is built upon the MM-DiT architecture, which integrates the implicit geometry priors by adding a split stream in the DiT. **Right:** We fuse geometry priors through self-attention mechanism, while injecting scale cues via a modified spatial-aware RoPE to address the scale drifting issue.

3D Cue Injection

We inject metric-scale depth cues into the diffusion backbone to resolve scale ambiguity in novel view synthesis.

Novel view synthesis requires consistent spatial scale between the camera pose and the scene. While explicit reconstruction methods maintain this via joint depth and pose estimation, implicit diffusion models often lack this coupling, leading to unstable viewpoint control where the scene scale drifts relative to the camera.

We anchor the generation process by injecting metric-scale depth cues directly into the model's positional encoding, forcing the internal spatial perception to align with the provided camera geometry.

The grid coordinates $(1, 2)$ are mapped to their respective RoPE subspaces.

The depth value $z=5.0$ is downsampled to match the latent token resolution and mapped to the third RoPE subspace.

The attention mechanism now computes similarity using the combined $(1, 2, 5.0)$ geometric embedding, effectively weighting tokens based on their 3D proximity rather than just 2D image distance.

By treating depth as a coordinate in the positional embedding, we allow the model to learn 3D spatial relationships without needing explicit 3D reconstruction or depth-lifting operations.

This approach avoids the computational overhead of handcrafted 3D operations. By downsampling the metric depth $z$ to match the latent token resolution, we inject sparse geometric cues that align the model's internal spatial perception with the camera geometry through the attention layers.

Implicit Relative Geometry Priors

Inject fine‑grained geometry tokens to steer diffusion generation without explicit 3D reconstruction.

Diffusion models can hallucinate object shapes and mis‑align poses when they lack explicit spatial guidance, especially in cluttered scenes where relative geometry matters most.

Think of the model as a painter who receives a quick sketch of the scene layout – the sketch does not dictate every brushstroke, but it keeps the painter from drawing a house upside‑down.

Project each $f_i$ with $W_G$: $g_1 = [1,0,1,0,0.5,0,0.5,0]$, $g_2 = [0,1.5,0,1.5,0,0,0,0]$, $g_3 = [0.5,0,0.5,0,0.5,0,0.5,0]$, $g_4 = [2,0,2,0,0,0,0,0]$.

Stack the four $g_i$ rows to form $G\in\mathbb{R}^{4\times8}$.

Assume the image token stream $X$ contains 6 tokens (each 8‑dim). Concatenate $X$ and $G$ → 10 tokens.

Compute $Q,K,V$ for the 10‑token sequence, then apply softmax attention. Geometry tokens receive higher attention weights from source tokens because their positional encoding is anchored at the origin.

The resulting attended representation blends visual content with the geometry sketch, preventing a downstream token from drifting into an implausible pose.

Even a tiny, linearly projected geometry sketch can steer the diffusion process away from severe distortions without ever reconstructing a full 3‑D mesh.

How does this differ from feeding an explicit depth map into the diffusion model?

An explicit depth map would be treated as a dense, absolute 3‑D signal that the model must reconstruct, tying it to a specific camera pose. Implicit geometry priors are lightweight token embeddings that only convey relative layout cues; they are fused via attention and never force the backbone to predict a full depth field, preserving the model’s flexibility and generalization.

Geometry-Guided Parallel Attention

Inject geometry cues via a parallel attention stream while keeping the pretrained backbone untouched.

Standard attention layers cannot absorb explicit 3‑D cues without altering the pretrained weights, which quickly erases the rich semantic knowledge they contain. The remedy is to keep the backbone immutable and add a side‑channel that brings geometry into the model. This side‑channel is the parallel attention mechanism described next.

A frozen multimodal diffusion transformer that already knows how to fuse image and text tokens into a joint representation.

We run a second attention lane that lets geometry tokens talk directly to image tokens, then simply add its output to the original attention – like a dedicated construction lane that merges back onto the main highway without forcing the existing traffic to change lanes.

Project $X$ with $W_X^{O}$ → $Q_X,K_X,V_X$ (each $2\times4$).

Project $G$ with $W_G$ then $W_X^{O}$ → $Q_G,K_G,V_G$ (each $2\times4$).

Compute image‑geometry attention: $A_{XG}= \text{softmax}(Q_X K_G^\top / \sqrt{4}) V_G$, yielding a $2\times4$ matrix.

Compute original image‑text attention (omitted here) → $A_{XC}$ of shape $2\times4$.

Add the two results: $\text{Out}=A_{XC}+A_{XG}$, producing the final $2\times4$ feature.

The parallel stream injects spatial cues without expanding the token sequence, so the frozen backbone sees exactly the same token count as before.

How does this parallel attention differ from simply concatenating geometry tokens to the image tokens?

Concatenation would force the original frozen attention heads to process a longer token sequence, mixing geometry and image in the same query/key space and requiring the backbone to adapt. Our parallel stream keeps the original image‑text attention unchanged and adds a separate image‑geometry attention with its own projections, so only the new parameters learn to use geometry.

Experimental Results

Experimental setup, the new DMD metric, and the headline quantitative gains.

MetaView attains the lowest DMD of 2.56 on the easy DL3DV subset, beating the runner‑up (Gen3C, 41.07) by 38.51.

Table 1 (DL3DV‑Easy) shows the DMD scores for all baselines.

DMD quantifies how far matched points shift between the source view and the generated view, directly measuring viewpoint consistency.

How does DMD differ from traditional PSNR/SSIM metrics?

PSNR/SSIM compare pixel‑wise intensity differences and are insensitive to spatial misalignment; DMD explicitly measures the displacement of matched points, so it remains informative even when the view changes dramatically.

**Fig. 4: Visual comparisons.** Our method enables more precise camera control and significantly outperforms baseline methods under large viewpoint changes.

**Table 1.** Quantitative comparison of novel view synthesis methods on DL3DV, RealEstate10K, and Sekai-Real-Walk-HQ datasets.

**Table 2.** Quantitative comparisons on different datasets.

**Fig. 5: Visualization of the ablation.** Scale cues provide the model with accurate spatial awareness, while geometry priors facilitate fine-grained relative consistency.

Related Work

MetaView adds geometry priors to a frozen MM-DiT to synthesize novel views without full 3D reconstruction.

ViewCrafter merges geometry‑aware tokens with the frozen diffusion backbone output to produce a coherent novel view.

Diffusion models iteratively denoise random noise to match a data distribution, enabling high‑fidelity image synthesis from text prompts.

Recent works replace classic multi‑stage pipelines with transformer‑based models that predict camera poses and depth maps in a single forward pass.

Methods aim to generate images from unseen camera viewpoints, traditionally via explicit 3D reconstruction or implicit neural representations.

Ablation Study

We isolate each design choice to see how it impacts view synthesis quality.

We run a series of controlled experiments on the DL3DV‑medium test split, removing one component at a time to measure its contribution.

Removing geometry tokens collapses fine‑grained relational cues: objects lose correct relative placement and delicate contours (e.g., a signboard vanishes), which drives PSNR down by 1.7 dB and inflates DMD.

Dropping the z‑axis from RoPE eliminates the scale anchor; the camera’s translation drifts, hurting low‑level fidelity (PSNR − 0.3 dB, SSIM − 0.02) while DMD rises, confirming the axis’s role in scale‑aware reasoning.

Switching the frozen backbone to FLUX.1‑Kontext reduces performance modestly, showing that the overall framework is robust to backbone changes but still benefits from the stronger Qwen‑Image‑Edit features of the default MM‑DiT.

**Fig. 6:** Comparison of metric reliability. Low-frequency structural alignment dominates PSNR and SSIM, while our proposed DMD aligns with human preference.

**Fig. 7: Trends of Different Metrics.** While traditional metrics (PSNR, SSIM, and LPIPS) fail to show a monotonic trend as the viewpoint changes, our proposed DMD exhibits monotonicity, accurately quantifying the discrepancy between views.

**Fig. 8: Application on open-domain data.** Our method preserves the rich semantic knowledge of the pretrained model, enabling generalization across diverse scenes.

Geometry Token Analysis

Additional visual analysis, limitations, and extended qualitative results for MetaView.

The attention maps reveal that generation tokens $X_{\text{gen}}$ consistently allocate higher weights to the geometry token $G$ across all parallel self‑attention layers. Averaged over heads, the bright regions correspond to $G$, indicating that the model actively pulls geometric cues during view synthesis.

MetaView inherits the semantic gaps of its frozen backbone, leading to extrapolation failures on out‑of‑domain scenes whose structures were never seen during pre‑training. In very large, distant scenes the model struggles with accurate scale estimation, and because it assumes static environments it cannot maintain geometric consistency for dynamic 4D content. Future directions include extending the pipeline with video‑generation models for spatio‑temporal synthesis and leveraging the native text‑conditioning to enable scene editing.

We supplement the main paper with additional qualitative examples on three benchmarks: DL3DV, RealEstate10K, and Sekai‑Real‑Walking‑HQ. The following figures display source, generated, and ground‑truth triplets, illustrating that MetaView preserves structural layout, lighting, and object placement across diverse indoor and outdoor settings.

**Fig. C.** Visualization results of MetaView on DL3DV.

**Fig. D.** Visualization results of MetaView on RealEstate10K.

**Fig. E:** Visualization results of MetaView on Sekai-Real-Walking.

Implementation and Qualitative Results

Implementation details, data curation, and view‑overlap computation for MetaView experiments.

We evaluate MetaView against several public baselines by running each model with its official pre‑trained weights and default settings. ViewCrafter generates 25‑frame videos at 1024 × 576 using VIPE‑derived poses; Gen3C produces 121‑frame videos at 1280 × 720, padding or uniformly sampling trajectories to meet the length requirement; Voyager creates 49‑frame clips at 768 × 512 via uniform pose interpolation; PE‑Field runs at 960 × 528; HY‑World renders 77‑frame videos at 832 × 480 with four inference steps; Lingbot‑World outputs 41‑frame videos at 832 × 480. All methods share the same VIPE‑derived camera poses and depth maps, and we resize every output to 960 × 528 before evaluation to eliminate resolution‑related metric bias.

Our training and test data are drawn from DL3DV, RealEstate10K, and Sekai‑Real‑Walk‑HQ. For each clip we obtain camera intrinsics, extrinsics, and per‑pixel depth using the VIPE pipeline, which leverages DepthAnything3‑Metric as a scale prior to align depth magnitudes with the estimated poses. To ensure high‑quality inputs we apply a multi‑stage filter: (1) discard any scene containing NaN/Inf depth values; (2) remove outdoor scenes whose minimum depth exceeds 50 m; (3) exclude pairs with a maximum relative camera translation larger than 100 m; (4) automatically detect and drop dynamic clips (people, animals, moving vehicles, subtitles) by prompting the vision‑language model Qwen3‑VL‑32B‑Instruct with a binary query and discarding any clip that returns “False”. This pipeline yields a static, metrically consistent dataset suitable for monocular novel‑view synthesis.

We quantify the difficulty of a view pair by its view‑overlap ratio, computed via depth‑based reprojection. Given camera intrinsics $K$, target relative extrinsics $T$, and source/target depth maps $z_{\text{src}}$, $z_{\text{tgt}}$, we unproject source pixels to 3D, transform them with $T$, and re‑project using $K$ to obtain $\tilde{z}$. A pixel is counted as co‑visible if (i) $\tilde{z}>0$ (in front of the target camera), (ii) its 2‑D projection lies inside the target image, and (iii) $|\tilde{z}-z_{\text{tgt}}|/z_{\text{tgt}}<0.10$. The overlap ratio is the fraction of co‑visible pixels over the full resolution. During training we restrict pairs to a maximum frame interval of 40 and require an overlap of at least 30 %. For evaluation we split the DL3DV test set into easy, medium, and hard subsets based on this ratio. Even when the frame interval is large and the camera zooms, the overlap remains high, which explains why baseline pipelines that rely on explicit geometry still struggle on the “easy” subset.

**Fig. A: Comparison of a zooming case.** Although the source image provides sufficient reference information, large viewpoint changes result in incomplete reprojection, causing certain reconstruction-based methods to fail in generating valid novel views.

**Fig. B: Visualization of attention maps for $X^{gen}$.** The maps depict the attention weights averaged across all heads. $X^{gen}$ consistently allocates significantly higher attention (brighter regions) to the geometry token $G$ than to other components.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers