InfiniSplat: Implicit Gaussian Decoding for Large-Baseline Monocular View Synthesis

Jiawei Wang, Hao Yu, Yongzhen Hu, Xinyi Yang, Tao Ni, Xin Zhan, Junbo Chen, Xiaowei Zhou, Ruizhen Hu, Sida Peng

InfiniSplat replaces pixel-grid Gaussian placement with geometry-guided surface sampling for stable large-baseline novel view synthesis.

How can we generate a renderable 3D Gaussian scene representation from a single image that remains structurally coherent under large viewpoint changes?

Existing single-image 3D Gaussian Splatting (3DGS) methods generate primitives from fixed pixel grids, causing them to tear, drift, or lose structural coherence when the camera moves significantly from the source view. InfiniSplat shifts to a surface-aligned representation: it uses monocular depth priors to sample Gaussian supports based on local surface area, then uses a shared implicit decoder to predict Gaussian attributes at these flexible locations. This approach achieves state-of-the-art performance across multiple cross-dataset benchmarks, maintaining stable scene geometry and fewer rendering artifacts under large-baseline viewpoint changes.

Paper Primer

The core move is a two-stage pipeline: geometry-guided sampling and query-conditioned implicit decoding. The sampling stage acts like a surveyor, placing more Gaussian supports where the scene surface is larger or more slanted, while the decoder acts like a universal translator, mapping image features at those arbitrary coordinates into consistent Gaussian parameters.

InfiniSplat significantly improves structural stability and rendering fidelity for large-baseline novel view synthesis compared to pixel-aligned baselines.

Across four real-scene datasets (ETH3D, ScanNet++, Tanks and Temples, DL3DV), InfiniSplat-RGB outperforms the SHARP baseline by +1.919 PSNR and +0.048 SSIM on average. +1.919 PSNR average improvement over the previous state-of-the-art.

The framework remains effective when provided with sparse depth prompts, outperforming specialized LiDAR-conditioned baselines.

InfiniSplat-LiDAR improves over ADGaussian by +10.299 PSNR and +0.212 SSIM on the ETH3D, ScanNet++, and Tanks-and-Temples datasets.

Why does the paper move away from the standard pixel-aligned grid?

Pixel-aligned representations treat Gaussians as independent splats tied to fixed image coordinates, which fails to capture coherent 3D surfaces. By decoupling placement from the grid, the model can allocate more primitives to geometrically demanding regions, preventing the tearing and structural drift common in large-baseline views.

What is the role of the dual-branch encoder in the decoding process?

The encoder extracts complementary information: a DINO branch provides high-level semantic context for robust scene-level generalization, while a CNN branch captures low-level texture cues for fine-grained local appearance. The decoder fuses these to predict accurate Gaussian attributes at each sampled support.

For researchers in novel view synthesis, this paper demonstrates that the bottleneck in single-image 3DGS is not just the rendering engine, but the geometric organization of the primitives. Moving from pixel-aligned to surface-aligned support sampling is a necessary step for stable, large-baseline scene reconstruction.

The Challenge of Single-Image 3DGS

We expose why pixel‑aligned 3D Gaussian Splatting breaks under large viewpoint shifts.

Feed‑forward 3D Gaussian Splatting (3DGS) promises a single‑image pipeline for novel view synthesis, yet its reliance on a pixel‑aligned representation causes structural degeneration when the camera moves far from the input view.

When Gaussians are anchored to fixed image‑grid locations, they cannot faithfully follow the true surface geometry, so large viewpoint shifts expose “cracks” and broken structures.

**Fig. 1.** InfiniSplat generates a surface-aligned 3D scene representation from a single image, enabling high-quality novel view synthesis across diverse scenes. Compared with SHARP [Mescheder et al. 2025], InfiniSplat maintains more surface-aligned structures and produces more plausible renderings.

Grid‑binding fails for large‑baseline synthesis because fixed‑grid Gaussians cannot track true surface geometry.

Prior Approaches to Novel View Synthesis

We situate InfiniSplat within prior single-image NVS methods and define the SHARP and ADGaussian baselines.

Single-image novel view synthesis (NVS) remains challenging because a single photo provides limited 3D cues. We review the main families of prior work and introduce two baselines that serve as reference points for our experiments.

SHARP predicts a set of 3D Gaussians in a single forward pass, aligning each Gaussian to the input image grid.

ADGaussian also generates 3D Gaussians in one pass but augments the pixel‑aligned positions with an adaptive displacement field.

Early single‑image NVS approaches warp a depth map, use soft layering, layered depth images, or multiplane images to synthesize nearby views.

Recent methods employ diffusion models or large transformers to generate novel views directly, often via iterative synthesis.

These approaches reconstruct geometry and texture from a single image, focusing on object‑centric assets or generative 3D creation.

Predict explicit 3D Gaussians in a single forward pass for efficient novel‑view rendering, typically using pixel‑aligned generation.

Model signals as continuous functions over query locations, applied to images, depth, and 3D shapes.

Learn compact 3D Gaussians from multi‑view inputs using learnable query tokens that aggregate features and decode Gaussian attributes.

InfiniSplat Pipeline Overview

InfiniSplat generates surface‑aligned 3D Gaussians from a single image via geometry‑guided sampling and implicit decoding.

Existing feed‑forward 3D Gaussian Splatting binds Gaussian generation to the discrete image grid, which leads to structural cracks when the viewpoint moves. InfiniSplat removes this bottleneck by sampling surface‑aligned supports and decoding them with a geometry‑aware network.

Predict dense depth $D$ and camera intrinsics $\hat{K}$ from the input image $I$ using the frozen monocular depth model $\Phi_{geo}$.

Extract semantic features $F_{dino}$ (via DINO) and texture features $F_{cnn}$ (via CNN) from $I$ with the trainable dual‑branch encoder $\Phi_{img}$.

Apply geometry‑guided sampling $Q$: back‑project $D$ to obtain per‑patch 3D surface areas, distribute $N$ supports proportionally to area, and initialize each support as a base Gaussian $\bar{G}$.

Run implicit Gaussian decoding $D_\theta$: for each support, query $F_{dino}$ and $F_{cnn}$, fuse them through a gated MLP, and predict bounded offsets that transform $\bar{G}$ into the final Gaussian set $G$.

Back‑project each pixel to a 3D point: the four points lie at $(0.5,0.5,1)$, $(1.0,0.5,2)$, $(0.5,1.0,1)$, $(1.0,1.0,2)$.

Compute surface area per pixel (here all equal), so supports are distributed uniformly across the four points.

Initialize a base Gaussian at each point with default scale $(0.1,0.1,0.1)$, zero rotation, white color, and opacity $0.8$.

Even a tiny depth map yields a set of surface‑aligned supports that break the regular image grid, enabling the later decoder to operate on true geometry rather than pixel coordinates.

**Fig. 2. InfiniSplat pipeline.** Given a single input image, InfiniSplat predicts geometry priors and extracts DINO/CNN image features. Guided by the predicted depth-induced surface layout, it samples 2D supports and initializes base Gaussians beyond the fixed pixel lattice. The implicit Gaussian decoder queries image features at these supports and predicts bounded updates to produce a surface-aligned 3D Gaussian representation for feed-forward novel view synthesis.

How does this differ from a naïve per‑pixel regression that predicts Gaussian parameters directly on the image grid?

Per‑pixel regression assumes a regular grid and lacks any notion of underlying surface geometry, so the predicted Gaussians inherit the grid’s discretization artifacts. Geometry‑guided sampling first constructs a surface‑aware layout, and the implicit decoder operates on that irregular layout, preserving surface continuity and avoiding the “cracks” seen in grid‑bound methods.

Surface-Aligned Gaussian Representation

InfiniSplat replaces grid‑bound Gaussian generation with geometry‑guided sampling and an implicit decoder that refines surface‑aligned Gaussians.

Grid‑bound Gaussian generation in prior 3DGS pipelines leads to “cracks” when the viewpoint moves, because the Gaussians are forced onto a discrete pixel lattice. InfiniSplat solves this by first sampling supports that follow the underlying surface geometry and then refining those supports with a query‑conditioned implicit decoder.

Instead of placing Gaussians on every pixel, we sample them from surface patches whose size reflects the true 3D area, so the supports naturally follow the scene geometry.

Compute $A_t$ using the cross‑product formula → $0.55$.

Normalize by the total area (here only one triangle) → $w_t=1.0$, so the triangle receives the full sampling budget.

Draw a single query $q$ uniformly inside the pixel triangle (e.g., $q=(0.3,0.4)$).

Back‑project $q$ to 3D: $\bar\mu = \Pi^{-1}(q, D(q); \hat K) \approx (0.3,0.4,1.02)$.

Sample color $\bar c = I(q)$ from the RGB image (e.g., $\bar c = (0.7,0.5,0.3)$).

Area‑weighted sampling automatically allocates more queries to larger or more slanted surface patches, avoiding the uniform‑grid bias that causes degenerate splats.

How does geometry‑guided sampling differ from simply sampling uniformly on the image grid?

Uniform sampling treats every pixel as equally important, ignoring the actual 3D surface area each pixel covers. Geometry‑guided sampling first computes the true 3D area of each local triangle, discards triangles that straddle depth jumps, and then draws samples proportionally to those areas, so the resulting supports follow the scene’s geometry rather than the pixel lattice.

Instead of predicting a full set of Gaussians from scratch, we start from geometry‑aligned base Gaussians and let a shared MLP refine their attributes conditioned on image features queried at the base locations.

Fuse features: $h = 0.65\cdot(0.2,0.1) + 0.35\cdot(0.8,0.7) = (0.44,0.34)$.

MLP predicts residuals $\Delta u=(0.02, -0.01)$, $\Delta z=0.05$, $\Delta s=(0.01,0.01,0.01)$, $\Delta r=(0,0,0,1)$, $\Delta c=(0.02,-0.01,0.00)$, $\Delta\alpha=0.01$.

Apply scaling $\lambda_{xy}=0.5$, $\lambda_z=0.3$: updated image‑plane coordinate $(0.5+0.5\cdot0.02,\,0.4+0.5\cdot(-0.01))=(0.51,0.395)$, depth $1.2+0.3\cdot0.05=1.215$.

Back‑project to 3D: $\mu = \Pi^{-1}((0.51,0.395), 1.215; \hat K)$ yields a refined mean slightly shifted toward the true surface.

Update color: $\bar c + \Delta c = (0.62,0.59,0.60)$ and opacity $=0.5+0.01=0.51$.

The decoder’s residual updates fine‑tune the base Gaussian while keeping it anchored to the geometry‑guided support, preventing the “pixel‑grid drift” that plagues naïve feed‑forward predictions.

Why not predict Gaussian parameters directly from image pixels instead of using an implicit decoder?

Direct prediction ties each Gaussian to a fixed pixel, inheriting the grid‑binding degeneration that causes cracks under view changes. The implicit decoder starts from geometry‑aligned base Gaussians and learns residual updates conditioned on image features, allowing the final Gaussians to move off the pixel lattice and stay coherent with the underlying surface.

InfiniSplat offers two model variants: the RGB‑only version that relies solely on a monocular depth prior, and a LiDAR‑conditioned version that augments the depth prior with sparse LiDAR measurements. Both share the geometry‑guided sampling pipeline and the implicit decoder, differing only in the source of the depth scaffold.

Training optimizes a combination of rendering losses (pixel‑wise $L_1$ and perceptual VGG losses) and Gaussian regularization (scale bounds and local smoothness). The rendering loss drives the Gaussians to reproduce target views, while the regularization keeps scale and opacity stable across neighboring supports.

Experimental Setup and Quantitative Results

We detail the datasets, baselines, and training protocol used to evaluate InfiniSplat.

We evaluate InfiniSplat in a cross‑dataset zero‑shot novel‑view synthesis scenario, synthesizing a target view from a single source image.

For each of the four real‑scene datasets we generate 512 source‑target pairs with >60 % frustum overlap, covering four baseline distance ranges, and align the rendered targets to the dataset‑specific resolutions before computing PSNR, SSIM, and LPIPS.

Visibility masks are obtained by projecting the source geometry into the target view and refining with simple morphological operations, applied uniformly to all methods.

InfiniSplat‑RGB improves average PSNR by +3.0 dB over the strongest RGB‑only baseline while also lowering LPIPS by 0.022.

Table 1 shows the quantitative comparison.

**Table 1.** Quantitative evaluation on zero-shot novel-view synthesis. PSNR, SSIM, and LPIPS are reported as separate metric columns. Higher is better for PSNR and SSIM, while lower is better for LPIPS. The best and second-best results are highlighted in green and yellow, respectively, within each setting block.

Ablation Studies and Qualitative Analysis

We examine how each component of InfiniSplat affects visual fidelity and surface coherence.

Recall that InfiniSplat decouples Gaussian generation from the image grid, using geometry‑guided sampling and an implicit decoder to keep Gaussians aligned with scene surfaces.

**Fig. 7.** Qualitative effect of learned Gaussian updates. Without learned updates, the model only renders base Gaussians initialized from geometry and sampled color. The result preserves coarse scene structure but remains blurry and lacks local texture and boundary details.

**Fig. 8.** Qualitative effect of image feature branches. Removing the CNN branch weakens low-level appearance and boundary details, producing blurrier renderings. Removing the DINO branch is more destructive: the model loses its main semantic backbone and fails to generalize, producing large holes and missing structures.

**Fig. 9.** Qualitative effect of Gaussian regularization. Removing Gaussian regularization produces degenerate Gaussian attributes, leading to locally thin, transparent, and unstable regions in the rendered target view.

**Fig. 10.** Qualitative effect of geometry-guided sampling and implicit decoding. Replacing geometry-guided sampled supports with pixel-aligned supports introduces cracks under large viewpoint changes. Replacing the implicit decoder with a DPT decoder further amplifies these discontinuities, producing more severe holes and surface breaks.

Removing learned Gaussian updates drops PSNR by 1.7 on ETH3D.

Full model PSNR 20.531 → 18.819 (Table 2).

Omitting the DINO branch reduces PSNR by 9.7 on ScanNet++.

Full model PSNR 22.240 → 12.501 (Table 2).

Removing the CNN branch lowers PSNR by 1.9 on ScanNet++.

Full model PSNR 22.240 → 20.368 (Table 2).

Discarding Gaussian regularization cuts PSNR by 1.5 on ScanNet++.

Full model PSNR 22.240 → 20.717 (Table 2).

Replacing geometry‑guided sampling reduces PSNR by 0.7 on ScanNet++.

Full model PSNR 22.240 → 21.576 (Table 2).

Using a pixel‑aligned DPT decoder instead of the implicit decoder lowers PSNR by 1.4 on ScanNet++.

Full model PSNR 22.240 → 20.811 (Table 2).

Default support budget (1.5 M) yields 0.009 s inference time.

Table 3 shows inference 0.009 s at 1.5 M supports.

Multiplicative depth noise of 5 % degrades PSNR by 2.3.

Table 4: PSNR 25.880 → 23.595 as noise $\sigma$ rises to 5 %.

Limitations and Future Directions

We discuss remaining failure modes of InfiniSplat and recap its contributions.

Recall that InfiniSplat decouples Gaussian generation from the input image grid by using geometry‑guided sampling and an implicit decoder, which reduces the “cracks” seen in prior feed‑forward 3D Gaussian Splatting methods. Despite this, the method still inherits two fundamental sources of error: (1) single‑view ambiguity, which leaves occluded regions undefined, and (2) dependence on a pretrained depth prior, which can be inaccurate on challenging surfaces.

**Fig. 11. Qualitative failure cases.** The columns show large disocclusion, an incorrect depth prior, and extreme extrapolation. From top to bottom, the rows show the input image, the depth predicted by the monocular depth model and warped to the selected target viewpoint, and the novel-view rendering of the Gaussian scene predicted by InfiniSplat. Brown boxes highlight the affected regions. The examples exhibit incomplete geometry under large disocclusion, artifacts inherited from incorrect depth priors, and severe structural distortion under extreme extrapolation.

Future work could mitigate these issues by integrating stronger generative priors, making geometry estimation uncertainty‑aware, or incorporating sparse multi‑view inputs while preserving the efficiency of feed‑forward Gaussian prediction.

In conclusion, InfiniSplat delivers a feed‑forward pipeline that aligns Gaussian supports with depth‑induced surfaces, yielding state‑of‑the‑art quantitative performance on ETH3D, ScanNet++, Tanks and Temples, and DL3DV, and demonstrably cleaner novel‑view renderings. The results show that single‑image 3D Gaussian Splatting can move beyond pixel‑aligned splat expansion toward a more structurally stable representation.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers