InfiniSplat V2: Analyzing and Improving Feed-Forward 3DGS from Any Views

Yongzhen Hu, Hao Yu, Jiawei Wang, Xianming Zeng, Jianyun Xu, Xingliang Liu, Ruizhen Hu, Xiaowei Zhou, Hujun Bao, Sida Peng

InfiniSplat v2 improves feed-forward 3D Gaussian reconstruction by decoupling geometry from attribute learning and using large-angle supervision.

How can we improve feed-forward 3D Gaussian Splatting to maintain high-quality rendering when input viewpoints change significantly?

Feed-forward 3D Gaussian reconstruction methods often struggle to produce high-quality novel views because they jointly optimize geometry and appearance, leading to geometric degradation and persistent artifacts like ghosting or blur. The authors decouple these tasks by freezing the geometry branch and using ground-truth camera poses for training-time back-projection, while introducing large-angle target view supervision to better constrain Gaussian shapes. This approach achieves state-of-the-art perceptual quality across six datasets, reducing mean LPIPS by 25.8% and DISTS by 32.9% in pose-conditioned settings compared to the strongest baselines.

Paper Primer

The core mechanism is a dual-branch architecture: a frozen geometry branch provides stable point positions, while a trainable attribute branch adapts to rendering feedback. This is like a surveyor who fixes the foundation of a building before a decorator chooses the paint and finishes; by keeping the foundation rigid, the decorator cannot "fix" a crooked wall by moving it, which forces the model to learn better visual attributes instead of distorting the scene structure.

Decoupling geometry and attribute prediction prevents geometric degradation.

Ablation studies on ETH3D and DTU show that fine-tuning geometry or learning positional offsets worsens rendering metrics compared to keeping centers fixed. Freezing geometry reduces LPIPS by 16.6% to 23.2% compared to joint optimization.

Learned opacity effectively suppresses geometrically inconsistent points.

Analysis of validation scenes shows that pruning points with low opacity removes over 50% of geometrically inconsistent outliers while retaining over 93% of surface-consistent points. Selective pruning removes 51.8% of inconsistent points at a 0.01 opacity threshold.

Why does the model use ground-truth camera poses during training if it is intended for feed-forward inference?

Using predicted camera poses during training introduces pose errors that misguide the network under large-angle supervision. By using ground-truth poses for training-time back-projection, the model learns to align Gaussians correctly, which improves performance even when only predicted poses are available at inference time.

What is the role of the learned refiner in this architecture?

The refiner uses input-view rendering gradients to iteratively update Gaussian attributes (like color and opacity) while keeping the point positions fixed. This allows the model to refine the scene appearance based on rendering feedback without risking the structural distortions that occur if the geometry itself is updated.

For researchers building feed-forward 3D reconstruction systems, this paper demonstrates that rigid geometric priors are superior to joint optimization; fixing the geometry forces the attribute-learning branch to handle visual inconsistencies through opacity and appearance rather than structural distortion.

Introduction and Motivation

We expose why feed‑forward Gaussian reconstruction breaks under large viewpoint changes and outline InfiniSplat v2’s remedy.

Feed‑forward Gaussian reconstruction builds 3D scenes directly from input images, but under large viewpoint changes the predicted Gaussian attributes become misaligned with the underlying geometry, leading to blur, ghosting, and missing detail. Prior feed‑forward pipelines jointly optimize geometry and attributes, which lets geometry degrade when the attribute branch pushes for visual fidelity. InfiniSplat v2 tackles this by (1) freezing a geometry branch to keep geometry stable, (2) using ground‑truth camera poses during training to align Gaussians, and (3) supervising with large‑angle target views to force robustness.

It predicts a full 3D Gaussian scene from a handful of input images in a single forward pass, bypassing per‑scene optimization.

**Figure 1.** From point clouds to Gaussian scenes. InfiniSplat v2 retains point positions from a geometry foundation model and learns Gaussian attributes for high-quality novel view rendering. Left: input views, point clouds, and corresponding Gaussian renderings, with enlarged regions outlined in orange. Right: additional examples comparing point clouds with the resulting Gaussian scenes.

The core challenge is learning Gaussian attributes that remain accurate when the underlying geometry is misaligned under large viewpoint changes.

Foundations and Prior Work

We situate our approach among prior geometry and Gaussian reconstruction methods.

We first review prior geometry foundation models and feed‑forward Gaussian reconstruction pipelines that our method builds upon.

They infer scene structure (depth maps and camera poses) from images, providing a geometric scaffold for downstream feed‑forward Gaussian reconstruction.

DUSt3R learns a dense 3‑D scene representation from images, producing point clouds that can be directly rendered as Gaussians.

MASt3R extends DUSt3R by incorporating multi‑view consistency checks, improving geometry quality for scenes captured from many angles.

GS‑LRM predicts Gaussian attributes from multiple views using a learned reconstruction loss, aligning appearance with geometry.

The InfiniSplat v2 Pipeline

Scale-aligned training aligns predicted depths and camera scales to eliminate pose‑induced errors.

Predicted back‑projection poses often drift from the ground‑truth cameras, so the supervision signal is applied to mis‑aligned geometry. This misalignment injects camera‑induced residuals into the attribute‑learning loss, degrading reconstruction quality under large viewpoint changes.

**Figure 2.** Training pipeline of InfiniSplat v2’s feed-forward reconstructor. A frozen geometry branch predicts depth and cameras, while a trainable branch predicts Gaussian attributes. Scale-aligned GT cameras construct the point cloud; after opacity pruning, input and large-angle target views supervise Gaussian learning.

Instead of feeding predicted poses directly to the loss, we rescale both depths and GT camera translations to a common unit defined by the average radial distance of back‑projected points.

Compute $\text{snorm} = (2+3+4+5)/4 = 3.5$.

Normalize a depth value of 7 → $\bar D = 7/3.5 = 2.0$.

Predicted camera translation $\mathbf{t}= (0.7,\,0.0,\,0.0)$ becomes $\mathbf{t}/\text{snorm} = (0.2,\,0.0,\,0.0)$.

GT camera center is at $(1.0,\,0.0,\,0.0)$. Fitting a similarity transform to the normalized predicted center yields $s_C = 0.2$.

Scale‑aligned GT translation becomes $\mathbf{t}^\star = 0.2 \times (1.0,\,0.0,\,0.0) = (0.2,\,0.0,\,0.0)$, matching the normalized predicted translation.

Normalizing by the average radial distance forces both predicted and GT cameras onto the same length scale, so the loss no longer confounds depth scaling with attribute errors.

How does Scale‑Aligned Training differ from simply using the predicted poses directly in the loss?

Using raw predicted poses mixes two error sources: depth‑scale drift and pose mis‑alignment. Scale‑Aligned Training first removes the arbitrary scale (snorm) and then rescales GT translations ($s_C$), so the supervision targets only attribute mismatches, not pose‑induced geometry errors.

Even with perfectly aligned geometry, supervising only from input viewpoints leaves Gaussian scales under‑constrained. To force the model to learn shape from diverse perspectives, we deliberately sample target cameras that look far away from the input directions.

We bias the selection of target cameras toward those whose optical axes differ most from any input camera, ensuring that each supervision step sees the scene from a novel angle.

Start from the pruned scene G₀.

Render $G_t$ into all input views and compute per‑pixel reconstruction‑loss gradients w.r.t. Gaussian attributes (RenderGrad).

Stop‑gradient the raw gradients and concatenate them with fixed DPT image features f.

Feed the combined features into a recurrent KNN Transformer U, which updates each Gaussian’s hidden state and predicts attribute residuals.

Apply the residuals in the attribute parameter space to obtain the next scene $G_{t+1}$.

Repeat for a fixed number of refinement steps; the final scene is `G_ref`.

Training proceeds in two stages: Stage I updates only the last K transformer blocks of the attribute branch and the Gaussian decoder, while Stage II freezes the reconstructor and trains the feature projector and refiner using the same input‑view losses and the large‑angle supervision.

**Figure 10.** Three-step refinement on ETH3D with eight input views. Red boxes highlight the compared floor region.

At inference time the same model can operate either with known camera poses (pose‑conditioned) or without them (pose‑free), using the geometry branch’s predicted cameras and the same scale‑alignment machinery to construct the initial point cloud before refinement.

Experimental Results

Feed‑forward InfiniSplat v2 beats all baselines on LPIPS and DISTS across six datasets.

Feed‑forward Gaussian reconstruction often fails under large viewpoint changes because geometry and attributes become entangled; InfiniSplat v2 decouples them with scale‑aligned training and ground‑truth cameras.

Our feed‑forward model attains the lowest mean LPIPS and DISTS among all baselines on six datasets, in both pose‑conditioned and pose‑free settings.

Table 1 shows Ours/Ours∗ leading on every metric; the LPIPS gap reaches 25.8 % over the strongest baseline (ReSplat) when poses are known.

**Figure 3.** Object-level qualitative comparisons under multi-view input settings.

**Figure 4.** Scene-level qualitative comparisons under multi-view input settings.

**Figure 11:** Pose-free rendering on ETH3D: grass textures and relief details.

**Figure 12.** Feed-forward reconstructions and renderings with 12–64 input views.

**Figure 13.** Large-scale reconstruction on KITTI. Insets show elevated-view renderings at locations A–F.

Comprehensive Quantitative Results

Full quantitative results across six datasets, showing our method leads on all metrics.

Our refined output (Ours*) ranks first on all four mean metrics in the pose‑conditioned setting.

Table 4 shows Ours* achieving the highest PSNR (20.9334 dB), SSIM (0.7131), and the lowest LPIPS (0.2392) and DISTS (0.1159) among all methods.

**Table 1.** Quantitative comparison of our method against state-of-the-art approaches on the DL3DV, ETH3D, NRGBD, Mip-NeRF 360, RE10K, and DTU datasets. We report LPIPS and DISTS metrics (lower is better). The best results are bolded.

**Table 4.** Pose-conditioned results averaged over six datasets.

Tables 5–7 further break down LPIPS and DISTS for four, eight, and sixteen input views, confirming consistent gains of our method across view counts.

Ablation Studies

We test each reconstruction component and refinement strategy to see how they affect quality and speed.

We evaluate the feed‑forward reconstructor on ETH3D and DTU, averaging over N = 4, 8, 16 views, while keeping all other training settings fixed.

The w/o A variant freezes the attribute backbone (K = 0) and trains only the Gaussian head; the w/o G variant fine‑tunes the geometry backbone at 1e‑6 with a fixed Gaussian‑head learning rate; the w/ O variant predicts 3D offsets in the scale‑aligned frame; the w/o C variant uses predicted cameras for back‑projection; the w/o V variant keeps local‑view sampling and target supervision.

Qualitatively, the offset variant produces noticeably stronger surface artifacts and depth deviations compared with the fixed‑center model, confirming that accurate center prediction is crucial for clean geometry.

Three‑step learned refinement improves PSNR by 0.64 dB over the plain feed‑forward output.

Table 8 shows Ours * achieving 17.6074 dB versus Ours at 16.9643 dB on ETH3D.

**Table 6.** Per-dataset results with $N = 8$ input views. Mean weights datasets equally.

**Table 7.** Per-dataset results with $N = 16$ input views. Mean weights datasets equally.

**Table 9.** Feed-forward network latency: mean and min–max over five runs.

**Figure 5.** Effect of positional offsets on rendering and depth in an NRGBD scene.

**Figure 6.** Effect of reconstruction components on surface artifacts and structural detail.

**Figure 7.** Opacity distributions and the effect of pruning on geometrically inconsistent points. GT depth independently defines the inconsistent subset; distributions are normalized separately.

Opacity and Geometric Consistency

Additional analyses of opacity, latency, and qualitative behavior reveal current limits.

We first examine how opacity learning interacts with geometric consistency on the held‑out Hypersim validation set. Geometric labels are derived from ground‑truth depth with a tolerance of max($0.03$ m, $0.01\,d$), and depth‑edge samples are excluded.

Next, we measure feed‑forward network latency at four input resolutions on a single H20 GPU using the same ETH3D scene and eight input views. Each resolution is warmed up once, followed by five synchronized CUDA‑event timings.

Qualitative pose‑free reconstructions on ETH3D show markedly fewer repetitive stripe artifacts and better‑preserved relief structures compared with several baselines.

We also evaluate on a broad set of scenes—DDAD, Waymo, ETH3D, Hypersim, DTU, Mip‑NeRF 360, and Mill19—covering streets, buildings, interiors, and objects, confirming the approach works across diverse environments.

For large‑scale reconstruction we apply the feed‑forward reconstructor to a KITTI street sequence of over 1,000 frames, using sliding windows to assemble a coherent Gaussian scene.

Limitations remain: opacity pruning can suppress unreliable Gaussians but cannot fix severe geometry errors; pixel‑aligned prediction introduces redundancy not fully removed by voxel merging; and sparse input coverage leaves holes in unobserved regions.

Experimental Setup Details

Details of data, architecture, training, and evaluation for reproducibility.

We train on three large‑scale 3D datasets—Hypersim, Raw3D, and ScanNet++—with sampling probabilities 60 %, 10 %, and 30 % respectively, using disjoint training and validation scenes. Each training sample contains four target views and a variable number $N\in\{4,\dots,12\}$ of input views. Image resolution scales with $N$: longer side 1022 px for $N\le8$, 756 px for $N=9,10$, and 518 px for $N=11,12$.

Target‑view selection ranks non‑input views by decreasing minimum optical‑axis angle $a(v, C)$ (Eq. 5), splitting them into large‑angle and small‑angle pools. A sample draws the large‑angle pool with probability 0.7 (or the small‑angle pool with 0.3) and then uniformly picks four target views without replacement, requiring at least four candidates in the chosen pool.

Visibility masks are built from ground‑truth depth. A pixel is kept if its GT depth exceeds $10^{-6}$, passes the validity mask, and after back‑projection satisfies $|z_{\text{proj}}-d_{\text{src}}|\le\max(τ_{\text{abs}},τ_{\text{rel}}\,d_{\text{src}})$ with default thresholds $τ_{\text{abs}}=0.03$ and $τ_{\text{rel}}=0.01$.

The “w/o V” variant samples local interpolation views between input views while preserving the full mask construction and target supervision.

Evaluation is performed on six benchmark datasets using $N\in\{4,8,16\}$ input views per sample.

Both model branches are initialized from the DA3NESTED‑GIANT‑LARGE‑1.1 checkpoint with independent parameters. Gaussian voxel merging uses a side length of 0.004 in the scale‑normalized reconstruction frame, and Gaussians with opacity below $τ_{\alpha}=0.01$ are pruned after merging and before refinement. All experiments employ gsplat with $ε_{\text{2d}}=10^{-2}$.

Scale alignment applies only a uniform scale factor $s_{C}$, obtained by fitting an Umeyama similarity transform to the GT trajectory and the normalized predicted trajectory; this scale is applied to both input and target translations while preserving GT intrinsics and relative rotations.

Optimization uses AdamW with $β=(0.9,0.95)$, a global batch size of 16, and L2 gradient clipping at 1.0 on 16 H20 GPUs. Each GPU processes one multi‑view sample without gradient accumulation.

Stage I trains only the last ten Transformer blocks of the attribute backbone and the Gaussian head, using peak learning rates $10^{-6}$ (backbone) and $2\times10^{-4}$ (head) over 25 000 steps, with a 2 000‑step linear warmup followed by cosine decay to 0.1× the peaks, taking roughly 48 hours.

Stage II freezes the reconstructor, trains the DPT feature projection and refiner for 5 000 steps, starting at learning rate $2\times10^{-4}$ without warmup and decaying cosine to 0.1× the initial value, consuming about 13 hours.

Stage‑I reconstruction loss renders the initial scene $G_{0}$ into all input and target views $v\in V=C\cup T$ as $I_{v}=R(G_{0};\Pi^{*}v)$, computes a spatial LPIPS distance $d_{v}$ on valid pixels, and aggregates the per‑view loss with a max‑clamp to ensure a minimum value of 1.

Stage‑II refinement projects DPT features to eight channels, merges them with Gaussian voxel groups, and feeds concatenated features, attributes, gradients, neighbor distances, and hidden states through four KNN Transformer blocks (single‑head attention over eight neighbors, LayerNorm, MLP, residual). The block outputs are concatenated with the original input and passed through an MLP to predict attribute and hidden‑state increments.

During training and inference, feedback loss $ℓ_{\text{fb}}=0.8L_{1}+0.2L_{\text{DSSIM}}$ (with $L_{\text{DSSIM}}=\frac{1-\text{SSIM}}{2}$) is applied to rendered views; gradients w.r.t. Gaussian attributes are normalized and detached before entering the shared updater.

Opacity regularization penalizes logit changes between final and initial Gaussians: $L_{\text{opacity}}=\frac{1}{M}\sum_{m=1}^{M}[\logit_{\varepsilon}(\alpha_{K_{r}}^{m})-\logit_{\varepsilon}(\alpha_{0}^{m})]^{2}$ with $\varepsilon=10^{-6}$.

A first‑order surrogate loss uses stop‑gradient copies $z_{k}=\operatorname{sg}(x_{k})$ to compute rendering gradients $r_{k}=\operatorname{sg}(\nabla_{z}L_{\text{render}}(z_{k}))$, forming a proxy $L_{k}^{\text{proxy}}$ that matches the rendering loss value but provides a gradient $\nabla_{\theta}L_{k}^{\text{proxy}}=(\partial x_{k}/\partial\theta)^{\!\top}r_{k}$ for trainable parameters $\theta$.

Chunked execution reuses a global KNN graph across steps; core sets of at most 200 000 points are expanded with four‑hop neighborhoods, and any chunk exceeding 300 000 points is recursively bisected along its longest axis, with neighborhoods recomputed for each sub‑chunk.

Camera protocol distinguishes pose‑conditioned and pose‑free evaluation. In the pose‑free setting, reference input cameras serve only to define the evaluation frame and estimate translation scale; they are not used for reconstruction. In the pose‑conditioned setting, reference input cameras are available for reconstruction and target intrinsics are fixed to ground‑truth values.

Metrics include PSNR, SSIM, LPIPS, and DISTS. For each input configuration, we average these metrics equally across target views, examples, scenes, the three input view counts, and the six datasets.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers