4D Human-Scene Reconstruction from Low-Overlap Captures

Minhyuk Hwang, Sangmin Kim, Seunguk Do, Daneul Kim, Jaesik Park

StudioRecon reconstructs 4D human scenes from sparse, low-overlap cameras by decoupling background and human priors.

How can we reconstruct high-fidelity 4D human-scene models from only a few sparse, low-overlap camera views?

Reconstructing dynamic human scenes from sparse, low-overlap cameras is difficult because standard methods struggle with missing viewpoint coverage and entangled human-background representations. StudioRecon solves this by decoupling the scene: it uses a video diffusion model to synthesize dense background supervision and parametric body models to constrain human geometry. This approach achieves state-of-the-art novel view synthesis quality across diverse real-world datasets, outperforming existing sparse-view reconstruction methods.

Paper Primer

The core challenge in "in-the-wild" studio capture is that cameras are often positioned 90 degrees apart, leaving large portions of the scene unobserved. Existing joint reconstruction methods fail here because errors in human pose and background geometry become entangled, leading to persistent artifacts in under-observed regions.

StudioRecon is a decoupled pipeline: it synthesizes dense novel views for background supervision while using SMPL body models to anchor human reconstruction. To harmonize these components, it employs a recursive enhancement module: it applies single-step diffusion with motion-adaptive consistency injection to remove artifacts and ensure temporal coherence across frames.

StudioRecon significantly improves perceptual quality and reconstruction fidelity in sparse-view settings.

Quantitative evaluation across 8 scenes (EgoHumans, Harmony4D, Mobile Stage, SelfCap) shows consistent outperformance of baselines like Dyn-3DGS and MonoFusion. Dense view synthesis provides a +2.4 PSNR gain, while recursive enhancement reduces LPIPS by 27%.

Why is a decoupled approach necessary for this problem?

Backgrounds and humans require different priors: backgrounds need dense viewpoint coverage to fill unobserved regions, whereas humans require geometric constraints (like SMPL) to maintain shape and articulation despite partial observation.

How does the method handle the flickering typically associated with diffusion-based video enhancement?

The pipeline uses motion-adaptive consistency injection, which blends warped outputs from previous frames into the current input using an exponential moving average, ensuring that ambiguous regions are resolved consistently over time.

The Challenge of Sparse-View 4D Reconstruction

We expose why sparse cameras break 4D reconstruction and outline our decoupled, enhancement-driven solution.

Sparse, low‑overlap cameras leave large portions of a scene unobserved, causing geometric ambiguity and visible artifacts in 4D reconstructions. Existing dense‑camera pipelines collapse under these conditions, and even recent sparse‑view methods still entangle background and human errors.

When cameras are few and viewpoints barely overlap, the background and the moving people each need a different kind of prior to be reconstructed reliably.

**Fig. 1.** Given only as few as four sparse, low-overlap input videos (left), StudioRecon first reconstructs decoupled Gaussians for background and humans (right). The reconstructed Gaussians enable rendering from novel viewpoints, and our recursive enhancement module further refines the rendered output (bottom).

Sparse camera inputs cause geometric ambiguity and artifacts in 4D reconstruction.

Prior Approaches to 4D Reconstruction

Related work spans 4D reconstruction, human‑scene modeling, and diffusion‑based video enhancement.

Prior work on 4D scene reconstruction, human‑scene modeling, and diffusion‑based video enhancement establishes the technical landscape this paper builds upon.

Dyn‑3DGS extends static 3D Gaussian Splatting by attaching a time dimension to each Gaussian, enabling fast rendering of dynamic geometry.

MonoFusion fuses depth and appearance cues from a single moving camera to reconstruct a scene without requiring multiple viewpoints.

These works collectively illustrate the trade‑off between dynamic capability, view sparsity robustness, and the adoption of diffusion priors.

StudioRecon Pipeline Overview

We detail StudioRecon’s four‑stage pipeline that decouples background and human reconstruction and refines them recursively.

Sparse, low‑overlap camera setups leave large blind spots, making it hard to recover both static background and moving people. StudioRecon tackles this by first densifying views, then estimating poses, decoupling background and human Gaussians, and finally refining everything with a recursive enhancement step.

**Fig. 2.** Overview of the proposed StudioRecon. Our pipeline consists of four stages: (1) sparse-to-dense view synthesis using camera-controlled video diffusion, (2) multi-view human pose estimation, (3) decoupled Gaussian reconstruction for background and humans, and (4) a recursive enhancement module.

Instead of forcing a single volumetric model to explain both static scenery and moving people, we allocate separate 3D Gaussian sets—one for the background, one for each human—so each can be supervised by the most reliable source.

How does this decoupling differ from simply masking out humans during a single‑model reconstruction?

Masking treats humans as missing data, leaving the background model to hallucinate those regions. Decoupling maintains an explicit human Gaussian representation, so the model learns a true geometry for each actor instead of filling gaps with background priors.

The module repeatedly refines the merged background‑human Gaussian field, using the current camera view to spot and correct residual artifacts that persist after the separate optimizations.

Render the merged field → composite image shows a 0.12 intensity mismatch at the hand region.

Compute residual map → non‑zero values concentrated around the hand.

Back‑propagate residual into H₁’s position and covariance → hand shifts 2 mm toward the correct location.

Update B₁ and B₂ similarly where background bleed‑through occurs.

Second recursion renders a cleaner image; residual drops to 0.03.

The recursive loop fixes view‑specific errors that static optimization cannot address, and each iteration yields diminishing residuals until convergence.

Why not simply run a single global optimization after merging the Gaussians instead of recursing?

A single pass cannot exploit the per‑view error signal; recursion lets the module adapt the Gaussian parameters to the actual camera trajectory, correcting artifacts that only appear from certain viewpoints.

Multi-View Human Pose Estimation

Estimating consistent 3D human pose across disjoint camera views.

Appearance‑based matching collapses when cameras capture largely non‑overlapping viewpoints, because the same person can look dramatically different across views.

Instead of comparing colors or textures, we lift each detection into 3‑D using calibrated geometry and match people by where they are and how their body pose looks.

Spatial distance $\|p_a-p_b\| = \sqrt{(0.2)^2+(0.1)^2+(0.1)^2}\approx0.245$ m.

Spatial term $=0.9\cdot\exp(-0.245/0.3)\approx0.9\cdot0.44=0.396$.

Pose distance $\|\theta_a-\theta_b\| = \sqrt{(0.02)^2+(0.02)^2+(0.02)^2}\approx0.035$.

Pose term $=0.1\cdot\exp(-0.035/0.05)\approx0.1\cdot0.50=0.050$.

Affinity $A(a,b)=0.396+0.050=0.446$ — a high score indicating a confident match.

The spatial cue dominates because cameras are widely separated; the pose cue only nudges the decision when two people are spatially close.

How does this differ from the usual appearance‑based matching used in multi‑camera setups?

Appearance matching relies on color or texture similarity, which breaks when the same person looks different from opposite sides. Our method ignores appearance entirely; it matches detections by their reconstructed 3‑D location and by the high‑dimensional SMPL pose vector, which is invariant to viewpoint.

Run a monocular pose estimator in each view to obtain SMPL pose $\theta$, shape $\beta$, 2‑D keypoints, and within‑view track IDs.

Unproject each detection’s pelvis keypoint to a 3‑D world position $p$ using calibrated camera intrinsics and extrinsics.

Compute the affinity $A(a,b)$ for every cross‑view pair and solve the assignment problem with the Hungarian algorithm, yielding global person IDs.

Triangulate matched 2‑D keypoints across all cameras using a robust Huber loss to obtain 3‑D joint and vertex positions.

Fit SMPL parameters $(\theta,\beta)$ to the triangulated points by minimizing the weighted 3‑D‑to‑3‑D loss (Equation 4) and apply exponential smoothing for temporal consistency.

**Fig. S2.** Cross-view identity association. Each color represents the same person matched across 4 cameras with low overlap. Our method achieves 97.8% association accuracy across all 8 scenes from EgoHumans, Harmony4D, Mobile Stage, and SelfCap (Table 3).

**Fig. S3.** Initialized human pose visualization across four scenes. Point clouds are rendered with overlaid SMPL meshes, demonstrating accurate body pose estimation from sparse multi-view inputs.

Decoupled Gaussian Reconstruction

Decoupled reconstruction separates static background and dynamic human Gaussians for tailored optimization.

Static backgrounds demand dense viewpoint coverage at a single time, whereas dynamic humans need temporal supervision but can rely on strong SMPL priors despite sparse observations.

We treat the scene as two independent Gaussian fields: one dense static background and one set of per‑person Gaussians that move with the pose.

Compute photometric residuals for all 25 pixels; only the 16 background pixels (outside M) contribute to L₁.

Apply SSIM and LPIPS only inside M, scaling them by $\lambda_{s}$ and $\lambda_{l}$.

Add density penalty $\lambda_{d}$·$L_{\text{den}$} for any Gaussian whose opacity exceeds 0.9.

Sum the three components to obtain a scalar loss ≈ 0.42.

The decoupled loss lets background Gaussians improve using all available views while the human region is handled separately, preventing cross‑talk.

How does this differ from jointly optimizing a single Gaussian field for both background and humans?

Joint optimization forces the same loss weights on static and moving regions, causing background Gaussians to chase moving silhouettes and human Gaussians to receive sparse supervision. Decoupling isolates the two objectives, so each field can use the supervision it actually has.

Background reconstruction starts from a point cloud and the novel views generated earlier; we initialize Gaussians at timestep t = 1 and mask out human pixels during optimization.

Iterative refinement of undersampled background regions using the recursive enhancement module.

**Fig. 9.** Effect of iterative refinement on background reconstruction (Sec. 3.3) Before refinement (left), undersampled regions appear blurry. After refinement (right), clarity improves.

Human reconstruction leverages SMPL parameters to place Gaussians on a canonical pose, then deforms them per frame with linear blend skinning.

Each canonical Gaussian is attached to bones; at any frame we blend its position with the bone transforms weighted by pre‑computed skinning coefficients.

Compute torso transform G₁^t (30° Y‑rotation, 0.5 m forward).

Compute arm transform G₂^t (–45° X‑rotation, 0.2 m right).

Apply weighted sum: $\mu_{t}$ = 0.7·G₁^t·$\mu_{c}$ + 0.3·G₂^t·$\mu_{c}$.

Resulting position $\mu_{t}$ ≈ (0.06, 0, 0.5) after matrix multiplication.

The blend respects both bone influences, so the Gaussian follows the torso’s translation while slightly offset toward the arm’s rotation.

Why can’t we simply attach each Gaussian to a single bone instead of blending?

Single‑bone attachment would cause visible seams at joint boundaries because the Gaussian would jump abruptly when the underlying bone changes; blending yields smooth transitions and preserves the continuous surface of the human mesh.

Recursive Enhancement Module

We smooth frame‑wise diffusion by blending motion‑aware priors from nearby outputs.

Even after decoupled reconstruction, the rendered Gaussians still show floaters, blurry patches in sparsely observed regions, and occasional human‑pose artifacts, because the static background and moving actors are optimized separately.

Compute the weighted sum of warped past outputs: $S = 0.6\!\times\!0.9\!\times\!O_{t-1} + 0.3\!\times\!0.7\!\times\!O_{t-2} + 0.1\!\times\!0.4\!\times\!O_{t-3}$.

Average over $K$: $\frac{1}{3}S$.

Blend with the current composite: $\tilde{I}_t = 0.5\,\hat{I}_t + 0.5\,\frac{1}{3}S$.

Feed $\tilde{I}_t$ to the diffusion model, which outputs the refined frame $O_t$.

Even with a modest injection strength, the EMA gives recent frames dominant influence while still borrowing stability from older frames, and the confidence masks automatically mute contributions where warping fails.

Why does feeding each frame independently to the diffusion model cause flickering?

When the model processes frames in isolation, it treats the noise in each frame as unrelated, so temporal consistency is lost; the same scene point can receive different stochastic refinements across time, producing visible jitter.

**Fig. 3.** Overview of our recursive enhancement module (Sec. 3.4).

Experimental Setup

Ours tops PSNR on 180° scenes, beating the runner‑up by 0.64 dB.

On the SelfCap (frontal) scene our method reaches a PSNR of 20.14 dB, beating the next‑best STG by 0.64 dB.

Table 2 reports PSNR = 20.14 for Ours versus 19.50 for STG.

We evaluate four methods—Dyn‑3DGS, MonoFusion, STG, and our approach—on two camera setups: full 360° coverage (EgoHumans, Harmony4D) and frontal‑only 180° coverage (Mobile Stage, SelfCap). For each dataset we train with four evenly spaced cameras and report PSNR, SSIM, and LPIPS.

**Table 2.** Quantitative comparison on 180° scenes. red: best, yellow: second-best.

On the 360° benchmark (Table 1) our approach also leads in PSNR for most scenes, matching Dyn‑3DGS on Mobile Stage and surpassing MonoFusion and STG across the board.

Quantitative and Qualitative Results

Our method outperforms baselines across all quantitative and visual metrics.

Recall that StudioRecon decouples static backgrounds from actors and refines geometry with a recursive enhancement module.

Our approach improves PSNR by 5.0 dB over the strongest baseline.

Ours 21.74 dB vs. best baseline 16.73 dB (Dyn‑3DGS) on the Mobile Stage Dance scene.

**Fig. 5.** Qualitative comparison on 360° scenes (Legoassemble, Grappling, Sword, Karate). Our method produces sharper backgrounds and more robust human reconstructions than baselines.

**Fig. 6.** Additional qualitative comparison (Tennis, Fencing, Dance, Yoga) from EgoHumans, Mobile Stage, and SelfCap. Our method produces sharper reconstructions with better human-scene separation. Dance © Xu et al. (Mobile Stage); Yoga © Xu et al. (SelfCap), used with permission.

Baseline methods suffer from missing viewpoints, leading to background artifacts and unstable human reconstructions; our dense view synthesis and SMPL‑guided modeling eliminate these issues.

Ablation Study

We isolate each design choice to quantify its impact on reconstruction quality.

This section systematically removes each component of StudioRecon to reveal how much reconstruction quality degrades.

Adding the hybrid cross‑view identity association (+ViewSynth) lifts PSNR by +2.41 dB over the 4‑view baseline.

Baseline PSNR = 18.13 dB; with ViewSynth PSNR = 20.54 dB (Table 3).

Appending our recursive enhancement (+Enhance) further reduces LPIPS by ‑0.535, confirming that the enhancement cleans geometric artifacts.

LPIPS drops from 0.654 (without enhancement) to 0.119 (with enhancement) in the motion‑adaptive injection ablation.

Motion‑adaptive consistency injection cuts Warp‑L2 error by ‑0.451, demonstrating tighter temporal alignment.

Warp‑L2 falls from 0.649 to 0.198 when injection is enabled (Tables 1, 2).

Relying solely on spatial proximity yields 93.3 % correct associations, while pose similarity alone drops to 81.4 %.

Manual annotation comparison shows these failure rates on the 8‑scene benchmark.

Combining both cues achieves 97.8 % accuracy with zero false positives, confirming the hybrid design’s robustness.

Hybrid cue performance reported on the same 8‑scene set.

Dense view synthesis via video diffusion contributes the largest gain: +2.4 dB PSNR and a 36 % LPIPS reduction.

Table 4 shows the dense synthesis ablation improves PSNR from 18.0 dB to 20.4 dB and cuts LPIPS by 0.36.

The single‑step diffusion enhancement adds a further 27 % LPIPS reduction, albeit with a slight PSNR/SSIM trade‑off.

Enhancement lowers LPIPS from 0.36 to 0.26 while PSNR drops marginally (Table 4).

**Fig. 7.** Ablation on recursive enhancement (Sec. 3.4). Raw renders (left) contain blur and geometric instabilities. Enhancement (right) produces clean, harmonized outputs.

Motion-Adaptive Consistency Injection

We quantify how each component affects visual fidelity and robustness.

We warp previously enhanced frames into the current view using optical flow, then blend them with the current input via a per‑pixel confidence‑weighted exponential moving average, forcing ambiguous regions to be resolved consistently over time.

How does this differ from a simple temporal filter applied to the enhanced frames?

Unlike a naïve filter that averages misaligned frames, our injection first aligns past outputs with optical flow and then weights each pixel by a confidence estimate, so only trustworthy, correctly warped information contributes to the current frame.

**Fig. 4.** Schematic of motion-adaptive consistency injection. For each frame, RAFT computes backward flow to warp previous enhanced outputs, which are blended with the current input via per-pixel confidence-weighted EMA before single-step diffusion.

Motion‑adaptive consistency injection cuts warp‑L2 error by 23 % compared to the baseline without injection.

Table 5 reports a 23 % reduction in the warp‑L2 metric when the injection is enabled.

**Fig. 10.** Ablation on motion-adaptive consistency injection (Sec. 3.4). Without injection (top), enhanced frames exhibit flickering. With injection (bottom), consistency improves.

Applications and Conclusion

StudioRecon enables novel camera paths, actor swaps, and high‑fidelity 4D capture.

The Gaussian representation models the scene as a cloud of 3‑D Gaussians that can be rendered from any viewpoint. Our representation therefore permits novel camera trajectories; we demonstrate dolly‑zoom and oscillating paths by simply defining the desired camera motion.

SMPL is a parametric human body model that provides pose and shape parameters. Because backgrounds and humans are reconstructed independently, we can replace an actor with a new identity—generated from a single reference image—while preserving the original pose and scene geometry.

**Fig. 8.** Qualitative results on EgoExo-4D [Grauman et al. 2024]. Our method applies to diverse activities and environments.

**Fig. 12.** Applications. Top: human replacement with a different identity (right) while keeping the background (left). Bottom: novel trajectory rendering with oscillating motion (left) and dolly zoom (right).

Limitations remain: fine facial and hand details are lost under sparse views, SMPL‑based actors cannot capture external props, and static shadows baked into the background do not follow moving subjects.

Future work will integrate high‑frequency detail models, handle hand‑held objects, and synthesize dynamic shadows to close the remaining gaps.

Supplementary Material and Limitations

Appendices detail pipelines, optimization settings, runtime analysis, and limitations of StudioRecon.

The appendix expands on every pipeline stage, providing exact hyper‑parameters, loss compositions, and additional qualitative analyses that support the main paper.

Preprocessing (Appendix A) uses four cameras ($N\!=\!4$), segments actors with SAM3 (“person”), estimates depth via Pi3 (confidence 0.1, 21‑pixel dilation), and synthesizes 481 novel views ($L\!=\!481$) at 1280 × 704 using GEN3C; rotations are interpolated with SLERP and translations linearly.

Human pose estimation (Appendix B) runs CoMotion per view to obtain SMPL pose, shape, and 2D keypoints, then links identities across views with a Hungarian algorithm (weights $w_p\!=\!0.9$, $w_\theta\!=\!0.1$, distance 0.3 m). World coordinates follow Equation S1, and SMPL parameters are refined over 30 iterations ($\beta$ = 10.0) with bone‑length alignment (L‑BFGS 100 iters) and temporal smoothing (EMA $\alpha\!=\!0.5$, Gaussian $\sigma\!=\!1$).

Background reconstruction (Appendix C) optimizes Gaussian splats for 7 k iterations on the $L$ synthesized views, using spherical harmonics of degree 3. The loss mixes L1 (0.7), SSIM (0.3), LPIPS (0.1) and density regularization (0.1). Position learning rate decays from $1.6\!\times\!10^{-4}$ to $1.6\!\times\!10^{-6}$, densification occurs every 500 iters up to 3 k (gradient threshold 0.0002), and ground‑truth views receive up to 3× higher weight via Equation S2. At iteration 3 500 we add 240 perturbed views (±15 % height) refined by Difix3D+ with a reduced loss weight of 0.7.

Human reconstruction (Appendix D) runs for 10 k iterations on the four input videos, employing spherical harmonics of degree 2. The composite loss (Equation S3) balances photometric terms (L1 0.8, SSIM 0.2), perceptual LPIPS 0.1, mask supervision 0.01, density regularization 0.1, and a strong pose regularizer 50.0. Position learning rate decays from $1.0\!\times\!10^{-3}$ to $2.0\!\times\!10^{-6}$, densification stops at 1 500 iters, and Linear Blend Skinning uses SMPL weights from a 643‑voxel grid. A temporal deformation network activates at iteration 3 000 (8‑layer MLP, 256 hidden units, skip at layer 4) and predicts per‑Gaussian residuals via Equation S4.

The recursive enhancement module (Appendix E) operates at inference time in per‑frame reference mode, applying a single‑step denoising at timestep $t\!=\!200$. Motion‑adaptive EMA injects the three previous frames ($\alpha\!=\!0.3$, decay 0.5); injection strength is gated by per‑pixel RGB error with threshold $\tau_e\!=\!30$, giving full injection to well‑aligned pixels and minimal injection to misaligned regions.

Runtime analysis (Appendix F) reports the per‑stage breakdown (Table S1): view synthesis 50 min, pose estimation 3 min, background reconstruction 30 min, human reconstruction 40 min, and recursive enhancement 3.3 s per frame. View synthesis is the primary bottleneck but scales linearly with GPU count (≈13 min on 8 GPUs). Human reconstruction adds roughly 3 min per extra actor, while other stages remain constant. End‑to‑end processing of a 121‑frame scene with 1–3 actors takes about 130 min on a single A6000 GPU or 60 min with eight GPUs.

Additional results (Appendix G) compare our camera configuration against the sparse‑view protocol of Kong et al. (Table S2). Prior work’s neighboring cameras overlap by 28°, whereas our setup achieves a 70° spread, dramatically increasing occlusion diversity.

Robustness analysis (Appendix H) disables individual components to quantify their impact. Removing multi‑view pose refinement yields blurry or semi‑transparent bodies; omitting temporal smoothing introduces noticeable jitter; skipping mask dilation leaves ghosting artifacts at human boundaries. Each ablation confirms the necessity of the corresponding module for high‑quality reconstructions.

Limitations (Appendix I) stem from the reliance on the SMPL body model: objects held by actors (e.g., a basketball or swords) are not reconstructed, and shadows baked at $t\!=\!0$ remain static as humans move.

**Fig. S7.** Limitations of our method. Comparing ground truth (left) with our reconstruction (right), dynamic objects held by actors are not reconstructed because they lie outside the SMPL body model.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers