WorldSculpt: Generating Compositional Worlds from Grounded Videos

Muyao Niu, Jixuan He, Ruihan Yu, Lian Fu, Yonghao Yu, Zheng-Hui Huang, Yifan Zhan, Fengbo Lan, Yongtao Ge, Yinqiang Zheng, Kaipeng Zhang, Zhixiang Wang

WorldSculpt generates compositional 3D scenes by conditioning a single-object generative prior on multi-view observations.

How can we generate high-fidelity, compositional 3D meshes for hundreds of objects in a cluttered scene from only a set of posed RGB images and coarse instance masks?

Generative world models can synthesize photorealistic 3D environments, but they typically output a monolithic "soup" of geometry that cannot be edited, moved, or used in physics simulators. WorldSculpt decomposes these scenes by mapping multi-view observations into an anchor-aligned canonical frame, where a frozen single-object generative prior is conditioned on the aggregated features. This approach enables the generation of complex scenes with hundreds of objects without scene-level training, consistently outperforming prior compositional methods in cluttered environments.

Paper Primer

The core challenge is that existing generative models either produce monolithic scene representations or are limited to simple, isolated objects. WorldSculpt bridges this by treating each object as an independent asset: it lifts features from multiple posed views into a shared canonical voxel volume, aggregates them using a permutation-invariant module, and injects this evidence into a pretrained 3D generative prior via low-rank adapters (LoRA).

WorldSculpt enables high-fidelity compositional scene generation in densely cluttered environments without requiring scene-level training.

Evaluation on the new UE-MeshyScene benchmark, which contains up to 701 objects per scene with severe mutual occlusion. Outperforms existing compositional baselines across all metrics, with gains increasing as scene complexity and occlusion levels rise.

The method relies on a conditioning-view augmentation curriculum that simulates partial and degraded observations during training. This allows the model to learn robust geometry completion, ensuring that even when an object is heavily occluded in most views, the final mesh remains geometrically plausible and correctly placed in the world frame.

Why use a single-object generative prior instead of training a model on entire scenes?

Training on entire scenes often leads to monolithic, non-editable outputs. By leveraging a strong single-object prior, WorldSculpt ensures each object is represented as an individual, independently usable mesh while still benefiting from the prior's ability to infer unobserved geometry.

What is the role of the anchor-aligned canonical frame?

It provides a consistent, normalized coordinate system for each object, allowing the generative model to operate on a standard cubic domain regardless of the object's actual position, orientation, or scale in the cluttered scene.

Introduction and Motivation

We frame the need for object‑centric 3D scene generation and expose its core challenges.

Generating a scene as a collection of individual object meshes is essential for gaming, AR/VR, simulation, and robotics, yet dense occlusion leaves most existing pipelines with a single fused geometry that cannot be edited per object.

We need a representation that treats each object as an independent mesh while still capturing the whole scene’s geometry, because downstream tools cannot work with an undifferentiated “soup” of surfaces.

**Figure 1.** WorldSculpt. Given a set of RGB images with instance-level 2D masks and 3D bounding boxes, WorldSculpt produces a compositional mesh representation for very complex scenes consisting of hundreds of individual objects.

The image displays the logo for "ALAYA LAB".

The WorldSculpt Pipeline

WorldSculpt builds per‑object canonical cubes and fuses multi‑view features into a 3D prior.

WorldSculpt proceeds in three stages: (1) anchor‑aligned canonicalization, (2) multi‑view conditioned generation, and (3) world‑frame placement of the generated meshes.

We align each object to a virtual cube whose front faces the anchor view, giving a consistent, normalized space for generation.

Compute $T^{ow}_{k}$ using $s_k=1.2$, $R_k$ aligned to view 1, and $c_k=(0,0,0)$ → yields a similarity transform that scales the unit cube to $1.2$ m and rotates it to match view 1.

Project voxel $x=(0,0,0)$ to view 2: apply $T^{ow}_{k}$, then $T^{cw}_{2}$, then $\bar{K}^{2}_{k}$, finally divide by depth → obtain image coordinates $(u,v)$ inside the cropped region.

Check that the projected cube covers the mask $S_{k}^{2}$; if not, increase $s_k$ to $1.4$ and recompute $T^{ow}_{k}$.

After adjustment, the projected cube fully contains the mask in both views, guaranteeing no clipping.

Scale adaptation preserves object proportions while guaranteeing that every view’s crop sees the entire object, eliminating the need for per‑view manual padding.

How does Anchor‑Aligned Canonicalization differ from simply using the coarse localization box $B^{\text{loc}}_{k}$ as the generation volume?

The coarse box may be non‑cubic and arbitrarily oriented, which would force the generative prior to operate on an irregular, anisotropic volume. By constructing an isotropic cube aligned to the anchor view we (1) provide a shape the prior was trained on, (2) remove orientation ambiguity, and (3) ensure a single consistent frame across all observations.

We lift 2‑D DINOv3 features from each view into the canonical voxel grid and merge them with a permutation‑invariant aggregator, so the generator sees a unified 3‑D condition.

Compute the mean: $\mu(x) = ((1+3)/2,\,(2+4)/2) = (2,3)$.

Compute variance: $\sigma^{2}(x) = ((1^2+3^2)/2 - 2^2,\,(2^2+4^2)/2 - 3^2) = (4-4,\;10-9) = (0,1)$.

Assume $\text{MLP}_{\text{feat}}$ is identity and $\text{MLP}_{\text{weight}}$ outputs logits $w_1=0.2$, $w_2=0.8$.

Softmax weights: $\alpha_1 = e^{0.2}/(e^{0.2}+e^{0.8})\approx0.37$, $\alpha_2\approx0.63$.

Aggregate: $g_{\text{out}}(x)=\mu + \alpha_1 g'_1 + \alpha_2 g'_2 = (2,3) + 0.37(1,2) + 0.63(3,4) = (2+0.37+1.89,\;3+0.74+2.52) = (4.26,\,6.26)$.

The softmax weighting lets the network rely more on the view that provides clearer features (here view 2), while the mean term preserves a robust baseline.

Why not simply average the lifted features instead of using the learned weighting scheme?

Simple averaging treats all views as equally reliable, which is false when some views are occluded or noisy. The learned logits allow the network to down‑weight unreliable views and up‑weight informative ones, while the mean term still guarantees a stable fallback.

Training samples random anchor views and a variable number of conditioning views; the multi‑view aggregator, LoRA adapters, and injection layers are optimized while the pretrained Pixal3D generator remains frozen. A curriculum of occlusion, pose perturbation, mask error, and downsampling gradually hardens the model against real‑world clutter.

**Figure 2. Overview of WorldSculpt.** Given posed scene images, instance masks, and coarse 3D object localizations, the anchor-aligned canonicalization stage constructs a virtual canonical cube for each object and derives crop-aware projections from its input views. Per-view DINOv3 features are lifted into the canonical voxel volume and fused with a permutation-invariant aggregator. The resulting 3D condition, together with global tokens from the anchor view, drives the two geometry stages of a Pixal3D prior through zero-initialized injection layers and LoRa adaptation. The generated canonical meshes are finally placed into the scene using their canonical-to-world transformations.

Inference and Scene Composition

Describes how each object is inferred and assembled into a full scene.

At inference we treat each object independently, first anchoring it in the view that shows it most completely, then building a canonical cube before generating and placing its mesh.

Select the anchor view where the object has the largest visible region.

Construct the anchor‑aligned canonical cube around the object as described in Section 3.1.

If the cube does not fully cover the object masks in all candidate views, increase its side length while keeping its center and orientation fixed.

Choose up to a fixed number of conditioning views, prioritizing those with larger visible object areas.

For each conditioning view, generate a masked crop and a crop‑aware projection function.

Run the sparse‑structure stage of Pixal3D to predict coarse occupied voxels, defining the sparse support for the next stage.

Feed the sparse support into the shape stage, which decodes a latent representation into the canonical‑space mesh $M^{c}_{k}$.

Transform the canonical mesh into world coordinates using the canonical‑to‑world mapping $T^{w}_{k}$ (Eq. 4), yielding $M^{w}_{k}$.

Aggregate all transformed meshes $M^{w}_{k}$ into the final compositional scene $M^{w}$.

Step 1: Initial cube $C_{0}$ spans $[-1,1]$ in each axis.

Step 2: Detect that the mask exceeds $C_{0}$’s $x$‑extent (needs $[-2.5,2.5]$).

Step 3: Increase side length to $3$ units while keeping the center at $(0,0,0)$, yielding cube $C_{1}$ spanning $[-1.5,1.5]$.

Step 4: Verify that $C_{1}$ now fully encloses the mask ($[-2.5,2.5]$ is still outside, so a second increase to $5$ units would be required; for this toy example we stop at $3$ to illustrate the expansion operation).

Expanding the cube’s side length preserves the anchor view’s orientation and center, ensuring that subsequent projection functions remain aligned with the original observation.

By processing objects independently and then stitching their canonical meshes into the shared world frame, the method achieves scalable, high‑fidelity scene composition.

Experimental Setup

WorldSculpt’s scalability is demonstrated on scenes with hundreds of objects.

WorldSculpt scales to scenes with up to 678 objects and 1,268 views per scene.

Table 1 shows the Office scene contains the maximum 678 objects and 1,268 rendered views among the six UE‑MeshyScene scenes.

We evaluate the method under three increasingly challenging settings: (i) canonical‑space generation of single objects on the Toys4k dataset, isolating geometry recovery from partial multi‑view inputs; (ii) compositional scene generation on synthetic Toys4k‑Scene and real‑world HouseCat6D, testing multi‑object synthesis; (iii) large‑scale compositional generation on UE‑MeshyScene, where scenes contain hundreds of objects under severe mutual occlusion.

The UE-MeshyScene Benchmark

UE‑MeshyScene offers up to 701 objects per scene, enabling large‑scale compositional evaluation.

UE‑MeshyScene contains up to 701 objects in a single scene, far exceeding prior 3‑D scene benchmarks.

Table 1 reports the object count per scene, with the maximum of 701 objects in the Desert Town scene.

A six‑scene collection of photorealistic Unreal Engine environments, each densely populated with objects and fully annotated for per‑object evaluation.

**Table 1.** Per-scene statistics of UE-MeshyScene. Every scene is rendered in Unreal Engine 5.8 at 2560 × 1440; “#Views” denotes the number of rendered frames.

We evaluate generation quality using Chamfer Distance (both ℓ₂ and ℓ₁), Earth Mover’s Distance, and F‑Score. All metrics are computed after per‑instance ICP alignment to remove canonical‑frame discrepancies.

Single-Object and Scene Generation Results

Multi-object scene generation demonstrates superior compositional quality and robustness across synthetic and real datasets.

We assess compositional scene generation on both synthetic and real‑world multi‑object scenes. Each predicted object is placed into the shared world frame and evaluated against its ground‑truth mesh.

Pixal3D is a single‑view pipeline that reconstructs an object mesh from one RGB image using a learned shape prior.

We compare our method against SAM3D, SceneGen, SceneMaker, RecGen, and ShapeR, providing depth maps to those that require them. All methods are evaluated with the same UE‑MeshyScene protocol.

**Table 3.** Compositional scene generation on the real-world HouseCat6D [34] dataset and synthetic Toys4k [86]-Scene benchmark. We report CD-$\ell_2$ ($\times 10^{-3}$, $\downarrow$), CD-$\ell_1$ ($\times 10^{-2}$, $\downarrow$), EMD ($\times 10^{-2}$, $\downarrow$), and F-Score ($\uparrow$). The best result in each column is shown in bold.

**Figure 4.** Qualitative comparison on Toys4k-Scene. Given the input observations, each method generates per-object meshes, whose surface normals are rendered in the shared world frame. Our method produces more detailed and accurately registered object geometry, closely matching the ground truth in the rightmost column. For each example, we enlarge the boxed region and visualize its pixelwise discrepancy with the ground truth in the row below. On overlapping surfaces, color denotes the angular error between surface normals, ranging from blue at $0^\circ$ to red at $90^\circ$. Non-overlapping regions are shown separately: pink indicates spurious predicted surfaces, while gray indicates missing ground-truth surfaces. These regions also contribute to the mask-IoU error.

**Figure 5.** Qualitative comparison on HouseCat6D. Our method recovers detailed object geometry that is accurately placed in the world frame and closely matches the scanned ground truth shown in the rightmost column, while the baselines often exhibit incomplete or distorted shapes and inaccurate placement. As in Fig. 4, each example includes an enlarged region of interest and a pixelwise discrepancy map with respect to the ground truth in the row below. On overlapping surfaces, color indicates the angular error between surface normals, ranging from blue at 0° to red at 90°. Non-overlapping regions are shown separately: pink denotes spurious predicted surfaces, while gray denotes missing ground-truth surfaces. Both contribute to the mask IoU error.

Performance on Cluttered Scenes

Our method beats ShapeR on UE‑MeshyScene across all metrics.

Our method achieves a $+3.09$ higher $F\text{-Score}$ than ShapeR on UE‑MeshyScene ($\tau$ = 0.05).

Table 4 reports $F\text{-Score}=3.7$ for our method versus $0.61$ for ShapeR.

ShapeR is a multi‑view baseline that aggregates per‑view feature grids by simple arithmetic‑mean fusion before reconstructing object meshes.

How does ShapeR differ from our IBR aggregator?

ShapeR averages aligned feature grids, treating every view equally. Our IBR aggregator learns to weight each view adaptively, allowing it to emphasize views that see an object more clearly and suppress noisy or occluded observations.

**Figure 6.** Qualitative comparison on UE-MeshyScene. We compare per-object meshes generated in densely cluttered and heavily occluded scenes against the multi-view baseline ShapeR [83]. All predicted meshes are placed in the shared world frame and visualized using surface normals. Our method recovers more detailed and accurately placed geometry, including for small and severely occluded objects, while the baseline often produces over-smooth or misplaced shapes in highly cluttered regions. These qualitative differences are consistent with the quantitative results reported in Table 4.

**Table 4.** Compositional generation on six scenes of the UE-MeshyScene dataset. Protocol and metrics as in Table 3 (F-Score at $\tau=0.05$), additionally reporting per-instance medians (CD$^{med}$, same scale as the corresponding mean). Best per column in **bold**.

Table 5 evaluates fusion strategies on the Toys4k canonical‑space benchmark; the learned IBR aggregator consistently outperforms simple averaging, especially under higher occlusion levels.

Table 6 compares scene‑level fusion on UE‑MeshyScene; the IBR aggregator yields a higher $F\text{-Score}$ (3.7 vs 0.61) despite a modest increase in Chamfer distance.

Ablations and Applications

We evaluate the learned IBR aggregator versus simple averaging and apply WorldSculpt to generated 3DGS worlds.

We ablate the multi‑view fusion module by swapping the learned IBR aggregator for a simple arithmetic mean (“Avg”) while keeping all other components identical.

The learned aggregator assigns view‑dependent weights to each voxel, allowing the network to emphasize reliable observations and suppress noisy ones.

On Toys4k with 2 views and 75 % occlusion, the IBR aggregator reduces CD‑ℓ₂ from 12.91 to 10.03 and raises F@.05 from 0.742 to 0.762.

Table 5, column “2 views, 75 % occlusion”.

Across the UE‑MeshyScene benchmark, the IBR aggregator cuts CD‑ℓ₂ by 12 % and lifts the F‑Score from 0.944 to 0.951.

Table 6, UE‑MeshyScene rows.

**Figure 7.** Application to a Marble-generated 3DGS world. Given a generated 3DGS scene, we render a sequence of posed observations and apply WorldSculpt to convert the scene into a compositional representation of individual per-object meshes in a shared world frame.

Render a sequence of RGB frames and metric depth maps from the 3DGS using the known virtual camera intrinsics and poses.

Run SAM3 on the RGB sequence to obtain per‑object mask tracks; merge fragmented tracks using temporal overlap and 3D bounding‑box consistency.

Back‑project valid masked pixels through the camera parameters to build partial point clouds; cluster with DBSCAN and keep the dominant cluster as the coarse object localization.

Feed the posed RGB frames, mask tracks, and bounding‑box localizations into the standard WorldSculpt inference pipeline to generate individual per‑object meshes in the shared world frame.

Related Work

We position WorldSculpt within the landscape of image-to-3D and scene generation research.

Feed-forward 3D reconstruction methods predict geometry from one or few images using large transformer architectures. Notable examples include LRM and its mesh- and Gaussian-based variants such as MeshLRM, InstantMesh, GRM, LGM, TripoSR, and SF3D.

Another family transfers 2D diffusion priors to 3D by synthesizing novel views before fusion. Methods like Zero-1-to-3, SyncDreamer, Wonder3D, CRM, and One-2-3-45 generate consistent multi-view observations that are later combined into a 3D shape.

Native-3D generative models learn distributions directly in 3D representations, covering point clouds (Point-E), implicit functions (Shap-E), aligned shape-image-text latent spaces (Michelangelo), and high-resolution structured representations such as TRELLIS, Hunyuan3D, and Pixal3D. These models are commonly trained on large-scale asset datasets like Objaverse and Objaverse-XL.

Amodal 3D reconstruction aims to recover complete object geometry from occluded views. Early work used physical constraints in robotic settings, while recent approaches such as Amodal3R and AmodalGen3D adapt native-3D generative priors to directly recover full geometry from partially occluded images.

Object-level 3D datasets provide ground-truth meshes for training and evaluation, ranging from synthetic collections (Objaverse, Objaverse-XL, Toys4k) to scanned assets (Google Scanned Objects, ABO) and real captures with point clouds (CO3D, MVImgNet). However, datasets offering accurate scanned meshes for in‑the‑wild photographs remain scarce.

Scene-level and compositional generation methods jointly recover object geometry and layout, with holistic image‑based approaches such as InstPIFu, PartCrafter, MIDI, SAM3D, SceneGen, SceneMaker, and RecGen evaluated on synthetic suites like 3D‑FRONT, 3D‑FUTURE, and Hypersim, while real‑world datasets like ScanNet, ScanNet++, Matterport3D, MultiScan, and Replica provide incomplete scene meshes. Additional resources include object‑level annotations (ARKitScenes, SUN RGB‑D, Objectron) and aligned CAD models (Scan2CAD, ROCA, CAD‑Estate).

Reconstructing objects from multiple images relies on camera pose estimation via structure‑from‑motion or deep visual SLAM, dense scene structure from monocular geometry estimators, and recent feed‑forward pointmap models that predict both poses and geometry from unposed image sets. Complementary techniques such as promptable video segmentation, open‑vocabulary detectors, and object‑level SLAM further refine object masks and 3D localizations, while classical silhouette‑based geometry can be applied when poses are known.

Limitations and Conclusion

We discuss remaining challenges of WorldSculpt and recap its contributions.

WorldSculpt decomposes cluttered scenes into individual object meshes by canonicalizing each object into an anchor‑aligned cube, enabling independent generation and precise placement in a shared world frame.

First, the method depends on upstream estimates of camera poses, per‑object instance masks, and coarse 3D localization boxes; while the conditioning‑view augmentation curriculum (Sec 3.3) mitigates moderate errors, large pose or mask inaccuracies still corrupt the generated geometry.

Second, WorldSculpt currently produces geometry only and does not model object appearance; extending the multi‑view conditioning pathway to the texture and material stages of Pixal3D or TRELLIS.2 would enable joint geometry‑appearance synthesis.

Third, the formulation assumes a static scene—each object’s multi‑view observations are treated as measurements of a fixed geometry and pose—so moving or deforming objects are unsupported; incorporating optical‑flow‑based motion estimation or articulated representations such as skeletal models or SMPL‑X could address this limitation.

In conclusion, WorldSculpt adapts a strong object‑level generative prior with a multi‑view conditioning pathway anchored in a canonical frame, achieving state‑of‑the‑art results on the UE‑MeshyScene benchmark; future work will broaden the pipeline to texture/material generation and to dynamic, deformable scenes.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers