PIXWORLD: Unifying 3D Scene Generation and Reconstruction in Pixel Space

Sensen Gao, Zhaoqing Wang, Qihang Cao, Dongdong Yu, Changhu Wang, Jia-Wang Bian

PixWorld unifies 3D scene generation and reconstruction by diffusing directly in pixel space, bypassing latent autoencoders.

How can we unify 3D scene reconstruction and generation into a single pixel-space diffusion model without relying on latent-space compression?

3D scene generation and reconstruction are typically treated as separate tasks, with generation relying on latent-space diffusion and reconstruction on feed-forward regression. This separation forces models to use intermediate latent representations, which introduce information loss and require expensive, pre-trained autoencoders.

Paper Primer

The core mechanism hinges on partitioning multi-view inputs into clean and noisy subsets: clean views provide the ground truth for reconstruction, while noisy views are generated conditioned on the clean ones. The model uses a two-stream diffusion transformer to decode these inputs into a pixel-aligned 3D Gaussian representation, which is then rendered back to images for direct pixel-space supervision.

To ensure geometric integrity, the authors introduce a geometry perception loss: it acts like a structural critic, comparing rendered views against ground truth in the feature space of a pre-trained 3D foundation model to prevent the "floaters" or depth-drift common in purely photometric 2D supervision.

PixWorld achieves superior 3D scene generation quality and reconstruction fidelity compared to latent-space baselines.

On the RealEstate10K dataset, PixWorld improves PSNR by +1.06 dB in single-image generation and consistently lowers LPIPS scores across all tested reconstruction settings.

Why move from latent space to pixel space for 3D modeling?

Latent-space methods suffer from information loss due to the autoencoder bottleneck and decouple the diffusion objective from the final 3D output. Pixel-space diffusion keeps the generative target in the same domain as the rendered output, allowing for direct supervision of the 3D representation.

How does the model handle the lack of 3D supervision in standard image-level losses?

The authors introduce a geometry perception loss that uses a frozen 3D foundation model to compare rendered and ground-truth views in a geometry-aware feature space, forcing the model to learn consistent 3D structure rather than just 2D appearance.

PixWorld demonstrates that pixel-space diffusion is a viable, unified paradigm for 3D modeling, effectively removing the need for intermediate latent autoencoders while improving both generation and reconstruction performance.

Unifying 3D Scene Modeling

We expose the split between 3D reconstruction and generation and motivate a pixel‑space unified solution.

3D reconstruction and 3D generation have evolved along parallel tracks: reconstruction regresses geometry from multi‑view images, while generation synthesizes scenes via latent‑space diffusion. This split forces separate models, datasets, and training pipelines, preventing the two tasks from benefiting each other.

Research treats reconstruction and generation as distinct problems, so advances in one do not automatically improve the other.

Recent attempts to unify the tasks operate in latent space, defining the diffusion objective on intermediate features rather than on the underlying 3D structure. This introduces two major problems: (1) the diffusion signal no longer directly optimizes the 3D output, and (2) the latent encoder (a VAE or RAE) discards information and adds a costly pre‑training step.

We therefore reformulate the problem as a pixel‑space diffusion task and present PixWorld, a single model that jointly performs reconstruction and generation. By supervising diffusion directly on rendered images, the training signal aligns with the fidelity of the final 3D scene, eliminating the need for any latent encoder.

PixWorld exposes both tasks to the same forward pass by partitioning multi‑view inputs: clean views are used for reconstruction, while noisy views are generated conditioned on the clean ones. Both streams produce a pixel‑aligned 3D Gaussian representation.

To compensate for the lack of explicit 3D supervision in pure image‑level losses, we add a geometry perception loss. This loss measures similarity between rendered views and ground truth in the geometry‑aware feature space of a pretrained 3D foundation model, encouraging true 3D structure.

Extensive experiments on RealEstate10K, DL3DV, and WorldScore show that a single PixWorld model matches state‑of‑the‑art reconstruction methods and outperforms prior latent‑space generation approaches, confirming the superiority of the pixel‑space unification.

**Figure 1:** PixWorld unifies 3D scene reconstruction and generation within a single model. Unlike prior approaches that compute losses in the latent space of a VAE (Kingma & Welling, 2013) or RAE (Zheng et al., 2025), PixWorld applies a flow matching objective directly in pixel space over multi-view renderings, enabling end-to-end optimization of the underlying 3D representation. This design avoids the information loss inherent to latent representations and eliminates the cost of pretraining a VAE or RAE.

The key shift is moving from latent‑space pipelines to a unified pixel‑space diffusion framework.

Prior Approaches in 3D Generation

We position PixWorld among prior 3D generation, reconstruction, and pixel‑space diffusion approaches.

Recent work on 3D content spans three strands: diffusion‑driven iterative generation, feed‑forward reconstruction, and direct pixel‑space diffusion. We briefly map these strands to highlight where PixWorld departs.

They generate images by running a diffusion process inside the compressed latent space of a pretrained VAE, then decode the final latent back to pixels.

Early pipelines use a pretrained 2D diffusion model as a prior and iteratively optimise a 3D representation (e.g., 3DGS or NeRF) so that rendered views match the diffusion‑generated images.

This line first synthesises multi‑view images or videos with a 2D diffusion model, then reconstructs a 3D scene from those views using standard multi‑view reconstruction or incremental outpainting.

Recent approaches move the diffusion process into a compressed latent space (e.g., VAE or Representation Autoencoder) and learn a decoder that maps latents to 3D Gaussians, typically targeting 3DGS.

Deterministic, single‑pass networks map sparse or dense multi‑view inputs directly to a 3D representation, eliminating costly per‑scene optimisation.

Instead of operating in a latent bottleneck, diffusion is performed directly on pixel values; with sufficient model capacity and data, this can achieve fidelity comparable to or exceeding latent‑space diffusion.

The PixWorld Framework

PixWorld replaces latent bottlenecks with a unified pixel‑space diffusion that directly supervises 3D geometry.

Latent diffusion inserts an encoder‑decoder bottleneck that decouples the diffusion variable from the final image, making it hard to align the loss with the underlying 3D scene. Pixel‑space diffusion keeps the diffusion variable in the RGB domain, allowing the loss to be applied directly to rendered views.

Instead of predicting a clean image, the model predicts the instantaneous velocity that would move a noisy image toward the clean one, and the loss measures how close this predicted velocity is to the true one.

Compute the noisy input: $x_t = 0.5\,x + 0.5\,\epsilon = [0.55,0.25,0.45,0.65]$.

Assume the denoiser predicts $\hat{x} = [0.3,0.5,0.7,0.9]$.

Calculate the predicted velocity: $\hat{v} = (\hat{x} - x_t)/(1-0.5) = [0.5,0.5,0.5,0.5]$.

Ground‑truth velocity: $v = (x - \epsilon)/(1-0.5) = [-1.4,-0.2,-0.6,-0.2]$.

Flow‑matching loss: $\|\hat{v} - v\|_2^2 = (0.5+1.4)^2 + (0.5+0.2)^2 + (0.5+0.6)^2 + (0.5+0.2)^2 \approx 5.04$.

The loss penalizes the mismatch between the predicted and true velocity, forcing the network to learn how each pixel should move toward its clean counterpart.

Why does PixWorld predict a velocity instead of directly predicting the clean image?

Predicting velocity makes the loss scale uniformly across timesteps: the same quadratic penalty applies whether the input is almost clean or almost pure noise. Direct image prediction would give a tiny error when $t\approx1$ and a huge error when $t\approx0$, destabilizing training.

The model treats the set of input views as two streams: a clean stream that provides reliable geometry cues, and a noisy stream that the diffusion process must denoise, enabling simultaneous reconstruction and generation.

Could the clean and noisy streams be merged into a single input without explicit partitioning?

No — without a clean anchor the diffusion process would have no reliable reference, leading to ambiguous geometry and unstable generation. The clean stream supplies a fixed point that grounds the noisy stream’s denoising.

Project clean images $\{I_n\}_{n\in\Omega_c}$ and noisy images $\{\tilde{I}_n^{t}\}_{n\in\Omega_n}$ with separate linear embeddings.

Inject the shared timestep embedding $t$ into the noisy stream and a fixed $t=1$ embedding into the clean stream.

Add camera positional encodings (PRoPE) and optional text embeddings via cross‑attention.

Pass the concatenated token sequence through $L$ shared DiT transformer blocks.

Decode per‑view depth maps $\hat{D}_n$ and a pixel‑aligned 3D Gaussian scene $\hat{G}$ from the transformer output.

Render $\hat{G}$ back to each view with a differentiable renderer $R$ to obtain $\bar{I}_n$.

Each pixel’s predicted depth is unprojected into a 3D point, which becomes the center of a tiny Gaussian; the collection of all such Gaussians forms a continuous scene representation that can be rendered from any viewpoint.

How does 3DGS differ from a voxel grid representation?

Voxel grids discretize space into fixed cells, incurring memory blow‑up for high resolution. 3DGS stores one Gaussian per pixel, so memory scales with image size, and each Gaussian can represent sub‑pixel detail via its covariance.

Beyond pixel‑wise photometric errors, the model is penalized if the rendered views disagree with the ground truth in a feature space that captures 3D structure, as extracted by a frozen 3D foundation model.

Why not use a simple depth‑only loss instead of the geometry perception loss?

Depth loss aligns each view independently, but it cannot detect inter‑view inconsistencies such as a surface that drifts along a ray. The geometry perception loss evaluates the whole multi‑view feature field, catching structural errors that depth alone would miss.

**Figure 2.** **Overview of PixWorld.** (a) PixWorld adopts a unified DiT-based framework that takes noisy and clean multi-view inputs, with optional text conditioning, and jointly predicts depth and 3DGS through shared transformer blocks. (b) A pixel-space flow matching loss is imposed on rendered multi-view images to directly optimize the underlying 3D representation. (c) A geometry perception loss further enforces structural consistency by aligning rendered views with ground-truth observations through a 3D foundation model.

**Figure 3.** **Visualization of PixWorld under different settings.** PixWorld flexibly handles both 3D reconstruction and generation: when all input views are clean, it performs reconstruction; when clean and noisy views are arbitrarily mixed, it performs generation. We visualize the camera trajectory, where blue and red frustums denote clean input views and generated views, respectively.

Empirical Evaluation

Quantitative results show PixWorld dominates prior methods on reconstruction and generation.

PixWorld is trained from scratch with 1.04 B parameters on a mixture of 67 K multi‑view scenes and 10 M single images, using AdamW for roughly 200 K steps on 32 A800‑SXM4‑80G GPUs.

PixWorld improves single‑image generation PSNR by +1.06 dB on RealEstate10K compared to the strongest baseline.

Table 2 reports 18.88 dB versus 17.82 dB for the next best method.

**Figure 4.** Visualization of comparisons with baselines. The large view on top denotes the input view, while the two smaller views below show novel views generated by each method.

**Figure 5.** Ablation study on the Geometry Perception loss in PixWorld. Given a single input image, our model generates the subsequent 7 frames (8 frames in total); we visualize 4 representative frames here for clarity. Pose accuracy is quantitatively evaluated by comparing the estimated camera poses against the ground-truth poses. Compared to the variant without Geometry Perception (w/o Geom.), the full model achieves more precise camera pose control and substantially mitigates the blurriness in later-view predictions, demonstrating that the global 3D perception loss is essential for maintaining both geometric consistency and visual fidelity over long generation horizons.

Across reconstruction and generation benchmarks, PixWorld sets new state‑of‑the‑art performance, confirming that pixel‑space flow matching with geometry‑aware supervision yields both higher fidelity and better pose control.

Summary and Limitations

PixWorld frames 3D reconstruction and generation as a conditional pixel‑space flow‑matching task.

PixWorld eliminates the intermediate VAE/RAE stage, so the diffusion objective can supervise the 3D representation directly via differentiable rendering, avoiding latent information loss and extra training cost.

A geometry perception loss further aligns rendered and ground‑truth views in the geometry‑aware feature space of a pretrained 3D foundation model, strengthening structural consistency beyond 2D photometric losses.

The appendix expands on five axes: B – two‑stream MMDiT denoiser architecture and parameter budget; C – disaggregated 1‑view/2‑view quantitative results; D – extra qualitative RGB and depth visualizations; E – inference‑speed comparison; F – responsible‑considerations.

The denoiser $f_\\theta$ is a 24‑layer DiT (hidden width $d\\!=\\!1024$, 16 heads of dimension $64$) with SwiGLU feed‑forward layers, RMSNorm on $Q$, $K$, and adaLN‑Zero modulation conditioned on the diffusion timestep.

Following the SD3‑style MMDiT design, each of the two parallel streams—clean ($\\Omega_c$) and noisy ($\\Omega_n$)—maintains independent pre‑LayerNorm, $QKV$, MLP, and adaLN‑Zero weights, while sharing a single cross‑attention to text tokens and a shared timestep embedder (clean at $t\\!=\\!1$, noisy at sampled $t$).

Within each attention layer, per‑stream $Q$, $K$, $V$ tensors are concatenated along the token axis, a joint full attention is computed over $[\\Omega_c;\\Omega_n]$, and the result is split back to stream‑specific output projections.

Camera parameters are injected via PRoPE, inputs are 16$\\times\$16 patches with learnable positional embeddings, and the final layer emits per‑pixel depth and 3D‑Gaussian attributes; Gaussian centers are obtained by unprojecting each pixel using its predicted depth.

**Table.** Module configurations and parameter counts for the model architecture, divided into (a) Tokenization & conditioning, (b) Two-stream MMDiT block, and (c) Multi-task output heads.

Training data combine RealEstate10K and DL3DV‑10K (~67 K posed scenes) with a 10 M image single‑view branch from the BLIP‑3o corpus; each scene provides $N\\in\\{4,\\dots,8\\}$ views, partitioned into $\\Omega_c/\\Omega_n$, biasing toward small $|\\Omega_c|$.

Batches alternate a 32‑image single‑view batch with a multi‑view batch of up to 32 images (e.g., 4–8 views × 4–8 scenes), ensuring balanced exposure to both regimes.

The loss combines rendering, depth, and geometry terms: $L\\!=\\!L_{render}+\\lambda_{depth}L_{depth}+\\lambda_{geo}L_{geo}$ with $\\lambda_{depth}=1.0$, $\\lambda_{lpips}=\\lambda_{geo}=0.1$, and $L_{lpips}$, $L_{geo}$ gated at $t>t_{th}=0.3$.

The frozen geometry critic $\\Psi$ uses $\\pi3$ with stopped gradients, providing a geometry‑aware feature space for $L_{geo}$.

Training runs from scratch with AdamW, a linear learning‑rate decay from $1\\times10^{-4}$ to $1\\times10^{-5}$, EMA decay $0.9995$, gradient clipping $1.0$, and classifier‑free text dropping $0.2$, for roughly $200\\,\\text{k}$ steps on 32 NVIDIA A800‑SXM4‑80G GPUs.

Extended Quantitative Results

PixWorld outperforms all baselines across every 1‑view and 2‑view configuration.

PixWorld achieves the highest PSNR (18.92) among all methods on every 1‑view and 2‑view configuration.

Table 7 shows PixWorld leading in PSNR, SSIM, LPIPS and all camera‑control AUC metrics for RealEstate10K and DL3DV‑10K.

Across the four configurations, every baseline suffers a larger drop on the harder side (First Frame for 1‑view, Extrapolation for 2‑view), especially in LPIPS and the Camera‑Control AUC metrics.

The relative ordering of methods stays the same across sub‑configurations, confirming that the averaged numbers in the main paper faithfully reflect each method’s behavior.

The two‑stream DiT denoiser (Table 6) processes clean and noisy patches in parallel, sharing a joint attention over the concatenated token sequence while keeping separate MLPs and adaLN‑Zero weights per stream.

**Figure 6.** More visualizations of reconstruction and generation under varying view selections, including camera trajectories with input and generated views marked, and the corresponding depth maps predicted by PixWorld.

**Figure 7.** More visualizations of generated scenes. The first view is the input, and we show both RGB renderings and predicted depth maps.

Inference Speed Benchmarks

We report PixWorld’s inference latency and compare it to prior video‑diffusion baselines.

We evaluate inference on a single A100‑SXM4‑80G GPU, measuring wall‑clock time per generated scene, the number of key frames, and the number of function evaluations (NFE).

PixWorld generates a scene in 15 seconds, matching the order of magnitude of the highly optimized FlashWorld (10 s).

Table 9 shows the per‑scene latency for all baselines.

PixWorld runs at a lower output resolution than the video‑diffusion baselines, which cuts per‑step compute, and it avoids dense frame synthesis by reconstructing a 3D scene from only eight key frames before rasterizing novel views.

Limitations and Ethics

We outline current limits, broader impacts, and clarify LLM usage.

Our experiments are limited to widely used scene‑level datasets such as RealEstate10K and DL3DV‑10K; evaluating on more diverse outdoor and object‑centric scenes would better reveal the framework’s generalization.

Pixel‑space diffusion with differentiable rendering is trained at finite resolution and compute budgets, leaving room for finer texture fidelity and scalability to higher‑resolution multi‑view settings.

We plan to accelerate inference through distillation (compressing a large model into a smaller one) and quantization (reducing numeric precision), further reducing the cost of high‑quality 3D scene generation.

Beyond reconstruction, PixWorld could enable efficient 3D content creation, robotic perception, simulation, and VR/AR, but it also raises privacy, synthetic‑content misuse, and safety‑critical deployment concerns.

We therefore recommend responsible data usage, thorough evaluation, and human oversight when deploying such systems in real‑world scenarios.

This work does not rely on large language models as a core component; any LLM‑based tools were limited to writing, editing, or formatting assistance and did not affect methodology or results.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers