DiffGI: Differentiable Geometry Images for High-Fidelity Thin-Shell 3D Generation

Eungjune Shim, Hansol Lee, Eunjung Ju

DiffGI enables high-fidelity, thin-shell 3D generation by replacing binary occupancy maps with a differentiable, continuous 2D TSDF.

How can we represent thin-shell 3D objects in a way that is both differentiable and capable of capturing fine geometric details without the topological constraints of volumetric fields?

Existing 3D generative models often struggle with thin-shell structures like garments, as they rely on binary occupancy maps that suffer from resolution-dependent staircase artifacts and non-differentiable mesh extraction. The authors introduce Differentiable Geometry Images (DiffGI), which represent surfaces using a continuous 2D Truncated Signed Distance Function (TSDF) and a Differentiable Marching Squares (DMS) module. This allows 3D geometric losses to backpropagate directly into the 2D latent space, enabling end-to-end optimization. This approach compresses complex 3D surfaces into an ultra-compact 32×32 latent space, achieving superior boundary precision and reconstruction fidelity compared to prior geometry-image and voxel-based methods.

Paper Primer

DiffGI treats 3D surface generation as a 2D image-like tensor problem, but replaces the standard binary mask with a continuous distance field. The core move is the Differentiable Marching Squares (DMS) module: it acts like a bridge, converting the continuous 2D distance values into 3D mesh vertices using linear interpolation, which keeps the entire pipeline differentiable.

DiffGI achieves state-of-the-art reconstruction fidelity for thin-shell, non-manifold geometries.

Quantitative evaluation on the GarmageSet dataset shows superior Chamfer Distance (CD) and Normal Consistency (NC) compared to GarmageNet and Omages. On GarmageSet, DiffGI reduces CD to 0.366 × 10⁻³ and achieves an NC of 0.961, significantly outperforming occupancy-based baselines.

Why is a continuous TSDF representation better than binary occupancy for 3D surfaces?

Binary occupancy maps encode boundaries as hard 0/1 transitions, which lose subpixel information and create staircase artifacts when downsampled. The TSDF encodes the distance to the nearest boundary, preserving subpixel detail even at low resolutions.

What is the primary limitation of this approach?

Because the DMS module treats each UV chart independently, the reconstructed geometry can exhibit visible seams at chart boundaries, which may complicate downstream physical simulations.

DiffGI demonstrates that surface-centric 2D representations can match the fidelity of volumetric models while being significantly more efficient, making high-quality thin-shell 3D generation viable on consumer-grade hardware.

Introduction: The Thin-Shell Problem

Framing the gap: volumetric models miss thin‑shell garments, DiffGI fixes it.

Implicit volumetric representations dominate current 3D generative models, but by enforcing watertight topology they cannot faithfully capture thin‑shell or non‑manifold structures such as garments.

Existing geometry‑image approaches mitigate this by mapping surfaces to 2D grids, yet they rely on discrete binary occupancy maps; the resulting resolution‑dependent stair‑case artifacts and non‑differentiable reconstruction steps limit fidelity.

We address both issues with Differentiable Geometry Image (DiffGI), which replaces binary occupancy with a continuous 2D TSDF, enabling sub‑pixel boundary encoding and a fully differentiable Marching Squares algorithm that propagates surface‑level gradients back to the 2D latent space.

**Fig. 1:** VAE reconstruction results with our TSDF-based DiffGI representation and normal rendering loss (left) versus an occupancy-based geometry image without normal loss (right), showing noticeably sharper boundaries and better preservation of thin-shell structures.

Volumetric models fail on garments because their watertight constraint forces artificial thickness, erasing the delicate thin‑shell structures that define realistic clothing.

Prior Approaches to 3D Generation

We survey implicit, geometry‑image, and differentiable extraction approaches relevant to DiffGI.

Implicit volumetric fields dominate recent 3D generation, but they enforce watertight topology and struggle with thin‑shell geometries. Geometry‑image approaches map surfaces to regular UV grids, enabling 2D generative backbones while preserving mesh structure. Differentiable iso‑surface extraction integrates field‑to‑surface conversion into training, offering end‑to‑end surface supervision.

Uses a geometry‑image prior with a binary occupancy channel to generate 3D shapes via 2D convolutional networks.

Extends geometry‑image generation with learned garment‑specific priors, still using a binary mask to delineate surface regions.

Employs a hierarchical geometry‑image representation with multiple resolution levels to capture fine‑grained surface detail.

Our DiffGI framework replaces the binary occupancy channel with a continuous 2D TSDF on a single 256×256 tensor, enabling smooth boundary localization without heuristic snapping. By applying Differentiable Marching Squares (DMS) to this TSDF, we achieve end‑to‑end surface optimization while preserving UV compatibility.

The DiffGI Representation

Continuous TSDF geometry images and a differentiable marching squares module enable end‑to‑end 3D generation.

Binary occupancy maps encode surface presence as hard 0/1 values, so downsampling erases fine boundary detail and introduces staircase artifacts.

A Geometry Image stores a 3D shape on a 2D raster by packing UV‑parameterized surface patches into a regular grid.

A 2D TSDF records, for every pixel, the signed distance to the nearest surface, smoothly varying across space and clamped at a fixed truncation distance.

Pixels inside the chart receive positive distances: the center pixel (1.5,1.5) gets +0.5, its immediate neighbors get +0.75, etc.

Pixels outside receive negative distances: the corner pixel (0,0) gets –1.5, the edge pixel (0,1) gets –0.5.

All distances larger than 3 are clamped to ±3, but none exceed the threshold in this tiny example.

The TSDF’s continuous values allow a downstream network to recover the exact square boundary even after downsampling to 2 × 2, where a binary map would have lost the interior‑exterior distinction.

UV patches are packed onto a 2D plane with uniform global scaling and padding to avoid bleeding between patches.

The mesh surface is sampled on a 1024 × 1024 regular grid via barycentric interpolation, producing a 3‑channel position map.

Edge pixel values are dilated outward into the undefined background to protect boundaries during downsampling.

For every pixel, the signed Euclidean distance to the nearest chart contour is computed and clamped at 15 pixels, yielding a 2D TSDF.

The 1024 × 1024 tensor is bilinearly downsampled to 256 × 256 × 4; the continuous TSDF prevents information loss.

Traditional Marching Squares determines topology via a discrete lookup table, which blocks gradient flow and prevents end‑to‑end learning of geometry.

DMS treats each vertex position as a continuous function of the underlying TSDF values, enabling gradients to flow from 3D losses back to the 2D representation.

Compute denominator: ϕ_A − ϕ_B + $\epsilon$·sgn(ϕ_A − ϕ_B) = 0.2 − (‑0.1) + 10⁻⁵ = 0.30001.

Compute x: ϕ_A / denominator = 0.2 / 0.30001 ≈ 0.6666.

The boundary point lies two‑thirds of the way from pixel A to pixel B; bilinear sampling of the position map yields the 3D vertex coordinates.

The interpolation formula yields a smooth, differentiable estimate of the boundary location even when the two TSDF values are very close, thanks to $\epsilon$.

**Fig. 2:** Overview of the proposed DiffGI framework. The input 3D mesh is first mapped to a 2D TSDF-based DiffGI representation, which is then encoded by a DiffGI-VAE into a compact latent space. The decoder reconstructs the TSDF geometry image, from which a 3D triangle mesh is recovered via Differentiable Marching Squares. Pixel-space losses on the TSDF and position maps, together with a geometry-aware normal rendering loss, are jointly applied to enable end-to-end optimization of high-fidelity 3D surfaces.

By making the marching‑squares extraction differentiable, the framework can train directly on 3D geometric objectives, achieving high‑fidelity reconstruction of thin‑shell and non‑manifold surfaces.

DiffGI-VAE and Training

We compress geometry images with a VAE and train a diffusion transformer for conditional 3D generation.

High‑resolution geometry images ($256\times256\times4$) are too memory‑intensive for direct generation, so we need a compact latent representation that still preserves fine surface detail.

The VAE encodes a 4‑channel geometry image (position + TSDF) into a $32\times32\times4$ latent tensor, then decodes it back to a mesh while keeping geometric fidelity.

How does this VAE differ from a standard image VAE?

Standard VAEs assume three‑channel RGB inputs and learn generic visual priors. Here the encoder is widened to four channels (UV + TSDF) and inherits a spatial compression prior from Stable Diffusion 1.5, which is crucial for preserving geometric structure rather than color.

Instead of only matching pixel values, we also match the surface normals rendered from the reconstructed mesh, forcing the latent code to capture high‑frequency curvature.

Encoder compresses the $4\times4$ grid to $2\times2$ by averaging neighboring pixels in each channel.

Decoder expands the $2\times2$ latent back to $4\times4$ using transposed convolutions.

DMS builds a mesh from the upsampled TSDF, producing a simple quad surface.

Rasterizer $R$ renders a normal map where each pixel’s normal points outward from the quad.

Ground‑truth normal map (flat surface) is identical, so $L_{\text{Normal}}$ evaluates to zero.

The normal loss forces the encoder to keep curvature information even when spatial resolution is aggressively reduced.

With the compressed latent space in place, we train a diffusion transformer that operates directly on the $32\times32\times4$ tensors, enabling conditional 3D generation.

The diffusion model treats each latent token as a global descriptor, allowing unrestricted attention across the entire geometry image.

Geometric data augmentation for training DiffGI‑VAE.

Experimental Setup

Key quantitative results and evaluation protocol for DiffGI‑VAE on three 3D benchmarks.

DiffGI‑VAE achieves the highest Normal Consistency (NC) on both ABO and GarmageSet benchmarks.

NC reaches 0.89, while the strongest baselines attain at most 0.21.

We evaluate on three benchmarks—ABO (furniture), GarmageSet (garments), and WARDROBE (real clothing images)—using a suite of metrics: CD, EMD, JSD (lower is better) and NC (higher is better), plus boundary‑focused BCD and worst‑case HD. Table 1 reports per‑method results for CD, EMD, JSD, and NC on ABO and GarmageSet, showing that DiffGI‑VAE matches or surpasses baselines across all metrics while dramatically improving NC.

Reconstruction Fidelity

We isolate how representation and normal loss each affect reconstruction quality.

This section quantifies how the two design choices—TSDF versus occupancy and the geometry‑aware normal rendering loss—contribute to DiffGI‑VAE’s reconstruction fidelity on GarmageSet.

Switching from occupancy to TSDF representation cuts CD by 1.04 × 10⁻³ on GarmageSet.

TSDF w/o NL achieves CD 0.461 versus Occupancy w/o NL at 1.503 (Table 2).

Adding the geometry‑aware normal rendering loss to the TSDF variant raises NC by 0.04.

TSDF w/ NL reaches NC 0.961 compared with TSDF w/o NL at 0.921 (Table 2).

Adding normal loss to the occupancy variant raises NC by 0.041.

Occupancy w/ NL attains NC 0.947 versus Occupancy w/o NL at 0.906 (Table 2).

Training DiffGI‑VAE from random initialization yields CD within 0.01 × 10⁻³ of the SD1.5‑initialized model, confirming the gains stem from the representation.

Random‑init CD 0.47 vs. SD1.5‑init CD 0.46 on GarmageSet (supplementary Sec. 2).

Efficiency and Performance

DiffGI achieves sub‑second GPU inference with far lower memory than prior 3D generators.

DiffGI cuts inference time by roughly 90 % compared to the Omages baseline.

Omages requires 52 s per sample while DiffGI‑Label finishes in 0.50 s.

Compared to volumetric models, DiffGI’s 2D latent space reduces peak VRAM from 16.28 GB (TRELLIS‑image) to just 1.18 GB, while still delivering comparable or faster inference.

On a consumer‑grade RTX 4070 (12 GB), DiffGI‑Image runs in 1.21 s, and even on a CPU‑only MacBook M4 the model completes generation in about 8.5 s, demonstrating scalability from server to edge.

The table compares the performance of "Omages" and "Ours" across four categories (Chair, Lamp, Sofa, Table) and a Mean value, using P-FID and P-KID metrics.

Despite the speed gains, DiffGI maintains high generation quality, achieving a mean P‑FID of 15.81 and P‑KID of 0.06 on the ABO dataset.

The table compares the performance of different methods (TRELLIS, TRELLIS.2 512^2, GarmageNet, and Ours (DiffGI)) across several metrics: number of vertices (#Vert.), Chamfer Distance (CD), F1 score, Hausdorff Distance (HD), and Boundary Chamfer Distance (BCD).

Generative Performance

DiffGI dramatically improves label‑conditioned 3D generation and excels on thin‑shell image‑to‑3D tasks.

DiffGI reduces label‑conditioned P‑FID by up to 50% on Chair and 49% on Lamp, outperforming prior image‑based baselines.

Table 4 shows consistent P‑FID improvements across all ABO categories, with the largest drops on Chair and Lamp.

**Table 4.** Label-conditioned 3D generation performance on the ABO dataset, reported as P-FID and P-KID per category and averaged (Mean).

**Fig. 5:** Qualitative label-conditioned generation results on the ABO dataset. We compare Omages and our DiffGI-based diffusion model, showing improved reconstruction of thin frames and open-boundary structures, with fewer staircase artifacts.

**Fig. 6:** Qualitative single-view image-to-3D comparison on GarmageSet. From left to right, each row shows the input front-view rendered normal map, the results of TRELLIS, TRELLIS.2, and GarmageNet, our DiffGI generation, and the ground-truth mesh. DiffGI preserves thin-shell details and produces cleaner open boundaries with far fewer vertices than the watertight foundation models (TRELLIS, TRELLIS.2) and the occupancy-based GarmageNet.

**Fig. 7:** Occupancy-conditioned garment generation and local edit propagation. Given a 2D occupancy map as input, modifying only the sleeve region in 2D leads to consistent updates of the corresponding 3D geometry, demonstrating that local pattern edits are faithfully reflected in the generated garment mesh.

Resolution and Initialization Analysis

Ablation studies on representation resolution and VAE initialization reveal their impact on reconstruction quality.

We first isolate the effect of the geometry‑image representation by comparing TSDF‑based and binary Occupancy images across five resolutions (64–1024) without any VAE compression.

**Fig. S1:** Resolution-dependent reconstruction quality of TSDF vs. Occupancy without VAE compression. **Top:** Hausdorff Distance (HD) vs. resolution on ABO (left) and GarmageSet (right); solid lines and shaded regions denote mean and standard deviation. **Bottom:** Per-vertex error maps for a representative ABO sample (colour scale: point-to-surface distance). TSDF achieves lower error at low resolutions (64, 128), with both representations converging beyond 256.

Next we assess how much of the final reconstruction quality depends on the Stable Diffusion 1.5 (SD1.5) initialization versus a random start, keeping the DiffGI‑VAE architecture identical.

**Fig. S2:** From-scratch vs. SD1.5-pretrained DiffGI-VAE. **Left:** total loss during early training; the SD1.5-initialized model (blue) reaches a low loss within far fewer steps, whereas the from-scratch model (orange) starts much higher and converges more slowly. **Right:** reconstructions over the course of training (top: geometry image, bottom: 3D mesh); the from-scratch model begins from noise but progressively recovers reconstructions essentially identical to the pretrained one. Both reach comparable final fidelity when fully trained (Tab. S1), so SD1.5 initialization mainly accelerates convergence rather than improving final quality.

Data Augmentation Impact

Geometric augmentation stabilizes training and unlocks larger model performance.

We evaluate how geometric data augmentation—specifically UV placement perturbation—affects training stability and model capacity on the ABO dataset. Visual examples of the perturbation are shown in Fig. S4.

**Fig. S3:** Impact of geometric data augmentation on DiT capacity scaling (ABO, 4 classes). **Left:** Evaluation loss vs. training step. Without augmentation (3,800 samples) the higher-capacity DiT-Base diverges and only DiT-Tiny trains stably, whereas with augmentation (500K samples) DiT-Base converges reliably. **Right:** Qualitative DiT-Base generations, showing that augmentation yields better sample quality than the non-augmented model.

When training on the original 3,800 samples, only DiT‑Tiny converges stably; DiT‑Base exhibits unstable validation loss and eventually diverges. Expanding the dataset to 500 K samples via UV‑aligned augmentation restores stable convergence for DiT‑Base and yields better generation quality than DiT‑Tiny, demonstrating that augmentation is essential for scaling up model capacity on limited data.

**Fig. S4:** UV placement perturbation examples. Different UV chart layouts are generated from the same 3D mesh while preserving geometric signals, enabling large-scale data expansion without changing 3D semantics.

Cases 6 and 9 of Marching Squares correspond to topologically ambiguous saddle‑point configurations; we deterministically treat the two patches as separate independent regions. Inter‑patch padding during UV packing (Sec. 3.1) spatially separates surface charts on the 2D grid, making such ambiguous diagonals unlikely at our operating resolution of $256\times256$, and narrow bridge‑like structures that could be mis‑split are rarely encountered.

Additional Qualitative Results

Qualitative examples highlight DiffGI’s sharper boundaries and mesh efficiency.

This section showcases extra qualitative outputs that illustrate how DiffGI improves mesh fidelity and diversity beyond the baselines.

DiffGI generates meshes with dramatically fewer vertices while preserving fine garment details.

Our reconstructions use roughly 23 K vertices, whereas TRELLIS and GarmageNet require 109–526 K.

**Fig. S5:** Additional image-conditioned results on GarmageSet (left to right: input, TRELLIS, GarmageNet, ours). Our method produces cleaner boundaries and thinner shells with significantly fewer vertices (~23K vs. 109–526K).

**Fig. S6:** Zero-shot image-to-3D on the WARDROBE dataset [14] (not used in training). Our DiffGI better matches input silhouettes and preserves open boundaries compared to TRELLIS.

**Fig. S7:** Generation diversity under a fixed condition. Each row shows an input image (left) and multiple 3D garments sampled from the same condition; insets show the generated geometry-image chart layout of each sample. Samples stay consistent with the input while varying naturally in chart layout, draping, and wrinkles.

Conclusion and Limitations

We recap DiffGI’s achievements, note its current limits, and outline future research directions.

DiffGI replaces binary occupancy maps with a continuous 2D TSDF, enabling sub‑pixel‑precise boundary reconstruction on geometry images. The Differentiable Marching Squares (DMS) module embeds mesh reconstruction within the learning graph, allowing 3D surface losses to flow back to the 2D latent space. Built on this, DiffGI‑VAE compresses complex non‑manifold surfaces into a compact 32×32×4 latent representation using a geometry‑aware normal rendering loss, and a transformer‑based latent diffusion model generates high‑quality 3D shapes in real time on consumer‑grade hardware.

TSDF‑based linear interpolation can introduce localized rounding on extreme sharp edges, such as mechanical parts. The current framework does not generate RGB textures or PBR material attributes alongside geometry. Because each UV chart is reconstructed independently, DMS treats adjacent patches as separate regions, leading to visible seams where neighboring charts meet, exemplified by the upper‑lower dress split in Figure 6.

Future work will integrate feature‑preserving iso‑surfacing (e.g., dual contouring) into DMS to better handle sharp‑edge domains. We also aim to extend the latent space to jointly generate high‑resolution textures and material maps. Finally, a two‑stage pipeline will separate 2D pattern layout generation from 3D shape reconstruction, offering stronger controllability for complex garment designs.

We thank Hyun Kang, Seungoh Han, Sihun Cha, Dong‑sig Kang, and Gyoo‑Chul Kang for valuable discussions and feedback. We also acknowledge CLO Virtual Fashion for providing the research environment and resources that made this work possible.

Implementation Details

Implementation specifics for DiffGI, covering TSDF map creation, dataset usage, augmentations, and model architectures.

The TSDF map is generated entirely on the 2D UV plane by rasterising packed UV charts into a binary mask at $1024\times1024$, then applying a 2‑D Euclidean distance transform. Each pixel receives its signed distance (positive inside a chart, negative outside) and the values are clamped at 15 pixels before downsampling.

The WARDROBE dataset is employed solely for qualitative zero‑shot evaluation and does not contribute to the quantitative benchmarks such as ABO or GarmageSet.

Geometric data augmentation is discussed in detail in Sec. S3, where the impact of various augmentation strategies on the DiffGI pipeline is analyzed.

Architecture specifications for the three conditional generation models (label‑conditioned, image‑conditioned, and occupancy‑conditioned) are provided in Table S2, covering backbones, conditioning modalities, injection mechanisms, schedulers, and latent sizes.

**Table S2.** Architecture details for the three conditional generation models.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers