Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
Igor Pavlovic, Thiemo Wandel, Anton Obukhov, Luca Bartolomei, Andrey Davydov, Fabio Tosi, Matteo Poggi, Sabine Süsstrunk, Dengxin Dai
Marigold V2 repurposes image-editing diffusion transformers into high-fidelity depth estimators using a two-stage fine-tuning protocol.
How can we efficiently adapt a pre-trained image-editing diffusion transformer into a high-performance monocular depth estimator using minimal compute?
Diffusion-based depth estimators often struggle to resolve fine-grained details, frequently producing oversmoothed boundaries or "flying pixels" that degrade 3D reconstruction quality. The authors introduce Marigold V2, which adapts a pretrained image-editing diffusion transformer (DiT) using a two-stage fine-tuning recipe: first aligning internal representations with ground-truth semantic features, then refining boundaries with a novel optimal-transport loss. This approach achieves state-of-the-art results on standard benchmarks, improving AbsRel by 16–26% over previous methods while preserving thin structures like hair and foliage.
Paper Primer
The core mechanism hinges on two innovations: iREPA-depth and SinkLoss. iREPA-depth aligns the DiT's internal representations with semantic features extracted from ground-truth depth maps rather than RGB inputs, while SinkLoss uses Sinkhorn-Knopp matching to enforce local depth consistency within image tiles without requiring strict pixel-wise alignment.
The method is like a specialized translator: it takes the broad geometric "vocabulary" learned by a massive image-editing model and forces it to map input pixels to depth values, using the Sinkhorn loss as a "soft" guide that allows the model to prioritize structural coherence over noisy, pixel-perfect ground truth.
Marigold V2 significantly improves boundary sharpness and detail preservation compared to existing diffusion-based depth estimators.
Evaluation on the HyperSim dataset using Soft Edge Error (SEE) metrics shows consistent superiority across all thresholds (SEE3, SEE5, SEE7). 16–26% improvement in AbsRel on KITTI and ETH3D benchmarks.
The training recipe is highly transferable to other dense regression tasks.
The same fine-tuning protocol achieves state-of-the-art performance on surface normal estimation, intrinsic image decomposition, and depth completion.
Why use a diffusion transformer (DiT) for depth estimation when discriminative models are faster?
Generative models are trained on billion-scale data, allowing them to encapsulate richer geometric and semantic world knowledge than discriminative models, which are often limited by the scarcity of high-quality depth-annotated training data.
What is the primary limitation of this approach?
The reliance on a large image-editing backbone precludes real-time inference, and the model remains sensitive to ambiguities like reflections, motion, or defocus blur.
Marigold V2 demonstrates that state-of-the-art geometric perception can be distilled from open-source generative models in a matter of days, providing a robust, high-fidelity tool for computational photography and 3D scene reconstruction.
Introduction and Motivation
We frame why training diffusion‑based depth models is costly and detail‑poor, and introduce a cheap fine‑tuning fix.
Monocular depth estimation underlies scene reconstruction, computational photography, and robotics, yet a single RGB image admits infinitely many 3D interpretations. Recent diffusion‑based estimators inherit the generative model’s richness but suffer from oversmoothed surfaces and lost fine details, especially on out‑of‑distribution inputs.
We need a depth estimator that keeps the geometric fidelity of diffusion models while removing their costly training and blurry‑edge artifacts.
Depth expressed in a coordinate‑free way so that rotating or translating the camera does not alter the depth values.
**Fig. 1.** We present Marigold V2, a model and a cost-effective fine-tuning protocol that repurposes an open-source image-editing diffusion transformer (Qwen-Image-Edit) into a state-of-the-art monocular depth estimator in less than a week on a single 32GB GPU. Our protocol extends Marigold and its DiT follow-ups via novel Sinkhorn and representation alignment losses to sharpen details and improve geometry, while remaining affordable for individual practitioners and small labs. Beyond quantitative accuracy, Marigold V2 excels qualitatively, as shown above: it faithfully reproduces sharp edges, fur, and hair-thin details, surpassing recent detail-oriented methods such as Pixel-Perfect Depth (PPD) and InfiniDepth, all while retaining the efficiency of a VAE-based model.
The shift from training diffusion models from scratch to a cost‑effective fine‑tuning pipeline unlocks high‑quality depth estimation for individual researchers.
Related Work
Survey of prior depth estimation approaches and their limitations.
Prior work splits into discriminative models that struggle with data scarcity and generative priors that leverage massive pre‑training but incur high cost.
Marigold V1 is the first version that repurposes an image‑editing transformer for depth prediction using a simple two‑stage fine‑tuning pipeline.
The Marigold V2 Protocol
We detail a two‑stage fine‑tuning pipeline that turns a pretrained image‑editing diffusion transformer into a high‑quality monocular depth estimator.
Training large diffusion models for depth estimation is costly and often yields edge artifacts; the core trick is to split learning into a cheap coarse stage and a focused refinement stage.
Stage 1: freeze the pretrained DiT backbone, attach rank‑128 QLoRA adapters, and train on latent rectified‑flow targets while also minimizing pixel‑space $L_1$ and gradient losses and the iREPA‑depth regularizer.
Stage 2: unfreeze the VAE decoder and add the SinkLoss; continue fine‑tuning the whole model on the same data.
At inference, feed the RGB image through the VAE encoder, apply the trained DiT once, and decode the resulting latent to obtain the depth map.
iREPA‑depth aligns internal diffusion features with depth‑domain visual cues, forcing the model to respect semantic structure that pixel‑level losses miss.
Compute cosine similarities: $\text{sim}(f_1,g_1)=0.99$, $\text{sim}(f_1,g_2)=0.30$, $\text{sim}(f_2,g_1)=0.31$, $\text{sim}(f_2,g_2)=0.99$.
Apply the contrastive loss with $\tau=0.07$: the numerator for each pair is $\exp(0.99/0.07)\approx 1.3\times10^{6}$, the denominator sums the two exponentials, yielding a loss $\approx 0.02$ per patch.
Back‑propagation pushes the mismatched embeddings apart while reinforcing the matched ones, tightening the feature alignment.
Even with tiny patches, the loss forces the model to respect semantic correspondence, which pixel‑wise $L_1$ cannot capture.
How does iREPA‑depth differ from a standard LPIPS perceptual loss?
LPIPS compares raw RGB pixels after a fixed feature extractor, ignoring the geometry of depth. iREPA‑depth instead aligns depth‑domain features, directly encouraging the model to preserve geometric structure rather than just visual similarity.
SinkLoss treats each $K\times K$ block as a bag of depth values and forces the predicted bag to be a permutation of the ground‑truth bag, eliminating the need for exact pixel‑wise alignment.
Form the cost matrix $C_{ij}=|\hat{d}_i-d_j|$, yielding a $4\times4$ matrix with values ranging from $0.01$ to $0.59$.
Apply Sinkhorn iterations on $G=\exp(-C/\tau)$ with $\tau=0.1$, obtaining a soft assignment $M$ that places most mass on the diagonal.
Compute $L_{\text{SinkLoss}} = \frac{\sum_{i,j} M_{ij} C_{ij}}{\sum_{i,j} M_{ij}} \approx 0.015$, a small average transport cost.
The loss is low because the predicted set is a near‑permutation of the ground‑truth set, even though individual pixel errors exist.
SinkLoss tolerates small spatial misalignments while still penalizing large depth mismatches, which explains its ability to suppress flying‑pixel artifacts.
Why not simply use a pixel‑wise $L_1$ loss instead of SinkLoss?
Pixel‑wise $L_1$ forces each predicted pixel to match its exact ground‑truth location, which is brittle when the annotation is noisy or ambiguous. SinkLoss relaxes this requirement to a set‑matching problem, allowing the model to focus on getting the right values without being punished for slight misplacements.
**Fig. 2. Marigold V2 training protocol.** During Stage 1, only QLoRA adapter weights are fine-tuned, regularized by iREPA-depth, which already produces a strong model. Stage 2 refines it further by adding SinkLoss and unfreezing the VAE decoder. At inference, the deployed model requires just one forward pass through the VAE and the DiT.
**Fig. 3. Impact of semantic feature losses.** Input image, baseline prediction, with LPIPS loss, with iREPA-depth loss. The iREPA-depth variant removes artifacts most efficiently and improves fine details.
Empirical Evaluation
We evaluate Marigold V2’s depth quality, edge fidelity, and efficiency.
Recall that Marigold V2 repurposes an image‑editing transformer via a two‑stage fine‑tuning protocol to produce high‑quality depth maps while keeping compute modest.
Marigold V2 attains an ETH3D AbsRel of $2.8$, beating the next‑best $3.8$ and setting a new state‑of‑the‑art for zero‑shot depth.
Table 1, ETH3D column, shows $2.8$ for Marigold V2 versus $3.8$ for the closest competitor.
**Fig. 4.** A crop of a HyperSim sample in native resolution. Thin structures appear semi-transparent in RGB and are not well-defined in ground-truth depth due to stochastic sampling within the rendering pipeline.
**Fig. 5.** Qualitative impact of SinkLoss and VAE decoder unfreezing. The Stage-1 checkpoint is compared against two continued-training variants with an unfrozen VAE decoder, with and without SinkLoss. SinkLoss substantially reduces flying pixels while preserving fine detail.
**Figure 6.** Qualitative comparison of state-of-the-art relative depth estimation methods across multiple datasets. Marigold V2 trained with iREPA-depth and SinkLoss preserves fine details while keeping flying pixels to a minimum. Pixel-Perfect Depth produces fewer flying pixels overall but loses substantial detail. All methods use the same inference resolution without any resizing. Last row courtesy of Franco Sulli via Pexels.
**Fig. 7. The impact of SinkLoss on evaluation metrics.** SinkLoss greatly improves fine details and edge sharpness, as measured by `SEE_k` metrics without affecting standard metrics (AbsRel or `delta_1`). With SinkLoss enabled during Stage 2 of our protocol, Marigold V2 outperforms PPD on `SEE_3`.
**Fig. 8. Qualitative comparison on in-the-wild images.** Marigold V2 preserves fine details better than the baselines across depth, surface normals, and albedo estimation. Rows 1, 2, and 6 are courtesy of cottonbro studio, VANNGO Ng, and Lucas Oliveira via Pexels, respectively.
Extensions and Dense Regression
Ablations show our dense‑regression extensions improve depth, see‑through depth, normals, and albedo.
Beyond relative depth we evaluate Marigold V2 on several dense regression tasks, each with a focused ablation to isolate the contribution of our training recipe.
We freeze the pretrained affine‑invariant depth prior and learn a tiny LoRA that adapts it to metric depth using only the sparse depth points available at test time.
See‑through depth variant reduces AbsRel from 13.66 % to 8.17 % and raises $\delta$₁ from 84.0 % to 92.7 % on the LayeredDepth‑Syn validation set.
Table 8
Albedo variant attains the highest PSNR (20.78 dB) and the lowest LPIPS (0.195) among all compared methods.
Table 11
Surface‑normal estimation benefits from the same affine‑invariant prior; despite its simplicity it achieves the lowest RMSE across all four benchmarks (Table 7).
Backbone Transfer Ablation demonstrates that the proposed loss recipe generalizes beyond the Qwen image‑editing backbone: applying it to Stable Diffusion V1.5 and FLUX.2 klein yields comparable gains on the SEE metric (Table 5).
Stage‑2 Ablation isolates the impact of the second‑stage fine‑tuning; using the Soft Edge Error (SEEₖ) metric on HyperSim shows marked improvements in boundary quality and a reduction of flying‑pixel artifacts.
**Fig. 9. See-through depth qualitative comparison.** From left to right: input RGB, predictions of our base model and of our see-through model.
**Fig. 10.** **Qualitative results: surface normals and albedo.** Left: Input, angular $L_1$, $L_1$ + iREPA, and SinkLoss + iREPA. Right: Input and albedo prediction. Flowers image is courtesy of Rüveyda Akkaya via Pexels.