Reliability-Aware Monocular Depth Supervision for Sparse-View Neural Reconstruction

Wei-Teng (Wayne) Chu^* · Yashasvini Gopalan^* · Changju Yuan^*

Stanford University · ^*Equal contribution

CS231N: Deep Learning for Computer Vision — Final Project

Given sparse forward-facing RGB inputs, we estimate a monocular depth prior with Depth Anything V2 and align it to metric depth via scale-shift fitting. An RGB-only baseline yields per-pixel photometric error, which builds a reliability mask that keeps low-error regions. The aligned prior is applied only to reliable pixels through a masked depth loss while the model is jointly optimized with the photometric loss.

Abstract

Sparse-view neural reconstruction is challenging in outdoor driving scenes, where cameras move along a narrow forward-facing trajectory and provide limited multi-view overlap. Although monocular depth estimators can provide dense geometric priors, their predictions are noisy and not uniformly reliable across image regions. In this work, we study monocular depth supervision for sparse-view neural reconstruction. We use Depth Anything V2 (DA-V2) as a dense monocular depth prior, align its predictions to metric depth using scale-shift fitting, and apply depth supervision selectively through photometric masks generated from an RGB-only baseline model. We evaluate this strategy on two representative scene representations: Mip-NeRF-360 and Splatfacto. On KITTISeq02 under an every-2 sparse-view setting, masked depth supervision gives only marginal gains for Mip-NeRF-360, while Splatfacto benefits more clearly, improving PSNR from 14.903 to 15.932 and reducing RMSE from 0.542 to 0.100. Matched-ratio mask ablations and additional KITTISeq05 experiments show that the gains come from selecting reliable low-error regions rather than simply reducing the number of depth-supervised pixels. Overall, monocular depth priors are useful for under-constrained sparse-view reconstruction, but should be applied selectively and with moderate weighting.

Method

Scale-Aligned Monocular Depth Prior

DA-V2 predicts relative depth, so we align each prediction to a metric reference before training. For KITTI the reference is valid LiDAR depth; for the Bicycle scene it is sparse COLMAP points. Given a monocular prediction \(d_m(u)\) and reference depth \(d_r(u)\) over valid anchor pixels \(u \in \Omega\), we solve a per-image least-squares scale-shift fit:

\[ s^{*},\, t^{*} = \operatorname*{arg\,min}_{s,t} \sum_{u \in \Omega} \big( s\, d_m(u) + t - d_r(u) \big)^2 \]

The aligned prior is \(\hat{d}(u) = s^{*} d_m(u) + t^{*}\), clipped to the 80 m evaluation range on KITTI.

Photometric Reliability Mask

Rather than supervising every pixel, we only supervise where the RGB-only baseline is already reliable. From a baseline render \(\hat{I}\) and the ground-truth image \(I\), we compute a per-pixel photometric error and threshold it at \(\tau\):

\[ e(u) = \frac{1}{3} \sum_{c \in \{R,G,B\}} \big| \hat{I}_c(u) - I_c(u) \big|, \qquad M_\tau(u) = \mathbb{1}\!\left[\, e(u) < \tau \,\right] \]

High photometric error flags unreliable regions — dynamic objects, reflective surfaces, occlusion boundaries, and sky — where a depth loss would push the model toward wrong geometry. Low-error pixels are already consistent with the RGB observations, so the aligned prior acts as a stabilizing regularizer. The mask is computed once from the RGB-only baseline and held fixed throughout depth-supervised training.

Masked Depth Supervision Objective

The effective supervision mask combines the photometric mask with the depth-validity mask \(D(u)\): \(M_\text{eff} = M_\tau \wedge D\). The depth loss is gated by \(M_\text{eff}\), while the RGB loss is evaluated over the full image:

\[ \mathcal{L} = \mathcal{L}_\text{rgb} + \lambda_\text{depth}\, \mathcal{L}_\text{depth}, \qquad \mathcal{L}_\text{depth} = \frac{1}{N} \sum_{u} M_\text{eff}(u) \big( \hat{d}(u) - \hat{d}_\text{prior}(u) \big)^2 \]

Training views are subsampled from KITTI Odometry to simulate sparse-view capture. Fixed photometric masks are generated from baseline renderings by thresholding the photometric error; DA-V2 predictions are scale-shift aligned to produce metric depth priors.

Results

Splatfacto — KITTISeq02 every-2

Splatfacto gains clearly from masked depth supervision. The best rendering quality is at τ=0.18, λ=0.10 (+1.03 dB PSNR, RMSE down from 0.542 to 0.100). Increasing λ to 0.15 improves RMSE further but trades away PSNR — a geometry–photometry tradeoff.

Setting (τ, λ)	PSNR ↑	SSIM ↑	LPIPS ↓	RMSE ↓
RGB-only	14.903	0.433	0.446	0.542
0.16, 0.10	15.452	0.454	0.434	0.106
0.18, 0.10 (ours)	15.932	0.477	0.408	0.100
0.18, 0.15	15.588	0.459	0.436	0.096
1.00, 0.10	15.494	0.448	0.434	0.101

Matched-Ratio Mask Ablation

To test whether the benefit comes from selecting reliable pixels or simply using fewer pixels, we compare against high-error and random masks with the identical per-frame pixel budget. The low-error mask wins on every metric — the improvement is not from fewer supervised pixels.

Mask type (λ=0.10)	PSNR ↑	SSIM ↑	LPIPS ↓	RMSE ↓
High-error, matched	14.932	0.437	0.455	0.111
Random, matched (3 seeds)	15.036	0.442	0.456	0.109
Low-error, τ=0.18 (ours)	15.932	0.477	0.408	0.100

Splatfacto lambda by tau qualitative ablation grid

Effect of depth-loss weight λ and reliability threshold τ on Splatfacto. Depth supervision improves geometric consistency and the reconstruction of thin structures, with fewer smeared artifacts around object boundaries and foreground vehicles.

Mip-NeRF-360 — KITTISeq02 every-2

The implicit density field is far more sensitive to noisy monocular depth. Masked supervision yields only marginal PSNR gains (best +0.23 dB) and does not improve geometry; SSIM, LPIPS, and RMSE slightly degrade.

Setting (τ, λ)	PSNR ↑	SSIM ↑	LPIPS ↓	RMSE ↓
RGB-only	20.378	0.601	0.409	2.703
0.18, 0.10	20.384	0.594	0.416	3.532
1.00, 0.15	20.607	0.595	0.412	3.580

Mip-NeRF-360 lambda by tau qualitative ablation grid

Effect of depth-loss weight λ and reliability threshold τ on Mip-NeRF-360 reconstruction of finer details such as the street pole. Masked depth supervision provides only weak, unstable regularization for the implicit density field.

PSNR as a function of λ on KITTISeq02 across six thresholds τ. Splatfacto peaks sharply at τ=0.18, λ=0.10 (15.93 dB); Mip-NeRF-360 rises only gently with λ.

Object-Centric Bicycle

When multi-view coverage is already strong, the story flips. On the Mip-NeRF-360 Bicycle scene, RGB-only Splatfacto gives the best rendering quality, while depth supervision consistently lowers RMSE (1.479 → 0.722) at the cost of PSNR / SSIM / LPIPS — depth regularization over-constrains an already well-posed reconstruction.

Bicycle lambda by tau qualitative ablation grid

Ablation of λ and τ for object-centric Splatfacto reconstruction. Depth supervision improves RMSE but generally reduces RGB rendering quality when multi-view coverage is already strong.

Takeaways

Monocular depth priors are most useful for explicit Gaussian representations in under-constrained, forward-facing sparse-view scenes, and less reliable for implicit NeRF-style density fields. A moderate depth weight on a low-error reliability mask improves both rendering and geometry for Splatfacto, while too-strong supervision sacrifices RGB quality for depth metrics. Future work: better scale alignment, uncertainty-aware masks, and adaptive depth-loss weighting.

Acknowledgements

This project builds on Depth Anything V2, Mip-NeRF 360, Nerfstudio / Splatfacto, and 3D Gaussian Splatting.