Reliability-Aware Monocular Depth Supervision for Sparse-View Neural Reconstruction

Given sparse forward-facing RGB inputs, we estimate a monocular depth prior with Depth Anything V2 and align it to metric depth via scale-shift fitting. An RGB-only baseline yields per-pixel photometric error, which builds a reliability mask that keeps low-error regions. The aligned prior is applied only to reliable pixels through a masked depth loss while the model is jointly optimized with the photometric loss.

Abstract

Sparse-view neural reconstruction is challenging in outdoor driving scenes, where cameras move along a narrow forward-facing trajectory and provide limited multi-view overlap. Monocular depth estimators can provide dense geometric priors, but their predictions are noisy and not uniformly reliable across image regions. We use Depth Anything V2 (DA-V2) as a dense monocular depth prior, align its predictions to metric depth via per-image scale-shift fitting on sparse anchors (LiDAR on KITTI, COLMAP on Bicycle), and apply depth supervision selectively through photometric masks generated from an RGB-only baseline. We evaluate this strategy on two representative scene representations: Mip-NeRF-360 and Splatfacto. Since the mask modifies an existing depth loss rather than introducing one, we compare against global (unmasked) supervision as the primary baseline. Across KITTI sequences 00/02/05, the mask yields +0.44 to +0.70 dB PSNR for Splatfacto over global supervision at tied or better RMSE, and no change on Mip-NeRF-360 — the mask primarily enhances rendering fidelity rather than geometry. Matched-ratio ablations confirm the gains come from selecting reliable low-error regions rather than simply reducing the number of depth-supervised pixels. Without any ground-truth depth, our mask also outperforms three LiDAR-supervised baselines. On the object-centric Mip-NeRF-360 Bicycle scene, depth supervision improves geometry but hurts rendering quality when multi-view coverage is already strong.

Method

Scale-Aligned Monocular Depth Prior

DA-V2 predicts relative depth, so we align each prediction to a metric reference before training. For KITTI the reference is valid LiDAR depth; for the Bicycle scene it is sparse COLMAP points. Given a monocular prediction \(d_m(u)\) and reference depth \(d_r(u)\) over valid anchor pixels \(u \in \Omega\), we solve a per-image least-squares scale-shift fit:

The aligned prior is \(\hat{d}(u) = s^{*} d_m(u) + t^{*}\), clipped to the 80 m evaluation range on KITTI.

Photometric Reliability Mask

Rather than supervising every pixel, we only supervise where the RGB-only baseline is already reliable. From a baseline render \(\hat{I}\) and the ground-truth image \(I\), we compute a per-pixel photometric error and threshold it at \(\tau\):

High photometric error flags unreliable regions — dynamic objects, reflective surfaces, occlusion boundaries, and sky — where a depth loss would push the model toward wrong geometry. Low-error pixels are already consistent with the RGB observations, so the aligned prior acts as a stabilizing regularizer. The mask is computed once from the RGB-only baseline and held fixed throughout depth-supervised training.

Masked Depth Supervision Objective

The effective supervision mask combines the photometric mask with the depth-validity mask \(D(u)\): \(M_\text{eff} = M_\tau \wedge D\). The depth loss is gated by \(M_\text{eff}\), while the RGB loss is evaluated over the full image:

Setting \(\tau=1.0\) gives \(M_\tau \equiv 1\) — global (unmasked) depth supervision. This condition, not the depth-free \(\lambda_\text{depth}=0\) baseline, is the primary baseline for judging the mask's contribution: it isolates the effect of where depth is supervised from the effect of whether depth is supervised at all.

Training views are subsampled from KITTI Odometry to simulate sparse-view capture. Fixed photometric masks are generated from baseline renderings by thresholding the photometric error; DA-V2 predictions are scale-shift aligned to produce metric depth priors.

Depth Prior Quality

Before using DA-V2 as supervision, we evaluate the aligned depth against valid KITTI LiDAR pixels. The mean absolute error of alignment is 4.22 m. Sweeping the number of LiDAR anchors used for scale-shift fitting shows the 2-DOF optimization saturates long before typical LiDAR density is reached — alignment MAE is stable from ~95k anchors down to 500 per frame, degrading only under extreme sparsity. The 4.22 m error floor reflects DA-V2's intrinsic local structural limitations rather than calibration artifacts.

Photometric Error as a Reliability Proxy

Because photometric and depth errors are measured against different references, low photometric error does not by construction imply low depth error. We validate the mask directly against GT LiDAR. Depth RMSE is 34–39% lower inside the mask at every threshold, and inside-mask RMSE increases monotonically with τ (r=1.0). Pixel-level differences are highly significant (p<10⁻⁵⁰); the frame-level paired t-test gives p=0.047 (n=8).

Results

Splatfacto — KITTISeq02 every-2

Splatfacto gains clearly from monocular depth supervision. Using any DA-V2 depth prior — even without a mask — improves PSNR (14.903 → 15.494 for global) and cuts RMSE by ~80% (0.542 → 0.101). Masking the depth loss at τ=0.18 adds a further +0.44 dB PSNR at tied RMSE. In short: the RMSE drop comes from using a depth prior at all, and masking is what improves rendering fidelity. Over three seeds, masked 15.30±0.57 vs global 15.07±0.38 dB (+0.23 dB mean); the −80% RMSE improvement over RGB-only is consistent across seeds.

Matched-Ratio Mask Ablation

To test whether the benefit comes from selecting reliable pixels or simply using fewer pixels, we compare against high-error and random masks with the identical per-frame pixel budget. The low-error mask wins on every metric — the improvement is not from fewer supervised pixels.

Multi-Sequence Generalization

The masked-vs-global advantage generalizes beyond KITTISeq02. Across KITTI 00/027, 02/034, and 05/018, the low-error mask (τ=0.18) outperforms global supervision on every metric, with PSNR gains of +0.44, +0.64, and +0.70 dB at tied or better RMSE.

Comparison to Depth-Supervised Baselines

We benchmark against four depth-supervised methods on KITTISeq02 Frag. 034, all retrained for 50k iterations on our sparse split. The top group uses no GT depth; the bottom group uses GT LiDAR. Our masked DA-V2 approach outperforms three LiDAR-supervised baselines — DNGaussian, DepthRegGS, and SparseGS — which target indoor or object-centric scenes. Only DN-Splatter leads (+0.29 dB PSNR, −0.12 LPIPS), using GT LiDAR unavailable at test time; its margin reflects depth-source quality rather than masking.

Comparison to a Depth-Inconsistency Mask

We compare our photometric mask against a static, one-shot approximation of the Depth-Inconsistency Mask (DIM) at a matched reliable-pixel fraction. Ratio = MAE(outside) / MAE(inside): >1 means the mask isolates accurate DA-V2 depth; <1 means no separation. Our mask separates accurate from inaccurate depth (ratio 1.17) while the DIM proxy does not (0.87). This is expected: DIM keys on distortions that depth-supervised training itself induces, so a one-shot proxy has nothing to key on. Both are computed once and held fixed, so the comparison is apples-to-apples.

Effect of depth-loss weight λ and reliability threshold τ on Splatfacto. Depth supervision improves geometric consistency and the reconstruction of thin structures, with fewer smeared artifacts around object boundaries and foreground vehicles.

Mip-NeRF-360 — KITTISeq02 every-2

The implicit density field is far more sensitive to noisy monocular depth. The best PSNR (20.607 dB) comes from global (τ=1.0) supervision — the optimal configuration bypasses the mask entirely. The best masked setting trails global by 0.026 dB, well within multi-seed variance (20.57±0.04 dB). All depth-supervised settings, including global, degrade SSIM, LPIPS, and geometry relative to RGB-only (RMSE rises from 2.703 to ≥3.532).

Mip-NeRF-360 — KITTISeq05

On KITTISeq05, both global and masked DA-V2 supervision degrade rendering and geometry compared with RGB-only training. Masking recovers 0.22 dB PSNR over global (16.612 vs 16.389), so it mitigates rather than reverses the harm — monocular depth priors are not reliably beneficial for the implicit density representation.

Effect of depth-loss weight λ and reliability threshold τ on Mip-NeRF-360 reconstruction of finer details such as the street pole. Masked depth supervision provides only weak, unstable regularization for the implicit density field.

PSNR as a function of λ on KITTISeq02 across six thresholds τ. Splatfacto peaks sharply at τ=0.18, λ=0.10 (15.93 dB); Mip-NeRF-360 rises only gently with λ.

Object-Centric Bicycle

When multi-view coverage is already strong, the story flips. On the Mip-NeRF-360 Bicycle scene, RGB-only Splatfacto gives the best rendering quality (17.731 PSNR), while depth supervision consistently lowers RMSE (1.479 → 0.722) at the cost of PSNR / SSIM / LPIPS — depth regularization over-constrains an already well-posed reconstruction. Global supervision gives the best PSNR among depth-supervised runs (17.593 vs 17.466 for the best mask), confirming the mask's rendering gain is specific to sparse forward-facing trajectories.

Ablation of λ and τ for object-centric Splatfacto reconstruction. Depth supervision improves RMSE but generally reduces RGB rendering quality when multi-view coverage is already strong.

Takeaways

Monocular depth priors are most useful for explicit Gaussian representations in under-constrained, forward-facing sparse-view scenes, and less reliable for implicit NeRF-style density fields. The RMSE drop for Splatfacto comes from using a depth prior at all; masking is what improves rendering fidelity, adding +0.44 to +0.70 dB PSNR over global supervision across KITTI 00/02/05 at tied or better RMSE, with no gain for Mip-NeRF-360 or the object-centric Bicycle scene. Without any ground-truth depth, our mask also outperforms three LiDAR-supervised baselines (DNGaussian, DepthRegGS, SparseGS). Our numbers use DA-V2, but the explicit-vs-implicit architectural insights are prior-agnostic; sweep values (τ, λ) are dataset-specific. Future work: better scale alignment, uncertainty-aware masks, and adaptive depth-loss weighting.

BibTeX

@misc{chu2026reliability,
    title         = {Reliability-Aware Monocular Depth Supervision for Sparse-View Neural Reconstruction},
    author        = {Wei-Teng Chu and Yashasvini Gopalan and Changju Yuan},
    year          = {2026},
    eprint        = {2607.02554},
    archivePrefix = {arXiv},
    primaryClass  = {cs.CV},
    url           = {https://arxiv.org/abs/2607.02554}
}

Acknowledgements

This project builds on Depth Anything V2, Mip-NeRF 360, Nerfstudio / Splatfacto, and 3D Gaussian Splatting.