Depthcues Eval

depthcues-eval

DepthCues: Evaluating Monocular Depth Perception in Large Vision Models — Danier et al. (2024) (arXiv:2411.17385, 2024)

What this evaluates

Evaluates whether large vision models inherently understand human monocular depth cues (e.g., occlusion, perspective, texture gradient) through classification tasks, and measures their downstream monocular depth estimation performance on standard datasets.

Datasets

• DepthCues — total ?; splits: train (-1), test (-1)
• NYUv2 — total ?; splits: test (-1)
• DIW — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified depth cue instances per task. Models are ranked by average accuracy across the six monocular depth cues.
• NYUv2 accuracy — range: percent
  • Percentage of pixels where the ratio between predicted and ground-truth depth is less than 1.25.
• WHDR — range: percent
  • Weighted Human Disagreement Rate, measuring the percentage of pixels where the predicted depth differs from the ground truth by more than a threshold, weighted by human disagreement.

Input / output format

Input: RGB images (optionally with mask channels concatenated for baseline verification). For depth estimation tasks, raw images are provided.

Output: Per DepthCues cue: class label prediction. For depth estimation: predicted depth map or per-pixel depth values.

Scoring recipe

# DepthCues cue accuracy
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
cue_accuracy = (correct / len(predictions)) * 100

# NYUv2 accuracy
ratio = predicted_depth / ground_truth_depth
nyu_accuracy = (np.sum(ratio < 1.25) / predicted_depth.size) * 100

# DIW WHDR
# Standard WHDR calculation based on pixel-wise depth errors and human disagreement weights
whdr = calculate_whdr(predicted_depth, ground_truth_depth)

Common pitfalls

• Masks used in some baselines are only for feature construction verification and are not fed to the probes; assuming they leak task information is incorrect.
• Texture-grad cue uses synthetic data, which lowers its correlation with other cues and should not be directly compared to real-image cues without accounting for domain shift.
• DepthCues provides only image-level sparse annotations, not dense depth maps, so it should not be confused with dense depth estimation benchmarks.

Evidence (verbatim from paper)

For NYUv2, we report accuracy as the percentage of pixels where the ratio between predicted and ground-truth depth is less than 1.25. ... As before, we report accuracy for NYUv2, and use Weighted Human Disagreement Rate (WHDR) to measure performance on DIW.

Citation

@misc{danier2024depthcues,
  title={DepthCues: Evaluating Monocular Depth Perception in Large Vision Models},
  author={Danier et al. (2024)},
  year={2024},
  note={arXiv:2411.17385}
}

• arXiv: 2411.17385