Co Semdepth Eval

co-semdepth-eval

Co-SemDepth: Fast Joint Semantic Segmentation and Depth Estimation on Aerial Images — AlaaEldin et al. (2025) (arXiv:2503.17982, 2025)

What this evaluates

Evaluates a joint deep learning architecture for simultaneous monocular depth estimation and semantic segmentation on aerial drone imagery, measuring prediction accuracy and inference speed against single-task and joint baselines.

Datasets

• MidAir — total 420000; splits: train (-1), val (-1), test (-1)
• Aeroscapes — total 3269; splits: train (-1), test (-1)

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union across all semantic classes. Computed as the average of per-class IoU (intersection over union of predicted and ground truth masks).
• RMSE — range: other
  • Linear root mean square error between predicted and ground truth depth maps.
• AbsRelErr — range: other
  • Absolute relative error, computed as the mean of |predicted - ground truth| / ground truth over valid pixels.
• δ<1.25 — range: percent
  • Percentage of pixels where max(predicted/ground_truth, ground_truth/predicted) < 1.25.
• δ<1.25^2 — range: percent
  • Percentage of pixels where max(predicted/ground_truth, ground_truth/predicted) < 1.25^2.
• δ<1.25^3 — range: percent
  • Percentage of pixels where max(predicted/ground_truth, ground_truth/predicted) < 1.25^3.
• Inference Time — range: other
  • Average time to process one frame, measured in milliseconds per frame (ms/f).

Input / output format

Input: Monocular RGB images (384x384 for MidAir, 1280x720 for Aeroscapes). For joint evaluation, video sequences are used to leverage temporal information.

Output: Predicted depth maps and semantic segmentation maps. Depth and semantic maps are initially predicted at half the input resolution and then upsampled to original resolution using nearest-neighbor interpolation.

Scoring recipe

# Depth metrics
valid_pixels = ground_truth_depth > 0
pred_depth = pred_depth[valid_pixels]
gt_depth = gt_depth[valid_pixels]
rmse = np.sqrt(np.mean((pred_depth - gt_depth)**2))
abs_rel_err = np.mean(np.abs(pred_depth - gt_depth) / gt_depth)
ratios = np.maximum(pred_depth / gt_depth, gt_depth / pred_depth)
delta1 = np.mean(ratios < 1.25) * 100
delta2 = np.mean(ratios < 1.25**2) * 100
delta3 = np.mean(ratios < 1.25**3) * 100

# Semantic metric
per_class_iou = []
for class_id in range(num_classes):
    pred_mask = (pred_semantic == class_id)
    gt_mask = (gt_semantic == class_id)
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    iou = intersection / union if union > 0 else 0
    per_class_iou.append(iou)
miou = np.mean(per_class_iou)

Common pitfalls

• Depth values are capped at 80.0 meters during evaluation, which truncates long-range predictions.
• MidAir's original 14 semantic classes are mapped to 7 classes (e.g., Ground Vegetation, Rocky Ground, Dirt Ground → Land) before evaluation.
• Aeroscapes lacks depth annotations, so it is only used for evaluating the single-task semantic segmentation baseline (M4Semantic), not the joint architecture.
• Inference is performed at half resolution and upsampled via nearest-neighbor, which may slightly reduce accuracy but is required for memory constraints.

Evidence (verbatim from paper)

To quantitatively evaluate the depth prediction results, we consider the commonly used evaluation metrics in prior works[[11], [3], [16]]. These include the linear root mean square error (RMSE), the absolute relative error, and accuracy under a threshold. For semantic segmentation, we use the commonly used mean Intersection over Union $mIoU$ metric. The Inference Time (Inf. Time) is computed in milliseconds per frame (ms/f).

Citation

@misc{alaaeldin2025cosemdepth,
  title={Co-SemDepth: Fast Joint Semantic Segmentation and Depth Estimation on Aerial Images},
  author={AlaaEldin et al. (2025)},
  year={2025},
  note={arXiv:2503.17982}
}

• arXiv: 2503.17982