Urban Tree Detection Eval

urban-tree-detection-eval

OAM-TCD: A globally diverse dataset of high-resolution tree cover maps — Veitch-Michaelis et al. (2024) (arXiv:2407.11743, 2024)

What this evaluates

Evaluates the capability of segmentation models to detect and delineate individual tree crowns and canopy coverage from high-resolution aerial imagery across diverse urban and tropical environments.

Datasets

• Zurich Municipal Tree Inventory & Swisstopo Imagery — total 71000; splits: test (71000)
• WeRobotics Open AI Challenge (Tonga) — total 13402; splits: test (13402)

Metrics

• IoU — range: [0, 1]
  • Intersection over Union: the area of overlap between predicted and ground truth canopy masks divided by the area of their union.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified pixels (tree vs. non-tree) in the semantic segmentation mask.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for semantic canopy segmentation.
• Recall (primary) — range: [0, 1]
  • Proportion of ground truth tree keypoints matched to either predicted tree instances or predicted canopy regions. TP / (TP + FN).

Input / output format

Input: Tiled high-resolution aerial orthomosaics (10 cm/px resolution) processed in overlapping patches.

Output: Prediction maps for semantic canopy coverage and instance-level tree keypoints/polygons, with overlapping instances merged using heuristics.

Scoring recipe

# Semantic (Canopy)
iou = np.sum(pred & gt) / np.sum(pred | gt)
accuracy = np.mean(pred == gt)
f1 = 2 * (precision * recall) / (precision + recall)

# Instance/Keypoint
matched = 0
for gt_kp in gt_keypoints:
    if any(pred_inst.contains(gt_kp) or pred_canopy.contains(gt_kp)):
        matched += 1
recall = matched / len(gt_keypoints)

Common pitfalls

• Models detect non-municipal trees not present in the ground truth inventory, making false positive rate calculation impossible.
• Tiled inference requires custom heuristics to merge overlapping predictions across tile boundaries.
• Clustered trees are often predicted as single groups, artificially lowering instance-level recall.

Evidence (verbatim from paper)

The Federal Institute of Topography in Switzerland (Swisstopo) produces 3-year aerial surveys of the entire country, at 10 cm resolution. By combining 2022 imagery with a up-to-date municipal tree inventory from the city of Zurich, we can assess model performance against an accurate ground truth. We report a recall of 0.36 for keypoints that were matched to tree labels and 0.73 when also including canopy regions. This dataset is a tree detection benchmark. The input is a 325 ha orthomosaic captured over the Kingdom of Tonga. Tree centers for four species were annotated by humans (13402 total keypoints); not all trees in the image are labelled and our model predicts around twice this many instances over the orthomosaic. We report a recall of 0.64 for keypoints that were matched to tree labels and 0.94 when also including canopy regions.

Citation

@misc{veitchmichaelis2024oamtcd,
  title={OAM-TCD: A globally diverse dataset of high-resolution tree cover maps},
  author={Veitch-Michaelis et al. (2024)},
  year={2024},
  note={arXiv:2407.11743}
}

• arXiv: 2407.11743