Finnwoodlands Eval

finnwoodlands-eval

FinnWoodlands Dataset — Lagos et al. (2023) (arXiv:2304.00793, 2023)

What this evaluates

Evaluates computer vision models on forest scene understanding, specifically testing instance segmentation, panoptic segmentation, and depth completion in unstructured, densely populated natural environments.

Datasets

• FinnWoodlands — total 300; splits: train (150), test (50); repo https://github.com/juanb09111/FinnForest.git

Metrics

• mAP@50 (primary) — range: [0, 1]
  • Mean Average Precision at IoU threshold 0.50, averaged across all object classes.
• mIoU — range: [0, 1]
  • Mean Intersection over Union computed across all semantic classes.
• PQ — range: [0, 1]
  • Panoptic Quality, calculated as the product of Segmentation Quality (SQ) and Recognition Quality (RQ) for both things and stuff categories.
• RMSE — range: other
  • Root Mean Square Error between predicted dense depth maps and ground truth depth maps.

Input / output format

Input: RGB images (and sparse depth maps for depth completion models).

Output: Instance/Panoptic: bounding boxes, class labels, and pixel-wise segmentation masks. Depth completion: fully dense depth maps.

Scoring recipe

# Instance/Panoptic Segmentation (COCO standard)
matches = match_predictions_to_ground_truth(predictions, ground_truth, iou_threshold=0.5)
mAP = compute_average_precision(matches)
mIoU = mean(compute_iou(pred_mask, gt_mask) for class in classes)
PQ = compute_panoptic_quality(predictions, ground_truth) # PQ = SQ * RQ
# Depth Completion
RMSE = sqrt(mean((pred_depth - gt_depth)**2))

Common pitfalls

• Dense forest scenes cause tree trunks to overlap closely, making instance segmentation masks inaccurate.
• Sparse depth inputs lack fine structural boundaries, leading to blurred depth completion outputs.
• Panoptic segmentation performance drops significantly on 'things' (individual trees) compared to 'stuff' (background).

Evidence (verbatim from paper)

We used the standard COCO evaluation metrics. More specifically, we computed the mean Average Precision (mAP) for evaluating instance segmentation, Mean Intersection over Union (mIoU) for semantic segmentation, Panoptic Quality (PQ), Segmentation Quality (SQ), and Recognition Quality (RQ) for panoptic segmentation. We computed the Root Mean Square Error (RMSE) to evaluate the depth completion task.

Citation

@misc{lagos2023finnwoodlands,
  title={FinnWoodlands Dataset},
  author={Lagos et al. (2023)},
  year={2023},
  note={arXiv:2304.00793}
}

• arXiv: 2304.00793