Wildbe Eval

wildbe-eval

Wild Berry image dataset collected in Finnish forests and peatlands using drones — Riz et al. (2024) (arXiv:2405.07550, 2024)

What this evaluates

Evaluates object detection algorithms on drone-captured images of wild berries in cluttered, dynamic forest environments. It probes localization and classification capabilities under severe lighting variations, occlusion, and cross-domain transfer settings (different areas, cameras, and datasets).

Datasets

• WildBe — total 3516; splits: train (3164), val (176), test (176)

Metrics

• Average Precision (AP) (primary) — range: percent
  • COCO evaluation metric computing precision-recall curve over IoU thresholds (0.50:0.05:0.95) and averaging AP across thresholds. Also reported for small (≤32×32 px) and medium (32×32 to 96×96 px) objects.
• Instance-weighted Average Precision (WAvg) — range: percent
  • Average AP weighted by the number of ground-truth instances per class, used to mitigate performance masking due to class imbalance in multi-class settings.

Input / output format

Input: RGB images captured by drones, annotated with bounding boxes and class labels (bilberry, cloudberry, crowberry, lingonberry, or generic 'berry').

Output: Bounding box coordinates and predicted class labels for each detected object in the image.

Scoring recipe

coco_gt = load_annotations(gold_boxes, gold_classes)
coco_dt = load_predictions(pred_boxes, pred_classes, scores)
coco_eval = COCOeval(coco_gt, coco_dt, 'bbox')
coco_eval.evaluate()
coco_eval.accumulate()
ap = coco_eval.summaries[0]  # AP@[.50:.95]
ap_s = coco_eval.summaries[1]  # AP small
ap_m = coco_eval.summaries[2]  # AP medium
class_aps = [coco_eval.summaries[0] for cid in classes]
wavg = sum(ap * count for ap, count in zip(class_aps, class_counts)) / sum(class_counts)

Common pitfalls

• No large detections (≥96×96 px) exist in WildBe, so AP_large is always 0 or undefined.
• WAvg is explicitly recommended over simple Avg AP when class distributions are highly imbalanced (e.g., bilberry vs. lingonberry) to prevent class bias.
• Transfer learning folds are defined by geographic area or camera sensor, not random image splits; models must be trained on 3/4 of folds and tested on the held-out fold.

Evidence (verbatim from paper)

We use COCO evaluation and report results in terms of Average Precision (AP). For the single-class experiments, we also report the AP for small ($ extrm{AP}{ extrm{S}}$) and medium ($ extrm{AP}{ extrm{M}}$) detections. COCO defines detections as “small” if they occupy up to $32 imes 32$ pixels, “medium” if they occupy between $32 imes 32$ and $96 imes 96$ pixels, “large” otherwise. No large detections are present in WildBe. For the multi-class experiments, we report the per-class AP, the average AP (Avg), and the instance-weighted average AP (WAvg).

Citation

@misc{riz2024wildberry,
  title={Wild Berry image dataset collected in Finnish forests and peatlands using drones},
  author={Riz et al. (2024)},
  year={2024},
  note={arXiv:2405.07550}
}

• arXiv: 2405.07550