Scatspotter Eval

scatspotter-eval

"ScatSpotter" 2024 -- A Distributed Dog Poop Detection Dataset — Crall (2024) (arXiv:2412.16473, 2024)

What this evaluates

Evaluates object detection and instance segmentation capabilities on real-world images of dog feces. It specifically probes model robustness to camouflage, occlusion, varying lighting conditions, and small object detection in outdoor urban environments.

Datasets

• ScatSpotter — total 6648; splits: train (5747), val (691), test (30)

Metrics

• mAP (primary) — range: [0, 1]
  • Mean Average Precision computed over IoU thresholds 0.50:0.95 for both bounding box and polygon segmentation tasks, following the standard COCO evaluation protocol.

Input / output format

Input: High-resolution RGB JPEG images (typically 4032x3024) captured in outdoor urban environments.

Output: COCO JSON format containing bounding boxes and/or polygon coordinates for each detected instance, along with class labels and confidence scores.

Scoring recipe

from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval

def compute_mAP(gt_path, pred_path):
    coco_gt = COCO(gt_path)
    coco_dt = coco_gt.loadRes(pred_path)
    img_ids = sorted(coco_gt.getImgIds())
    eval = COCOeval(coco_gt, coco_dt, 'bbox')
    eval.params.imgIds = img_ids
    eval.evaluate()
    eval.accumulate()
    eval.summarize()
    return eval.stats[0]  # mAP @ IoU 0.50:0.95

Common pitfalls

• Approximately 65% of images are negatives (no annotations) due to the before/after/negative protocol, which can mislead models if treated as positive samples.
• Shadows and low-light conditions frequently cause both annotation errors (SAM failures) and detection failures.
• Small object size and camouflage against natural backgrounds (leaves, dirt, snow) significantly reduce recall.

Evidence (verbatim from paper)

Baseline models (ViT, Mask R-CNN) achieve 0.858 mAP on validation, but failure cases highlight persistent difficulties in detecting camouflaged or small poop objects.

Citation

@misc{crall2024scatspotter,
  title={"ScatSpotter" 2024 -- A Distributed Dog Poop Detection Dataset},
  author={Crall (2024)},
  year={2024},
  note={arXiv:2412.16473}
}

• arXiv: 2412.16473