Roboflow 100 Eval

roboflow-100-eval

Roboflow 100: A Rich, Multi-Domain Object Detection Benchmark — Ciaglia et al. (2022) (arXiv:2211.13523, 2022)

What this evaluates

Evaluates object detection models' ability to generalize across diverse, real-world, domain-specific visual tasks. It probes fine-tuning performance and zero-shot transfer capabilities on crowdsourced, practitioner-curated datasets spanning multiple imaging modalities.

Datasets

• Roboflow 100 — total 224714; splits: train (-1), val (-1), test (-1)

Metrics

• mAP@.50 (primary) — range: [0, 1]
  • Mean Average Precision at Intersection over Union (IoU) threshold of 0.50. Computed per class and averaged across all classes in a dataset or category.
• mAP@.50:.95 — range: [0, 1]
  • Mean Average Precision averaged over IoU thresholds from 0.50 to 0.95 in steps of 0.05. Used primarily for the GLIP zero-shot model evaluation.

Input / output format

Input: RGB images resized to 640x640 pixels with corresponding bounding box annotations and class labels.

Output: Predicted bounding boxes with class labels and confidence scores.

Scoring recipe

def compute_map(predictions, ground_truth, iou_thresh=0.5):
    # 1. Group predictions and ground truth by class
    # 2. For each class, sort predictions by confidence score descending
    # 3. Match each prediction to the highest IoU ground truth box (IoU >= iou_thresh)
    # 4. Mark matches as True Positives, unmatched as False Positives
    # 5. Compute precision and recall at each threshold, build PR curve
    # 6. Interpolate AP at 11 recall points (or use modern trapezoidal rule)
    # 7. Average AP across all classes to yield mAP@.50
    return mAP_value

Common pitfalls

• Split ratios are not standardized across the 100 datasets; they follow original authors' splits unless underrepresented, complicating cross-dataset comparison.
• Class names vary in descriptiveness, which heavily impacts zero-shot models like GLIP that rely on text prompts for detection.
• Bounding box sizes differ drastically across domains (e.g., aerial vs. documents), affecting detection difficulty and metric interpretation.

Evidence (verbatim from paper)

We trained both models with default hyperparameters for 100 epochs at 640x640 resolution. We recorded the average mAP@.50 value for the YOLOv5 and YOLOv7 models and the mAP@.50:.95 for the GLIP model for each category.

Citation

@misc{ciaglia2022roboflow100,
  title={Roboflow 100: A Rich, Multi-Domain Object Detection Benchmark},
  author={Ciaglia et al. (2022)},
  year={2022},
  note={arXiv:2211.13523}
}

• arXiv: 2211.13523