Crowdhuman Eval

crowdhuman-eval

CrowdHuman: A Benchmark for Detecting Human in a Crowd — Shuai Shao et al. (2018) (arXiv:1805.00123, 2018)

What this evaluates

Evaluates object detectors' ability to identify humans in highly crowded and heavily occluded scenes. It covers three annotation levels (full body, visible body, head) and assesses cross-dataset generalization for pedestrian and head detection tasks.

Datasets

• CrowdHuman — total 15000; splits: train (-1), val (-1), test (-1)

Metrics

• mMR (primary) — range: percent
  • Average log miss rate over false positives per image, computed over the range [10^{-2}, 10^{0}]. Lower values indicate better performance.
• AP — range: [0, 1]
  • Average Precision, standard object detection metric averaging precision over recall thresholds.
• Recall — range: percent
  • Percentage of ground truth human instances correctly detected.

Input / output format

Input: RGB images resized so the short edge is 800 pixels and the long edge is at most 1400 pixels. Ground truth bounding boxes for full body, visible body, or head regions.

Output: Predicted bounding boxes for each detected human instance.

Scoring recipe

def compute_mMR(predictions, ground_truth, fp_range=(0.01, 1.0)):
    miss_rates = []
    fps_per_image = []
    for img in images:
        mr, fp = evaluate_detection(img['gt'], img['pred'])
        miss_rates.append(mr)
        fps_per_image.append(fp)
    mask = (np.array(fps_per_image) >= fp_range[0]) & (np.array(fps_per_image) <= fp_range[1])
    mMR = np.mean(np.log10(np.array(miss_rates)[mask]))
    return mMR

Common pitfalls

• Test set annotations are not publicly released; evaluation must be performed via the official online server.
• Multi-scale training and testing are explicitly disabled to ensure fair comparisons across methods.
• Ignored regions in ground truth annotations are excluded from the mMR calculation.

Evidence (verbatim from paper)

We follow the evaluation metric used for Caltech, denoted as mMR, which is the average log miss rate over false positives per-image ranging in [10^{-2},10^{0}]. mMR is a good indicator for the algorithms applied in the real world applications. Results on ignored regions will not considered in the evaluation. Besides, Average Precision (AP) and recall of the algorithms are included for reference.

Citation

@misc{shao2018crowdhuman,
  title={CrowdHuman: A Benchmark for Detecting Human in a Crowd},
  author={Shuai Shao et al. (2018)},
  year={2018},
  note={arXiv:1805.00123}
}

• arXiv: 1805.00123