Posebench Robustness Eval

posebench-robustness-eval

PoseBench: Benchmarking the Robustness of Pose Estimation Models under Corruptions — Ma et al. (2024) (arXiv:2406.14367, 2024)

What this evaluates

Evaluates the robustness of human and animal pose estimation models when subjected to real-world image corruptions such as blur, noise, compression, lighting changes, and occlusion masks. It measures how much model accuracy degrades relative to clean-image performance across varying corruption severities.

Datasets

• COCO-C — total 5000; splits: val (5000)
• OCHuman-C — total 5081; splits: val (5081)
• AP10K-C — total 10015; splits: val (10015)

Metrics

• mAP — range: [0, 1]
  • Mean Average Precision, computed as the average of Average Precision (AP) scores at IoU thresholds from 0.5 to 0.95 with a step of 0.05, following COCO keypoint evaluation standards.
• mAR — range: [0, 1]
  • Mean Average Recall, computed analogously to mAP but measuring the maximum recall achievable across different numbers of detected instances per image.
• mRR (primary) — range: [0, 1]
  • Mean Relative Robustness, defined as the average of relative robustness scores across C corruption types. For each corruption c, RR_c is the mean over 5 severity levels of (1 - (mAP_clean - mAP_c,s) / mAP_clean), which simplifies to the ratio of corrupted mAP to clean mAP.

Input / output format

Input: Corrupted images (10 corruption types × 5 severity levels) with ground truth keypoint annotations for each person/animal instance.

Output: Predicted keypoint coordinates (x, y) and confidence scores for each detected instance.

Scoring recipe

def compute_mRR(pred_clean, gt_clean, pred_corrupted, gt_corrupted):
    mAP_clean = compute_coco_mAP(pred_clean, gt_clean)
    mAP_corrupted = compute_coco_mAP(pred_corrupted, gt_corrupted)
    # Relative robustness for a specific corruption/severity
    rr = mAP_corrupted / mAP_clean
    return rr
# mRR = mean over 5 severity levels per corruption, then mean over C corruptions

Common pitfalls

• mRR is a relative ratio (higher is better), not an absolute performance drop. Interpreting it as a percentage decrease will invert the ranking.
• Must average over all 5 severity levels for each corruption type before averaging across the 10 corruption types, as specified in the formula.
• Uses COCO-style mAP (average of AP@[.5:.95]), not just AP@0.5.

Evidence (verbatim from paper)

Following[[48]], we introduce the robustness metric, mean Relative Robustness (mRR), to evaluate how much a model’s performance drops under certain corruptions compared to clean images. To calculate this metric, we first evaluate the model on clean images and obtain the mean Average Precision (mAP), denoted as $mAP_{clean}$. For any corruption $c$, we then calculate the mAP at each severity level $s$, denoted as $mAP_{c,s}$. The relative robustness for corruption $c$ and the mean Relative Robustness (mRR) are defined as follows: $RR_{c}=\frac{1}{5}\sum_{s=1}^{5}(1-\frac{mAP_{clean}-mAP_{c,s}}{mAP_{clean}}), \ \ mRR=\frac{1}{C}\sum_{c=1}^{C}RR_{c}$ where $c$ indexes the $C$ types of corruption.

Citation

@misc{ma2024posebench,
  title={PoseBench: Benchmarking the Robustness of Pose Estimation Models under Corruptions},
  author={Ma et al. (2024)},
  year={2024},
  note={arXiv:2406.14367}
}

• arXiv: 2406.14367