Human Pose Estimation Eval

human-pose-estimation-eval

Bias-Compensated Integral Regression for Human Pose Estimation — Gu et al. (2023) (arXiv:2301.10431, 2023)

What this evaluates

This evaluation protocol assesses the accuracy of human and hand pose estimation models in localizing anatomical keypoints on images. It probes the model's ability to handle varying instance scales, occlusion levels, and joint visibility by measuring localization error against ground truth annotations.

Datasets

• MS COCO — total 250000; splits: train (-1), val (-1)
• MPII Human Pose — total 49000; splits: train (-1), val (-1)
• RHD — total 43700; splits: train (41000), test (2700)

Metrics

• AP@OKS (primary) — range: [0, 1]
  • mean average precision computed over 10 OKS thresholds (0.50 to 0.95). OKS normalizes the squared Euclidean distance between predicted and ground truth keypoints by the squared area of the person/hand instance.
• EPE — range: other
  • squared Euclidean distance between the predicted keypoint coordinates and the ground truth coordinates.
• PCK — range: [0, 1]
  • Percentage of Correct Keypoints, measuring the fraction of predicted keypoints within a threshold distance of the ground truth.
• AUC — range: [0, 1]
  • Area Under the Curve of the performance or error distribution for hand pose estimation.

Input / output format

Input: RGB images with person/hand bounding boxes, cropped and resized to a fixed resolution (e.g., 256×192).

Output: A heatmap tensor of shape (K, H, W) (e.g., 17 or 21 joints × 64 × 48) representing the predicted probability distribution for each keypoint, from which coordinates are decoded via expectation or max-pooling.

Scoring recipe

def compute_metrics(preds, gts, sizes):
    oks_scores = []
    for pred, gt, size in zip(preds, gts, sizes):
        dist_sq = np.sum((pred - gt)**2, axis=1)
        oks = np.exp(-dist_sq / (2 * size**2))
        oks_scores.append(np.mean(oks))
    ap = average_precision(oks_scores, thresholds=np.arange(0.50, 1.0, 0.05))
    epe = np.mean(np.sum((preds - gts)**2, axis=1))
    return ap, epe

Common pitfalls

• EPE is defined as squared Euclidean distance in this paper, not standard Euclidean distance or pixel error, which can lead to misreported values if not squared.
• OKS normalization uses the instance size (s), which varies per person/hand; failing to use the correct per-instance size for normalization will invalidate AP@OKS scores.
• Heatmap decoding method (expectation vs. max) significantly impacts results; integral regression uses expectation while detection uses max, and mixing them without standardization skews comparisons.

Evidence (verbatim from paper)

We evaluate with the standard metric, Object Keypoint Similarity (OKS). OKS normalizes the absolute error between the predicted location and the ground truth location with the size of the person. We use the primary challenge evaluation metric, mean average precision (AP), over 10 OKS thresholds to evaluate the performance. We also report the value before normalization, the squared Euclidean distance between the prediction and ground truth, which we denote as End-Point Error (EPE).

Citation

@misc{gu2023biascompensated,
  title={Bias-Compensated Integral Regression for Human Pose Estimation},
  author={Gu et al. (2023)},
  year={2023},
  note={arXiv:2301.10431}
}

• arXiv: 2301.10431