Humbi Eval

humbi-eval

HUMBI: A Large Multiview Dataset of Human Body Expressions and Benchmark Challenge — Yoon et al. (2021) (arXiv:2110.00119, 2021)

What this evaluates

Probes the generalizability, diversity, and reconstruction accuracy of 3D human body expression models (gaze, face, hand, body) across multiple datasets and viewpoints. It evaluates how well models trained on HUMBI generalize to unseen datasets and how accurately they reconstruct 3D geometry from monocular images.

Datasets

• HUMBI — total ?; splits: train (-1), test (-1)

Metrics

• IoU (primary) — range: [0, 1]
  • Intersection over Union between the ground truth 2D silhouette mask and the 2D projection of the reconstructed 3D model.
• Chamfer distance — range: pixel
  • Average Euclidean distance between points in the ground truth mask and points in the 2D projection of the reconstructed 3D model.
• reprojection error — range: degree | pixel
  • Mean squared error of predicted gaze yaw/pitch angles (degrees) or pixel-wise reprojection error scaled to 256x256 resolution.
• AUC of PCK — range: [0, 1]
  • Area under the Percentage of Correct Keypoints curve, computed over an error range of 0-20 mm for hand or 0-150 mm for body.

Input / output format

Input: Single-view RGB image. For gaze prediction, additionally receives normalized eye patch and head pose yaw/pitch angles.

Output: Predicted 3D mesh parameters (shape coefficients, camera pose), gaze yaw/pitch angles, or 3D keypoints.

Scoring recipe

def compute_metrics(pred_mesh, cam_pose, gt_mask, H=256, W=256):
    pred_mask = project_3d_to_2d(pred_mesh, cam_pose, H, W)
    iou = np.sum(pred_mask & gt_mask) / (np.sum(pred_mask | gt_mask) + 1e-6)
    pred_pts = np.column_stack(np.where(pred_mask))
    gt_pts = np.column_stack(np.where(gt_mask))
    dists = np.linalg.norm(pred_pts[:, None] - gt_pts[None, :], axis=2)
    chamfer = np.mean(np.min(dists, axis=1))
    return iou, chamfer

Common pitfalls

• Domain gap between synthetic baseline datasets (e.g., UTMV, ObMan) and real HUMBI data causes significant performance drops if models are not trained on combined data.
• HUMBI Body Chamfer distance is artificially high due to shape mismatch between the parametric 3D body model and natural clothing/garments.
• Cross-data evaluation requires careful handling of viewpoint dependency and differing camera calibrations across baseline datasets.

Evidence (verbatim from paper)

We use the area under PCK curve (AUC) as a metric. HUMBI Hand shows superior performance on predicting 3D hand pose comparing to other three datasets by a margin of 0.02-0.16 AUC.

Citation

@misc{yoon2021humbi,
  title={HUMBI: A Large Multiview Dataset of Human Body Expressions and Benchmark Challenge},
  author={Yoon et al. (2021)},
  year={2021},
  note={arXiv:2110.00119}
}

• arXiv: 2110.00119