Humanbench Eval

humanbench-eval

HumanBench: Towards General Human-centric Perception with Projector Assisted Pretraining — Tang et al. (2023) (arXiv:2303.05675, 2023)

What this evaluates

Evaluates the generalization and task-agnostic representation learning of human-centric vision models across six diverse downstream tasks. It probes how well a model trained on a large, multi-task human-centric corpus can adapt to in-distribution, out-of-distribution, and completely unseen human perception tasks.

Datasets

• HumanBench — total ?; splits: in-dataset (-1), out-of-dataset (-1), unseen-task (-1); repo https://github.com/OpenGVLab/HumanBench

Metrics

• mAP/mIoU/mA/Top1/pACC/MR/MSE/EPE (primary) — range: [0, 1] or percent
  • Task-dependent standard metrics: mAP (ReID/detection/pose), mIoU (parsing), mA (attributes), Top1 (ReID), pACC (parsing), MR-2 (detection), MSE (counting), EPE (pose). Values are averaged across classes or tasks as appropriate. Lower is better for MR and MSE.

Input / output format

Input: RGB images containing human figures or crowd scenes, provided as raw pixel tensors.

Output: Task-specific predictions: bounding boxes and keypoints for detection/pose, segmentation masks for parsing, identity embeddings for ReID, binary attribute vectors, and scalar counts for crowd counting.

Scoring recipe

def compute_metric(preds, gold, task, dataset):
    if task == 'reid':
        return compute_mAP(preds, gold) if dataset != 'SenseReID' else compute_top1(preds, gold)
    elif task == 'parsing':
        return compute_pacc(preds, gold) if dataset == 'ATR' else compute_mIoU(preds, gold)
    elif task == 'detection':
        return compute_miss_rate(preds, gold, fppi=2.0) if dataset == 'Caltech' else compute_AP(preds, gold)
    elif task == 'pose':
        return compute_epe(preds, gold) if dataset == 'Human3.6M' else compute_mAP(preds, gold)
    elif task == 'attribute':
        return compute_mean_accuracy(preds, gold)
    elif task == 'counting':
        return compute_mse(preds, gold)
    return None

Common pitfalls

• Out-of-dataset evaluation uses the same task type but different datasets (e.g., ATR for parsing instead of LIP), not a completely different task.
• Miss Rate (MR-2) and MSE are inverse metrics where lower values indicate better performance, contrary to accuracy/mAP/mIoU.
• Full finetuning is not always optimal; partial or head-only finetuning often yields better results on smaller downstream datasets due to overfitting.

Evidence (verbatim from paper)

Following the most commonly-used metrics, for human parsing tasks, we report pACC for ATR, mIoU for others. For ReID, we report Top1 for SenseReID, mAP for others. For pedestrian detection, we report AP for CrowdHuman and Miss Rate for Caltech. For pose estimation, we report End-Point Error for Human3.6M and mAP for others. For attribute, we all report mean accuracy. For Counting, we report Mean Square Error (MSE).

Citation

@misc{tang2023humanbench,
  title={HumanBench: Towards General Human-centric Perception with Projector Assisted Pretraining},
  author={Tang et al. (2023)},
  year={2023},
  note={arXiv:2303.05675}
}

• arXiv: 2303.05675