Egohumans Eval

egohumans-eval

EgoHumans: An Egocentric 3D Multi-Human Benchmark — Khirodkar et al. (2023) (arXiv:2305.16487, 2023)

What this evaluates

Evaluates the ability of models to perform robust multi-human tracking and identity association in unconstrained egocentric 3D environments. It probes how well algorithms handle severe occlusions, dynamic activities, and camera-agnostic spatial reasoning when fusing egocentric and secondary views.

Datasets

• EgoHumans — total ?; splits: train (77260), test (47740)

Metrics

• IDF1 (primary) — range: percent
  • Instance F1 score for tracking; measures the percentage of ground truth tracks that are correctly matched to predicted tracks, focusing on identity preservation. Calculated as tp / (tp + 0.5 * (fn + fp)).
• MOTA — range: percent
  • Multi-Object Tracking Accuracy; focuses on bounding box detection accuracy by penalizing false positives, false negatives, and identity switches relative to total ground truth frames.
• HOTA — range: percent
  • Higher-Order Tracking Accuracy; explicitly balances the effect of accurate detection and consistent association by computing the geometric mean of detection and association scores across thresholds.

Input / output format

Input: RGB images (monocular or stereo) from egocentric glasses and secondary cameras, along with camera poses and calibration data.

Output: Per frame: bounding boxes, 2D/3D human poses, SMPL mesh parameters, and persistent instance IDs.

Scoring recipe

def compute_idf1(predictions, gold):
    tp = count_correct_id_associations(predictions, gold)
    fn = count_missed_gt_tracks(gold)
    fp = count_false_positive_tracks(predictions)
    idf1 = tp / (tp + 0.5 * (fn + fp))
    return idf1 * 100  # percentage

def compute_mota(predictions, gold):
    fp = count_false_positives(predictions, gold)
    fn = count_false_negatives(predictions, gold)
    ids = count_identity_switches(predictions, gold)
    total_gt = sum(len(gt) for gt in gold)
    mota = 1 - (fp + fn + ids) / total_gt
    return mota * 100  # percentage

Common pitfalls

• Using MOTA as the sole metric is misleading when detection quality is fixed across methods, as it fails to capture association performance and identity preservation.
• Applying standard 2D tracking baselines without 3D spatial reasoning leads to severe identity switches in egocentric views due to extreme occlusions and wide fields of view.
• Fine-tuning models using MOT17 hyperparameters without adapting to egocentric camera characteristics limits performance gains and ignores the dataset's unique spatial reasoning requirements.

Evidence (verbatim from paper)

To evaluate the 3D human tracking performance, we use the CLEAR metrics [7], including MOTA, FP, FN, IDs, etc. along with IDF1[98] and HOTA [47]. MOTA focuses on bbox detection accuracy. IDF1 evaluates the instance identity preservation and focuses on the association performance. Recently, HOTA has been proposed, which explicitly balances the effect of accurate detection and consistent association. Our experiments predominantly use an off-shelf bbox detector, so IDF1 is our primary metric.

Citation

@misc{khirodkar2023egohumans,
  title={EgoHumans: An Egocentric 3D Multi-Human Benchmark},
  author={Khirodkar et al. (2023)},
  year={2023},
  note={arXiv:2305.16487}
}

• arXiv: 2305.16487