Jrdb Par Eval

jrdb-par-eval

Panoramic Human Activity Recognition — Han et al. (2022) (arXiv:2203.03806, 2022)

What this evaluates

Evaluates a model's ability to jointly recognize individual actions, social group activities, and global crowd-level activities in crowded panoramic scenes. It probes multi-granular activity recognition and hierarchical graph-based scene understanding.

Datasets

• JRDB-PAR — total 1850; splits: train (1439), test (411); repo https://github.com/RuizeHan/PAR

Metrics

• F1 score — range: percent
  • Standard multi-label precision, recall, and F1 score computed per instance for individual and global activities. For social activities, it is computed only on groups correctly detected via the Half metric (group member IoU > 0.5).
• Overall F1 ($\mathcal{F}_a$) (primary) — range: percent
  • The arithmetic mean of the F1 scores for individual, social, and global activity recognition: $\mathcal{F}_a = \frac{1}{3} (\mathcal{F}_i + \mathcal{F}_p + \mathcal{F}_g)$.

Input / output format

Input: Uniformly sampled key frames (1 per 15 frames) from 360° RGB videos, containing human bounding boxes, IDs, and spatial coordinates. Models process visual features to predict activity labels.

Output: Multi-label predictions for individual actions (27 classes), social group activities (11 classes), and global activities (7 classes).

Scoring recipe

def compute_metrics(preds, gold):
    # Individual & Global (multi-label)
    P_i, R_i, F1_i = compute_prf_multi_label(preds['indiv'], gold['indiv'])
    P_g, R_g, F1_g = compute_prf_multi_label(preds['global'], gold['global'])
    
    # Social (two-stage: detection then classification)
    detected = [g for g in preds['social'] if group_iou(g, gold['social']) > 0.5]
    correct_social = [g for g in detected if g.activity == gold['social'][g.id].activity]
    P_p, R_p, F1_p = compute_prf(correct_social, gold['social'])
    
    # Overall
    F_a = (F1_i + F1_p + F1_g) / 3.0
    return {'F1_i': F1_i, 'F1_p': F1_p, 'F1_g': F1_g, 'Overall F1': F_a}

Common pitfalls

• Social activity F1 only counts groups correctly detected via the Half metric (IoU > 0.5); failing group detection directly penalizes the social activity metric.
• Evaluation uses uniformly sampled key frames (1 per 15 frames), not all video frames, which differs from standard dense video action recognition protocols.
• All three tasks use multi-label classification, meaning a single subject or frame can simultaneously hold multiple activity labels.

Evidence (verbatim from paper)

Protocol I. To evaluate the individual action detection, following the previous work [16] for multi-label classification task, we adopt the commonly used metrics - precision, recall and $\mathrm{F_1}$ score (denoted as $\mathcal{P}_i$, $\mathcal{R}_i$, and $\mathcal{F}_i$) as the evaluation metrics... Protocol II. Social (group) activity recognition includes the group detection and the activity category recognition... we use the classical Half metrics... group member IoU $>0.5$... calculate the precision, recall and $\mathbf{F}_1$ score (denoted as $\mathcal{P}_p$, $\mathcal{R}_p$, and $\mathcal{F}_p$)... Protocol III. Global activity recognition... apply the precision, recall, and $\mathbf{F}_1$ score (denoted as $\mathcal{P}_g$, $\mathcal{R}_g$, $\mathcal{F}_g$)... The overall metric for the panoramic activity detection task is the comprehensive results of the above three metrics. Here we simply compute the average value of the above $\mathrm{F_1}$ scores, i.e., $\mathcal{F}_a = \frac{1}{3} (\mathcal{F}_i + \mathcal{F}_p + \mathcal{F}_g)$ as the overall $\mathrm{F_1}$ score.

Citation

@misc{han2022panoramic,
  title={Panoramic Human Activity Recognition},
  author={Han et al. (2022)},
  year={2022},
  note={arXiv:2203.03806}
}

• arXiv: 2203.03806