Meccano Eval

meccano-eval

MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain — Ragusa et al. (2022) (arXiv:2209.08691, 2022)

What this evaluates

Evaluates multimodal egocentric video understanding in an industrial-like setting. It probes action recognition, active object detection, human-object interaction, and future action anticipation using synchronized RGB, depth, and gaze signals.

Datasets

• MECCANO — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Top-1 Accuracy (primary) — range: [0, 1]
  • Percentage of video segments where the predicted action class exactly matches the ground truth label.
• AVG F1-score — range: [0, 1]
  • Class-mean F1-score computed across all action classes, averaging precision and recall per class before macro-averaging.
• AP (IoU>0.5) — range: [0, 1]
  • Average Precision for active object detection using a 0.5 Intersection over Union threshold, following standard Pascal VOC mAP conventions.

Input / output format

Input: Synchronized RGB frames, depth maps, and gaze fixation maps from egocentric videos.

Output: Action class labels for temporal segments, bounding boxes with class labels for active objects, and interaction tuples for EHOIs.

Scoring recipe

def compute_metrics(preds, gold):
    # Top-1 Accuracy
    top1 = sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)
    
    # AP (IoU>0.5)
    ap = compute_ap(preds, gold, iou_threshold=0.5)
    
    # AVG F1-score
    f1s = []
    for cls in set(gold):
        tp = sum(1 for p, g in zip(preds, gold) if p == g == cls)
        fp = sum(1 for p in preds if p == cls and p not in gold)
        fn = sum(1 for g in gold if g == cls and g not in preds)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    avg_f1 = sum(f1s) / len(f1s)
    return {'Top-1 Accuracy': top1, 'AP (IoU>0.5)': ap, 'AVG F1-score': avg_f1}

Common pitfalls

• Gaze modality yields only marginal gains over RGB-Depth fusion due to the simplicity of the baseline architecture.
• Standard hand-object detectors fail to generalize without domain-specific retraining on the industrial dataset.
• Active objects can outnumber hands (up to 7 per frame), requiring relaxed distance thresholds for detection baselines.

Evidence (verbatim from paper)

We evaluate action recognition using Top-1 and Top-5 accuracy computed on the whole test set. As class-aware measures, we report class-mean precision, class-mean recall and F1-score.

Citation

@misc{ragusa2022meccano,
  title={MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain},
  author={Ragusa et al. (2022)},
  year={2022},
  note={arXiv:2209.08691}
}

• arXiv: 2209.08691