Matt Attribution Eval

matt-attribution-eval

Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos — Yayuan Li et al. (2025) (arXiv:2511.20525, 2025)

What this evaluates

This benchmark evaluates fine-grained mistake understanding in egocentric videos by attributing errors to specific semantic roles, temporal points of no return, and spatial locations. It probes a model's ability to align video content with instructional text, localize manipulation events, and classify whether actions deviate from intended goals.

Datasets

• Ego4D-M — total 256000; splits: train (206000), val (25000), test (25000)
• EPIC-KITCHENS-M — total 220000; splits: train (176000), val (22000), test (22000)
• EgoPER — total ?; splits: test (-1)

Metrics

• F1@0.5 (primary) — range: [0, 1]
  • Binary classification F1 score computed at a 0.5 confidence threshold. Calculated as 2TP/(2TP+FP+FN) per semantic role or averaged across roles.
• Accuracy — range: [0, 1]
  • Percentage of correctly classified instances (mistake vs. correct) per role or averaged.
• MAE (primary) — range: other
  • Mean Absolute Error between predicted and ground-truth Point-of-No-Return frame index or time duration.
• mIoU (primary) — range: [0, 1]
  • Mean Intersection-over-Union between predicted and ground-truth bounding boxes for the manipulated object/hands in the PNR frame.
• Center Distance (CD) — range: other
  • Normalized Euclidean distance between the centers of predicted and ground-truth bounding boxes.
• Box Size Error (BSE) — range: other
  • Normalized difference between the width and height dimensions of predicted and ground-truth bounding boxes.

Input / output format

Input: Egocentric video clips paired with corresponding instructional text. For baselines, structured prompts are used to elicit binary decisions per semantic role or localization outputs.

Output: Binary classification per semantic role (mistake/correct), predicted frame index or time for temporal localization, bounding box coordinates for spatial localization, and binary clip-level mistake label for detection.

Scoring recipe

def compute_metrics(preds, gold):
    # Semantic/Detection: F1@0.5 & Accuracy
    y_pred = [1 if p > 0.5 else 0 for p in preds]
    acc = sum(y_pred == gold) / len(gold)
    tp = sum(1 for t, p in zip(gold, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(gold, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(gold, y_pred) if t == 1 and p == 0)
    f1 = 2*tp / (2*tp + fp + fn) if (2*tp + fp + fn) > 0 else 0
    # Temporal: MAE (frames/seconds)
    mae = sum(abs(p - g) for p, g in zip(preds_frames, gold_frames)) / len(gold_frames)
    # Spatial: mIoU, CD, BSE
    miou = mean(iou(p_box, g_box) for p_box, g_box in zip(preds_boxes, gold_boxes))
    cd = mean(box_center_dist(p_box, g_box) for p_box, g_box in zip(preds_boxes, gold_boxes))
    bse = mean(box_size_error(p_box, g_box) for p_box, g_box in zip(preds_boxes, gold_boxes))
    return {'F1@0.5': f1, 'Accuracy': acc, 'MAE': mae, 'mIoU': miou, 'CD': cd, 'BSE': bse}

Common pitfalls

• Mistake detection baselines are designed for out-of-distribution detection on a small set of activities and fail on large-scale benchmarks because they cannot separate actual mistakes from benign variability across many activities.
• Standard hand-detection models (e.g., MediaPipe) cannot be used for spatial attribution because they do not localize the manipulated object, which is required for precise mistake grounding.
• Feeding raw text embeddings directly into attribution heads without a projection block significantly degrades semantic attribution performance.

Evidence (verbatim from paper)

For semantic attribution, we treat each semantic role as a binary classification problem. Following prior work[[23]], we report F1@0.5 and Accuracy as the main metrics, both per semantic role and averaged across roles. Following the PNR localization task[[12]], we evaluate temporal attribution by Mean Absolute Error, both per frame and per second. For spatial attribution, we report mean Intersection-over-Union (mIoU) between the predicted and ground-truth boxes, and additionally report Center Distance (CD) and Box Size Error (BSE).

Citation

@misc{li2025mistakeattribution,
  title={Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos},
  author={Yayuan Li et al. (2025)},
  year={2025},
  note={arXiv:2511.20525}
}

• arXiv: 2511.20525