Timid Robot Mistake Detection Eval

timid-robot-mistake-detection-eval

TIMID: Time-Dependent Mistake Detection in Videos of Robot Executions — Gallego et al. (2026) (arXiv:2603.09782, 2026)

What this evaluates

This benchmark evaluates a model's ability to detect time-dependent and physical mistakes in robotic task executions from video. It specifically probes temporal reasoning, semantic task violation detection, and sim-to-real generalization by comparing frame-level anomaly predictions against ground-truth annotations.

Datasets

• BridgeData V2 — total 1000; splits: train (800), test (200)
• Multi-robot dataset — total ?; splits: train (-1), test (-1)

Metrics

• Average Precision (AP) — range: [0, 100] percent
  • Area under the precision-recall curve computed over frame-level predictions, summarizing detection performance across all thresholds.
• Average Recall (AR) — range: [0, 100] percent
  • Average recall across frames or videos, measuring the fraction of actual mistake frames correctly identified by the model.
• F1 (primary) — range: [0, 100] percent
  • Harmonic mean of precision and recall at the frame level, balancing false positives and false negatives to report overall detection accuracy.

Input / output format

Input: Video sequences of robot executions (processed as frames or frame batches), accompanied by task/mistake prompts specifying the expected behavior and error types.

Output: Frame-level binary predictions or anomaly scores indicating whether a mistake is occurring at each frame.

Scoring recipe

def compute_frame_metrics(predictions, gold):
    tp = sum(p == 1 and g == 1 for p, g in zip(predictions, gold))
    fp = sum(p == 1 and g == 0 for p, g in zip(predictions, gold))
    fn = sum(p == 0 and g == 1 for p, g in zip(predictions, gold))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    ap = compute_average_precision_curve(precision, recall)
    return ap, recall, f1

Common pitfalls

• Models are trained with weak video-level supervision (correct vs. erroneous) but must generate frame-level predictions, creating a label alignment challenge that can inflate or deflate metrics depending on thresholding.
• Inference time varies drastically between baselines (e.g., VLMs take hours vs. seconds for TIMID), which is a critical deployment constraint often overlooked when comparing accuracy metrics.
• Sim-to-real domain shift causes significant performance drops across all architectures, so evaluating only on simulation data overestimates real-world applicability.

Evidence (verbatim from paper)

To evaluate the models, we measure standard detection metrics, Average Precision (AP), Average Recall (AR) and F1, computed at frame level.

Citation

@misc{gallego2026timid,
  title={TIMID: Time-Dependent Mistake Detection in Videos of Robot Executions},
  author={Gallego et al. (2026)},
  year={2026},
  note={arXiv:2603.09782}
}

• arXiv: 2603.09782