Botfails Eval

botfails-eval

Failure Identification in Imitation Learning Via Statistical and Semantic Filtering — Rolland et al. (2026) (arXiv:2604.13788, 2026)

What this evaluates

This evaluation probes a model's ability to detect and classify robotic failures in real-world manipulation tasks. It specifically tests whether a system can distinguish between genuine task-disrupting failures and benign environmental deviations using multimodal observations and nominal demonstrations.

Datasets

• BotFails — total ?; splits: train (-1), test (-1)
• Real-π dataset — total ?; splits: train (-1), test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive and false positive rates across all classification thresholds.
• failure-detection accuracy — range: [0, 1]
  • Percentage of correctly identified failure instances out of the total number of failure instances in the test set.

Input / output format

Input: Multimodal observations including camera images, proprioceptive state, and natural language task instructions, aligned against compact representations of nominal demonstrations.

Output: Continuous anomaly scores and spatial heatmaps, followed by a binary classification (failure vs. benign deviation) after vision-language model filtering.

Scoring recipe

def compute_metrics(predictions, labels):
    fpr, tpr, _ = roc_curve(labels, predictions)
    auroc = auc(fpr, tpr)
    preds_binary = (predictions >= 0.5).astype(int)
    accuracy = mean(preds_binary == labels)
    return {'AUROC': auroc, 'failure-detection accuracy': accuracy}

Common pitfalls

• The dataset lacks publicly available train/val/test splits and exact size, making direct replication difficult.
• Baseline methods are adapted to the experimental setting rather than used out-of-the-box, which may introduce implementation bias.
• Distinguishing benign deviations (e.g., background shifts) from genuine failures requires careful thresholding and semantic filtering, which can be sensitive to VLM prompt design.

Evidence (verbatim from paper)

Evaluated on the novel BotFails dataset, FIDeL achieves +5.30% AUROC and +17.38% failure-detection accuracy over baselines, demonstrating superior real-world applicability and interpretability.

Citation

@misc{rolland2026failureidentification,
  title={Failure Identification in Imitation Learning Via Statistical and Semantic Filtering},
  author={Rolland et al. (2026)},
  year={2026},
  note={arXiv:2604.13788}
}

• arXiv: 2604.13788