Isafetybench Eval

isafetybench-eval

iSafetyBench: A video-language benchmark for safety in industrial environment — Abdullah et al. (2025) (arXiv:2508.00399, 2025)

What this evaluates

Probes vision-language models' ability to recognize routine and hazardous industrial actions in real-world videos under zero-shot conditions. It tests both single-label precision and multi-label recall in safety-critical contexts, evaluating how well models discriminate between semantically similar distractors and identify multiple concurrent actions.

Datasets

• iSafetyBench — total 1100; splits: test (1100); repo https://github.com/iSafetyBench/data

Metrics

• accuracy (primary) — range: [0, 1]
  • For single-correct-choice questions, accuracy is the fraction of videos where the model correctly selects the single ground-truth action from 16 options.
• precision — range: [0, 1]
  • For multiple-correct-choice questions, precision is the ratio of correctly predicted actions to the total number of actions predicted by the model.
• recall — range: [0, 1]
  • For multiple-correct-choice questions, recall is the ratio of correctly predicted actions to the total number of ground-truth actions in the video.
• F1 score — range: [0, 1]
  • The harmonic mean of precision and recall for multiple-correct-choice questions: F1 = 2 * (precision * recall) / (precision + recall).

Input / output format

Input: A 4–8 second video clip paired with a multiple-choice question containing 16 options (1 ground-truth action + 15 semantically/visually similar distractors generated by Gemini 2.5 Pro).

Output: One or more action labels selected from the 16 provided options.

Scoring recipe

if single_choice:
    correct = 1 if pred == gold else 0
    accuracy = mean(correct)
else:
    tp = len(set(pred) & set(gold))
    fp = len(set(pred) - set(gold))
    fn = len(set(gold) - set(pred))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0

Common pitfalls

• Distractors are semantically and visually similar to the ground truth, preventing models from relying on coarse-grained recognition or simple keyword matching.
• Multi-label questions require identifying all valid actions; models that only predict the most prominent action will suffer low recall and F1 scores.
• Evaluation is strictly zero-shot; any fine-tuning or task-specific adaptation invalidates the benchmark protocol.

Evidence (verbatim from paper)

We adopt accuracy for single-correct-choice questions, where a response is correct only if the model selects the single ground-truth action. For multiple-correct-choice questions, we compute precision, recall, and F1 score based on the set of selected versus true labels. Precision captures how many of the predicted actions are correct, recall captures how many of the ground-truth actions are retrieved, and F1 score balances the two.

Citation

@misc{abdullah2025isafetybench,
  title={iSafetyBench: A video-language benchmark for safety in industrial environment},
  author={Abdullah et al. (2025)},
  year={2025},
  note={arXiv:2508.00399}
}

• arXiv: 2508.00399