qhjqhj00 silicone-prep-anomaly-eval
A VLM-based Method for Visual Anomaly Detection in Robotic Scientific Laboratories — Lin et al. (2025) (arXiv:2506.05405, 2025)
What this evaluates
Evaluates multimodal vision-language models' ability to detect context-dependent visual anomalies in robotic scientific laboratory workflows using first-person imagery and stage-specific textual prompts.
Datasets
- Silicone Preparation Workflow — total 1001; splits: test (1001)
Metrics
accuracy(primary) — range: [0, 1]- Binary classification accuracy; proportion of correctly classified normal vs. abnormal images. Exact formula not detailed in this section.
Input / output format
Input: First-person visual observation (image) paired with structured textual annotations providing stage-dependent contextual prompts and monitoring point information.
Output: Binary anomaly detection label (normal or abnormal) per image.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return correct / len(gold_labels)
Common pitfalls
- Spatial and viewpoint diversity across fixed/mobile robotic arms may cause distribution shifts not captured in a single split.
- Consensus-based annotation by three annotators may smooth out rare or ambiguous anomalies, affecting ground-truth reliability.
- Zero-shot evaluation without task-specific fine-tuning may underestimate model capability on domain-specific scientific imagery.
Evidence (verbatim from paper)
demonstrating that higher-level contextual prompts significantly improve detection accuracy and robustness, validating the method's adaptability across diverse scientific processes.
Citation
@misc{lin2025vlmanomaly,
title={A VLM-based Method for Visual Anomaly Detection in Robotic Scientific Laboratories},
author={Lin et al. (2025)},
year={2025},
note={arXiv:2506.05405}
}
- arXiv: 2506.05405