Pistachio Eval

pistachio-eval

Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmarks — Jie Li et al. (2025) (arXiv:2511.19474, 2025)

What this evaluates

Evaluates video anomaly detection and understanding on synthetic, long-form videos with diverse scenes and balanced anomaly categories. Probes models' temporal consistency, ability to detect subtle behavioral anomalies, and long-context narrative comprehension.

Datasets

• Pistachio — total 1676822; splits: train (1544546), test (132276)

Metrics

• frame-level AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve computed per frame to measure detection performance.
• frame-level AP — range: [0, 1]
  • Average Precision computed per frame, summarizing the precision-recall curve across confidence thresholds.
• F1-Score — range: [0, 1]
  • Harmonic mean of precision and recall. Precision and recall are calculated by comparing generated anomaly descriptions against ground truth, counting a prediction as correct if semantic similarity exceeds a predefined threshold.

Input / output format

Input: Video frames or clips for detection; full video for understanding.

Output: Per-frame anomaly scores/labels for detection; structured anomaly descriptions/narratives for understanding.

Scoring recipe

# VAD
auc = compute_auc(y_true_frames, y_pred_scores)
ap = compute_ap(y_true_frames, y_pred_scores)
# VAU
correct = sum(1 for pred, gt in zip(predictions, ground_truth) if semantic_similarity(pred, gt) > threshold)
precision = correct / len(predictions)
recall = correct / len(ground_truth)
f1 = 2 * (precision * recall) / (precision + recall)

Common pitfalls

• Models relying on likelihood-based normal distribution learning fail when normal video diversity is high, mistaking diverse normal scenes for anomalies.
• Long-form generation baselines suffer from prompt forgetting and ghosting artifacts when not using hierarchical prompting, leading to semantic drift over time.
• Backbone choice significantly impacts results: I3D features often outperform ViT features for specific anomaly categories despite ViT having higher overall AUC.

Evidence (verbatim from paper)

Following standard evaluation protocols, all methods were trained on the Pistachio training set and evaluated on our test set. Performance is measured by frame-level AUC and AP. To evaluate the benchmarks of video anomaly understanding, we adopt F1-Score as the main metric to measure the model’s capability in comprehending anomalies across different temporal granularities. The F1-Score is calculated as the harmonic mean of precision and recall, providing a balanced assessment of the model’s performance. For each level, we measure the F1-Score by comparing the model’s generated anomaly descriptions with ground truth annotations, considering a prediction as correct when the semantic similarity exceeds a predefined threshold.

Citation

@misc{li2025pistachio,
  title={Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmarks},
  author={Jie Li et al. (2025)},
  year={2025},
  note={arXiv:2511.19474}
}

• arXiv: 2511.19474