Deepaction Eval

deepaction-eval

Human Action CLIPs: Detecting AI-generated Human Motion — Bohacek et al. (2024) (arXiv:2412.00526, 2024)

What this evaluates

This benchmark evaluates the ability of multi-modal embedding classifiers to distinguish real human motion videos from AI-generated ones. It probes semantic consistency detection, robustness to video laundering (resolution/compression), and generalization to unseen generative models.

Datasets

• DeepAction — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Macro-average accuracy: the unweighted mean of per-class accuracies, (Accuracy_real + Accuracy_fake) / 2. Reported separately at frame and video levels, averaged over five random train/test repetitions.

Input / output format

Input: Video frames or entire videos represented as multi-modal semantic embeddings (e.g., CLIP, SigLIP, JinaCLIP, or fine-tuned CLIP).

Output: Binary label (real vs. AI-generated) or multi-class label (specific AI model), produced by an SVM classifier operating on the embeddings.

Scoring recipe

# predictions: list of predicted labels (0=real, 1=fake)
# gold: list of true labels
acc_real = sum(p == g for p, g in zip(predictions, gold) if g == 0) / sum(1 for g in gold if g == 0)
acc_fake = sum(p == g for p, g in zip(predictions, gold) if g == 1) / sum(1 for g in gold if g == 1)
macro_acc = (acc_real + acc_fake) / 2
return macro_acc * 100  # percent

Common pitfalls

• Splits are determined at the action (prompt) level, not the video level, to prevent data leakage across train/test partitions.
• The dataset is imbalanced with significantly more fake videos; under-sampling is applied, but macro-average accuracy must be reported to avoid class-bias.
• Video-level accuracy is evaluated independently from frame-level accuracy, not merely averaged from frame predictions.

Evidence (verbatim from paper)

In each case, we report the mean frame- and video-level accuracy on the test set, averaged over five random train/test repetitions. Because our dataset is imbalanced, with significantly more fake than real videos, we under-sample the fake videos. Throughout, we report accuracy as a macro-average by evenly weighting the class accuracies.

Citation

@misc{bohacek2024humanactionclips,
  title={Human Action CLIPs: Detecting AI-generated Human Motion},
  author={Bohacek et al. (2024)},
  year={2024},
  note={arXiv:2412.00526}
}

• arXiv: 2412.00526