Video Star Eval

video-star-eval

Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools — Yuan et al. (2025) (arXiv:2510.08480, 2025)

What this evaluates

Evaluates open-vocabulary action recognition by testing a model's ability to generalize to unseen action categories and cross-dataset distributions. It probes fine-grained video understanding and cross-modal reasoning capabilities under base-to-novel and cross-dataset generalization settings.

Datasets

• UCF-101 — total 13320; splits: train (-1), test (-1)
• HMDB-51 — total 6849; splits: train (-1), test (-1)
• Kinetics-400 — total ?; splits: train (-1), test (-1)
• Kinetics-600 — total ?; splits: train (-1), test (-1)
• Something-Something V2 — total ?; splits: train (-1), test (-1)

Metrics

• Top-1 accuracy (primary) — range: percent
  • Percentage of correctly predicted action classes out of the total number of test samples. Calculated as (number of correct predictions / total predictions) * 100.
• Harmonic Mean (HM) — range: percent
  • Harmonic mean of the Top-1 accuracy on base classes and novel classes. Calculated as 2 * (base_acc * novel_acc) / (base_acc + novel_acc). Used to balance performance across seen and unseen categories.

Input / output format

Input: Raw video clips representing human actions, processed as visual tokens by the vision-language model.

Output: Predicted action class label.

Scoring recipe

def compute_metrics(predictions, gold_labels, base_indices, novel_indices):
    top1 = sum(1 for p, g in zip(predictions, gold_labels) if p == g) / len(gold_labels) * 100
    base_acc = sum(1 for i in base_indices if predictions[i] == gold_labels[i]) / len(base_indices) * 100
    novel_acc = sum(1 for i in novel_indices if predictions[i] == gold_labels[i]) / len(novel_indices) * 100
    hm = 2 * (base_acc * novel_acc) / (base_acc + novel_acc) if (base_acc + novel_acc) > 0 else 0
    return {'top1_accuracy': top1, 'harmonic_mean': hm}

Common pitfalls

• The base-to-novel setting evaluates on both base and novel classes, but the model is only trained on base classes, making it a zero-shot generalization setup rather than standard supervised evaluation.
• Baselines are fine-tuned on the base set of each respective dataset, whereas Video-STAR is only fine-tuned on HMDB-51's base set and evaluated zero-shot on others, creating an asymmetric comparison.
• Cross-dataset evaluation trains on a source dataset and tests on a target dataset, which measures transferability rather than in-distribution performance.

Evidence (verbatim from paper)

Following prior work(Rasheed et al., [2023]; Ni et al., [2022]; Weng et al., [2023]), we report average top-1 accuracy under the two settings.

Citation

@misc{yuan2025videostar,
  title={Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools},
  author={Yuan et al. (2025)},
  year={2025},
  note={arXiv:2510.08480}
}

• arXiv: 2510.08480