Hasper Eval

hasper-eval

HaSPeR: An Image Repository for Hand Shadow Puppet Recognition — Raiyan et al. (2024) (arXiv:2408.10360, 2024)

What this evaluates

Evaluates the ability of computer vision models to classify hand shadow puppet silhouettes into one of 15 distinct categories. It probes feature extraction robustness, particularly for rotationally asymmetric and visually similar silhouettes under varying lighting and motion dynamics.

Datasets

• HaSPeR — total 15000; splits: train (-1), val (-1), test (-1); repo https://github.com/Starscream-11813/HaSPeR

Metrics

• top-1 accuracy (primary) — range: percent
  • Fraction of correctly predicted classes out of the total number of samples.
• top-k accuracy (k=2,3) — range: percent
  • Fraction of samples where the true class appears within the top k highest predicted probabilities.
• Precision — range: [0, 1]
  • Ratio of true positive predictions to all positive predictions for each class.
• Recall — range: [0, 1]
  • Ratio of true positive predictions to all actual positives for each class.
• F1-score — range: [0, 1]
  • Harmonic mean of Precision and Recall.

Input / output format

Input: Resized RGB image of a hand shadow puppet silhouette.

Output: Single class label from 15 predefined categories.

Scoring recipe

def compute_metrics(predictions, labels, k=1):
    correct = sum(1 for p, l in zip(predictions, labels) if p == l)
    accuracy = correct / len(labels)
    # Top-k accuracy, precision, recall, F1 computed via standard sklearn.metrics
    return accuracy

Common pitfalls

• Dataset exhibits slight class imbalance, particularly in professional source clips.
• Silhouettes are rotationally asymmetric and incongruent, requiring specific augmentations like random rotation and affine transforms.
• High inter-class similarity (e.g., Crab vs. Bird/Moose/Rabbit) leads to frequent confusion matrix errors.

Evidence (verbatim from paper)

We use top-$k$ validation accuracy values (with $k=1,2,3$), Precision, Recall, and F1-score as evaluation metrics to perform comparative analyses among the aforementioned models.

Citation

@misc{raiyan2024hasper,
  title={HaSPeR: An Image Repository for Hand Shadow Puppet Recognition},
  author={Raiyan et al. (2024)},
  year={2024},
  note={arXiv:2408.10360}
}

• arXiv: 2408.10360