Harrison Eval

harrison-eval

HARRISON: A Benchmark on HAshtag Recommendation for Real-world Images in Social Networks — Park et al. (2016) (arXiv:1605.05054, 2016)

What this evaluates

This benchmark evaluates a model's ability to recommend relevant hashtags for real-world social media images using only visual input. It probes contextual image understanding and multi-label classification by measuring how well predicted hashtags align with actual user-generated tags.

Datasets

• HARRISON — total 57383; splits: test (-1)

Metrics

• Precision@1 (primary) — range: percent
  • The fraction of hashtags in the top 1 predicted result that exactly match the ground truth set. Formula: |Result(1) ∩ GT| / |Result(1)|.
• Recall@5 — range: percent
  • The fraction of ground truth hashtags covered by the top 5 predicted results. Formula: |Result(5) ∩ GT| / |GT|.
• Accuracy@5 — range: percent
  • A binary indicator that is 1 if at least one predicted hashtag in the top 5 matches any ground truth hashtag, and 0 otherwise.

Input / output format

Input: Single real-world social media image (processed via pre-trained VGG-16 visual features).

Output: Ranked list of predicted hashtags (top K, where K=1 for precision and K=5 for recall/accuracy).

Scoring recipe

def compute_metrics(preds, gold, k_prec=1, k_rec=5, k_acc=5):
    top_k_prec = set(preds[:k_prec])
    top_k_rec = set(preds[:k_rec])
    gt = set(gold)
    inter_prec = top_k_prec & gt
    inter_rec = top_k_rec & gt
    prec = len(inter_prec) / len(top_k_prec) if top_k_prec else 0.0
    rec = len(inter_rec) / len(gt) if gt else 0.0
    acc = 1.0 if inter_rec else 0.0
    return {'precision@1': prec, 'recall@5': rec, 'accuracy@5': acc}

Common pitfalls

• K is metric-specific: Precision uses K=1, while Recall and Accuracy use K=5. Using a uniform K across all metrics will yield incorrect scores.
• Evaluation relies on exact string matching between predicted and ground truth hashtags. Semantic similarity, lemmatization, or synonym matching is not applied during scoring.
• The protocol treats hashtags as independent labels, ignoring co-occurrence dependencies or hierarchical relationships between tags.

Evidence (verbatim from paper)

In our experiments, we set K to 1 for precision and 5 for recall and accuracy considering the average number of associated hashtags per image in the HARRISON dataset. We evaluated the baseline results on Precision@1, Recall@5, and Accuracy@5 by averaging over all images in the test set.

Citation

@misc{park2016harrison,
  title={HARRISON: A Benchmark on HAshtag Recommendation for Real-world Images in Social Networks},
  author={Park et al. (2016)},
  year={2016},
  note={arXiv:1605.05054}
}

• arXiv: 1605.05054