Entity6k Eval

entity6k-eval

Entity6K: A Large Open-Domain Evaluation Dataset for Real-World Entity Recognition — Qiu et al. (2024) (arXiv:2403.12339, 2024)

What this evaluates

Evaluates open-domain entity recognition capabilities across four visual grounding and understanding tasks: object detection, zero-shot image classification, image captioning, and dense captioning. It measures how well models can localize, classify, and describe specific real-world entities in images without fine-tuning.

Datasets

• Entity6K — total ?; splits: test (-1)

Metrics

• AP (primary) — range: [0, 1]
  • Average Precision computed over predicted bounding boxes and ground truth labels, typically averaged across IoU thresholds.
• Accuracy — range: [0, 1]
  • Standard classification accuracy: the proportion of images correctly assigned to their true class label.
• BLEU/ROUGE/METEOR/BertScore — range: [0, 1]
  • Standard n-gram overlap (BLEU, ROUGE), word-level alignment (METEOR), and embedding-based similarity (BertScore) between generated captions and ground truth references.
• mAP — range: [0, 1]
  • Mean Average Precision averaged across pairwise thresholds: IoU thresholds of .3, .4, .5, .6, .7 and METEOR thresholds of 0, .05, .1, .15, .2, .25.

Input / output format

Input: Image (and optional text prompt for classification/captioning tasks).

Output: Object detection: bounding boxes and class labels. Zero-shot classification: predicted class label. Image captioning: generated text description. Dense captioning: bounding boxes and descriptive text for each entity.

Scoring recipe

def compute_metrics(predictions, gold, task):
    if task == 'detection':
        return compute_AP(predictions, gold, iou_thresholds=[0.5])
    elif task == 'classification':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task == 'captioning':
        return bleu, rouge, meteor, bertscore(predictions, gold)
    elif task == 'dense_captioning':
        ious = [0.3, 0.4, 0.5, 0.6, 0.7]
        meteors = [0.0, 0.05, 0.1, 0.15, 0.2, 0.25]
        aps = []
        for iou in ious:
            for met in meteors:
                aps.append(compute_AP(predictions, gold, iou=iou, met_threshold=met))
        return mean(aps)

Common pitfalls

• Fine-tuning or training the baseline models, which violates the explicit zero-shot/frozen weight evaluation protocol.
• Using non-standard IoU or METEOR thresholds for dense captioning, as the benchmark requires averaging across the specific ranges (.3-.7 for IoU, .0-.25 for METEOR).
• Ignoring the exact prompt/instruction format required by each baseline model, which can significantly alter zero-shot performance.

Evidence (verbatim from paper)

For object detection, we select Average Precision (AP) as the evaluation metric. For zero-shot image classification, we take the standard accuracy as the evaluation metric. For image captioning, we adopted the BLEU, ROUGE, Meteor, and BertScore as evaluation metrics. For the dense captioning task, we take mean Average Precision (mAP) as the evaluation metric.

Citation

@misc{qiu2024entity6k,
  title={Entity6K: A Large Open-Domain Evaluation Dataset for Real-World Entity Recognition},
  author={Qiu et al. (2024)},
  year={2024},
  note={arXiv:2403.12339}
}

• arXiv: 2403.12339