Commonsense Retrieval Eval

commonsense-retrieval-eval

Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles — Ye et al. (2022) (arXiv:2211.16504, 2022)

What this evaluates

This benchmark probes the commonsense reasoning capabilities of vision-language models by evaluating their ability to match images to text riddles (or vice versa) where the subject entity is replaced with a demonstrative pronoun. It specifically tests relational knowledge retrieval and generalization to unseen knowledge triples.

Datasets

• DANCE Diagnostic Set — total ?; splits: train (-1), test-seen (-1), test-unseen (-1); repo https://github.com/pleaseconnectwifi/DANCE

Metrics

• Acc@50 (primary) — range: [0, 1]
  • Models compute alignment scores between a query (riddle or image) and a candidate set of 50 items. The candidates are sorted by score, and accuracy is calculated as the fraction of queries where the ground-truth match appears in the top 50 predictions.

Input / output format

Input: A text riddle with a hidden subject (or an image) paired with a candidate list of 50 images (or riddles).

Output: A ranked list of the 50 candidate images (or riddles) sorted by the model's alignment score.

Scoring recipe

def compute_acc50(predictions, gold):
    correct = 0
    for pred_list, gt_idx in zip(predictions, gold):
        if gt_idx in pred_list[:50]:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Hard negatives are constructed using ConceptNet graph distance (RelatedTo, DistinctFrom, Antonym) rather than random sampling, making the evaluation sensitive to semantic entity correlation rather than just visual similarity.
• The test set is divided into 'test-seen' and 'test-unseen' splits based on knowledge triple presence in training, but all images are strictly held out from the training set to prevent memorization.
• Perplexity is used as the underlying scoring mechanism for alignment, but the reported metric is strictly top-50 accuracy, not perplexity values.

Evidence (verbatim from paper)

For automatic evaluation, we adopt perplexity score as the evaluation metric, following the works*[13, 50]*. In the experiment, we set the candidate number for each sample as $50$, with the number of positive samples $n$ between $1$ to $15$, and measure the average accuracy of retrieved ground truth inference, denoted as $Acc@50$.

Citation

@misc{ye2022improving,
  title={Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles},
  author={Ye et al. (2022)},
  year={2022},
  note={arXiv:2211.16504}
}

• arXiv: 2211.16504