Ir Triplet Eval

ir-triplet-eval

Distributed Representations of Sentences and Documents — Quoc V. Le et al. (2014) (arXiv:1405.4053, 2014)

What this evaluates

Evaluates the model's ability to capture semantic similarity between paragraphs for information retrieval. It tests whether fixed-length vector representations can effectively distinguish query-related documents from irrelevant ones using a triplet ranking protocol.

Datasets

• Information Retrieval (Paragraph Vectors) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• error rate (primary) — range: percent
  • Percentage of triplets where the distance between the two query-related paragraphs is not smaller than the distance between the first paragraph and the random paragraph.

Input / output format

Input: Triplets of paragraphs: two paragraphs retrieved for the same query, and one randomly sampled paragraph from a different query.

Output: Binary decision indicating whether the first two paragraphs belong to the same query (i.e., distance(d1, d2) < distance(d1, d3)).

Scoring recipe

d12 = distance(paragraph1, paragraph2)
d13 = distance(paragraph1, paragraph3)
error = 1 if d12 >= d13 else 0
error_rate = sum(errors) / total_triplets

Common pitfalls

• The dataset is custom-derived from search engine snippets and is not publicly released.
• Baselines use TF-IDF weighting, and the 'Weighted Bag-of-bigrams' baseline involves learning a linear weighting matrix.
• Distance metric is not explicitly specified in the text, but the protocol relies on relative ordering of distances.

Evidence (verbatim from paper)

The results show that Paragraph Vector works well and gives a 32% relative improvement in terms of error rate. The fact that the paragraph vector method significantly outperforms bag of words and bigrams suggests that our proposed method is useful for capturing the semantics of the input text.

Citation

@misc{le2014distributed,
  title={Distributed Representations of Sentences and Documents},
  author={Quoc V. Le et al. (2014)},
  year={2014},
  note={arXiv:1405.4053}
}

• arXiv: 1405.4053