Deeptilebars Eval

deeptilebars-eval

DeepTileBars: Visualizing Term Distribution for Neural Information Retrieval — Tang and Yang (2018) (arXiv:1811.00606, 2018)

What this evaluates

Evaluates neural information retrieval models on ad-hoc web search and benchmark datasets by measuring how well they rank relevant documents using segment-level matching and discourse structure modeling.

Datasets

• TREC 2010-2012 Web Track — total 38948; splits: train (-1), val (-1), test (-1)
• LETOR 4.0 MQ2008 — total 15211; splits: test (-1)

Metrics

• ERR@20 — range: [0, 1]
  • Expected Reciprocal Rank at cutoff 20. Computes the expected value of the reciprocal rank of the first relevant document, accounting for graded relevance judgments.
• nDCG@20 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff 20. Measures ranking quality by summing graded relevance scores discounted by position, normalized by the ideal DCG.
• P@20 — range: [0, 1]
  • Precision at cutoff 20. The fraction of relevant documents in the top 20 ranked results, using binary relevance judgments.
• nDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff 5. Same as nDCG@20 but evaluated at the top 5 results.
• nDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff 10. Same as nDCG@20 but evaluated at the top 10 results.
• P@5 — range: [0, 1]
  • Precision at cutoff 5. Fraction of relevant documents in the top 5 ranked results.
• P@10 — range: [0, 1]
  • Precision at cutoff 10. Fraction of relevant documents in the top 10 ranked results.

Input / output format

Input: Query and document text. Documents are segmented into topical units via TextTiling. The model constructs a query-document interaction matrix.

Output: Ranked list of documents per query, or a relevance score for each document.

Scoring recipe

import math
def precision_at_k(ranked_docs, relevant_docs, k):
    return sum(1 for d in ranked_docs[:k] if d in relevant_docs) / k
def ndcg_at_k(ranked_docs, rel_scores, k):
    dcg = sum(r / math.log2(i + 2) for i, r in enumerate(rel_scores[:k]))
    ideal = sorted(rel_scores, reverse=True)
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(ideal[:k]))
    return dcg / idcg if idcg > 0 else 0.0
def err_at_k(ranked_docs, rel_scores, k):
    err = 0.0
    for i, r in enumerate(rel_scores[:k]):
        util = r / (i + 1)
        cum_rel = sum(rel_scores[:i])
        prob_not_seen = 1.0 - (cum_rel / 3.0)
        err += util * prob_not_seen
    return err

Common pitfalls

• 10-fold cross-validation is used for TREC Web, but exact train/val/test split sizes are not reported.
• LETOR results are taken directly from original papers rather than re-run, which may introduce implementation differences.
• DUET was excluded from TREC evaluation due to insufficient training data.
• Graded relevance is used for ERR/nDCG, while binary relevance is used for Precision; mixing them without clarification causes confusion.

Evidence (verbatim from paper)

The official metrics used in TREC 2010-2012 Web Track ad-hoc tasks include Expected Reciprocal Rank (ERR)@20 (?), normalized Discounted Cumulative Gain (nDCG)@20 (?) and Precision (P)@20. ERR and nDCG handle graded relevance judgments and Precision handles binary relevance judgements. We also test our full model on the most recent MQ2008 dataset for LETOR 4.0. LETOR 4.0 is a common benchmark used by Neu-IR models. LETOR MQ2008 contains 784 queries and 15,211 annotated documents. The official metrics used in LETOR includes nDCG and Precision at different cutoff positions.

Citation

@misc{tang2018deeptilebars,
  title={DeepTileBars: Visualizing Term Distribution for Neural Information Retrieval},
  author={Tang and Yang (2018)},
  year={2018},
  note={arXiv:1811.00606}
}

• arXiv: 1811.00606