Interactive Retrieval Eval

interactive-retrieval-eval

Interactive Retrieval Based on Wikipedia Concepts — Zhang (2014) (arXiv:1412.8281, 2014)

What this evaluates

Evaluates the effectiveness of interactive document retrieval using user-identified Wikipedia concepts for query expansion and re-ranking. It also tests methods for selecting the most relevant Wikipedia concepts from a large pool based on semantic relevance and document ranking signals.

Datasets

• TREC Filtering-02, HARD-03, HARD-05 — total ?; splits: test (-1)

Metrics

• MAP (primary) — range: [0, 1]
  • Mean Average Precision; averages the precision values at each rank where a relevant document is retrieved across all queries.
• P@10 (primary) — range: [0, 1]
  • Precision at rank 10; calculates the fraction of relevant documents found in the top 10 retrieved results.
• NDCG — range: [0, 1]
  • Normalized Discounted Cumulative Gain; measures the quality of a predicted concept ranking against a human assessor's ground truth ranking, normalized by the ideal ranking score.

Input / output format

Input: Query string, initial top-ranked documents (from BM25/IQ baseline), and a pool of candidate Wikipedia concepts.

Output: Ranked list of documents (for retrieval evaluation) or ranked list of candidate Wikipedia concepts (for concept selection evaluation).

Scoring recipe

def compute_metrics(pred_docs, gold_docs, pred_concepts, gold_concepts):
    rel = [1 if d in gold_docs else 0 for d in pred_docs]
    ap = sum(sum(rel[:i+1])/(i+1) for i, r in enumerate(rel) if r) / max(sum(rel), 1)
    p10 = sum(rel[:10]) / 10
    def dcg(r, k): return sum(r[:k] / math.log2(i + 2) for i in range(k))
    idcg = dcg(sorted(gold_concepts, reverse=True), len(gold_concepts))
    ndcg = dcg(pred_concepts, len(pred_concepts)) / max(idcg, 1e-9)
    return {'MAP': ap, 'P@10': p10, 'NDCG': ndcg}

Common pitfalls

• The feedback method combines multiple ranking signals (WA, CM, CT, AT, RD) that must be combined as per Equation 16; using only one signal will significantly underperform the reported results.
• Statistical significance is only claimed for improvements marked with * (paired t-test, p<0.05); unmarked improvements should not be treated as statistically significant.
• NDCG for concept selection is evaluated against a human assessor's ranking, not an automatic ground truth, requiring careful handling of human judgment variance.

Evidence (verbatim from paper)

According to Table 1, MAP is significantly improved on all three data sets, P@10 is significantly improved on HARD-03 and HARD-05, while not significantly on Filtering-02, probably because P@10 of the baseline run is already high on this data set. Table 3 shows the performances of all concept selection methods. For each method, the NDCG is calculated based on how well each method's ranking of concepts corresponds to that of the human assessor.

Citation

@misc{zhang2014interactive,
  title={Interactive Retrieval Based on Wikipedia Concepts},
  author={Zhang (2014)},
  year={2014},
  note={arXiv:1412.8281}
}

• arXiv: 1412.8281