Trec Ir Eval

trec-ir-eval

Least Information Modeling for Information Retrieval — Ke (2012) (arXiv:1205.0312, 2012)

What this evaluates

Evaluates information retrieval ranking methods on ad hoc and routing tasks across diverse text collections. It probes how well term weighting schemes capture semantic relevance and handle difficult or verbose queries compared to classic baselines like BM25 and TF-IDF.

Datasets

• TREC Collections and Topics — total ?; splits: TREC 2 routing (51-100) (-1), TREC 4 ad hoc (201-250) (-1), TREC 7 ad hoc (351-400) (-1), TREC 2005 HARD/Robust (50) (-1)

Metrics

• MAP (primary) — range: [0, 1]
  • Mean Average Precision: arithmetic average of average precision scores across all queries.
• gMAP — range: [0, 1]
  • Geometric Mean Average Precision: geometric average of average precision scores, sensitive to poorly performed tasks.
• P@10 — range: [0, 1]
  • Best precision at rank 10: maximum precision value observed in the top 10 ranked documents.
• nDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10: measures ranking quality by discounting gains logarithmically based on position, normalized by ideal DCG.
• Recall and Precision — range: [0, 1]
  • Standard retrieval metrics measuring fraction of relevant documents retrieved and fraction of retrieved documents that are relevant.

Input / output format

Input: Document collection (TIPSTER, TREC Disks, AQUAINT I) and query topics (TREC 2, 4, 7, 2005) with human relevance judgments (QRELs).

Output: Ranked list of documents for each query, scored using the proposed term weighting methods (LIB, LIF, LIB+LIF, LIB*LIF, LICos) or baselines (TF-IDF, BM25).

Scoring recipe

def compute_metrics(ranks, qrels, k=10):
    map_scores, ndcg_scores = [], []
    for q, ranked_docs in ranks.items():
        rel_docs = set(qrels[q])
        hits, ap = 0, 0.0
        for i, doc in enumerate(ranked_docs, 1):
            if doc in rel_docs:
                hits += 1
                ap += hits / i
        map_scores.append(ap / len(rel_docs))
        dcg, idcg = 0.0, 0.0
        for i, doc in enumerate(ranked_docs[:k], 1):
            rel = 1 if doc in rel_docs else 0
            dcg += rel / math.log2(i + 1)
        ideal_rels = sorted(rel_docs, reverse=True)[:k]
        for i, rel in enumerate(ideal_rels, 1):
            idcg += rel / math.log2(i + 1)
        ndcg_scores.append(dcg / idcg if idcg > 0 else 0.0)
    return sum(map_scores)/len(map_scores), sum(ndcg_scores)/len(ndcg_scores)

Common pitfalls

• Confusing arithmetic MAP with geometric MAP (gMAP), which penalizes poor performance on specific queries more heavily.
• Forgetting that experiments were run both with and without stemming, which significantly affects term weighting scores.
• Assuming all TREC topics are standard ad hoc queries; TREC 2 includes routing tasks with verbose concept lists that require different query generation strategies.

Evidence (verbatim from paper)

Evaluation metrics included mean average precision with arithmetic averaging (MAP) and geometric (gMAP), best precision at rank 10, normalized discounted cumulative gain at 10 ( $nDCG_{10}$ ) and recall precision. While arithmetic average MAP provides a simple mean score across multiple queries, the geometric average (gMAP) is sensitive to poorly performed tasks and is a very useful metric developed for 2005 HARD track [27].

Citation

@misc{ke2012least,
  title={Least Information Modeling for Information Retrieval},
  author={Ke (2012)},
  year={2012},
  note={arXiv:1205.0312}
}

• arXiv: 1205.0312