Trec Data Fusion Eval

trec-data-fusion-eval

Streamlined Data Fusion: Unleashing the Power of Linear Combination with Minimal Relevance Judgments — Xu et al. (2023) (arXiv:2309.04981, 2023)

What this evaluates

Evaluates the effectiveness of linear combination data fusion methods for information retrieval when trained on partial relevance judgments. It probes whether multiple linear regression can learn near-optimal fusion weights using only 20% or 50% of relevant documents instead of full official qrels.

Datasets

• TREC 2018-2021 Precision Medicine & Deep Learning Tracks — total ?; splits: (unstated)

Metrics

• MAP (primary) — range: [0, 1]
  • Mean Average Precision across all queries. Averages the average precision (AP) for each query, where AP is the mean of precision values at ranks where relevant documents appear.
• RP — range: [0, 1]
  • Reciprocal Precision (system-oriented metric; exact formula not detailed beyond acronym in the text).
• P@10 — range: [0, 1]
  • Precision at rank 10. The fraction of relevant documents in the top 10 positions of the fused ranked list.
• P@20 — range: [0, 1]
  • Precision at rank 20. The fraction of relevant documents in the top 20 positions of the fused ranked list.

Input / output format

Input: Per query: ranked document lists from multiple retrieval systems (19, 14, 15, or 16 runs per dataset). Relevance labels (official or partial qrels) are provided for training fusion weights.

Output: Per query: a fused ranked list of documents generated by linear combination of reciprocal rank scores from the input runs.

Scoring recipe

def compute_ir_metrics(pred_ranks, gold_rels):
    ap_sum, p10_sum, p20_sum = 0.0, 0.0, 0.0
    rel_count = 0
    for i, doc in enumerate(pred_ranks):
        if gold_rels.get(doc, 0) == 1:
            rel_count += 1
            ap_sum += rel_count / (i + 1)
        if i == 9:
            p10_sum = rel_count / 10
        if i == 19:
            p20_sum = rel_count / 20
    n_queries = len(pred_ranks)
    return {
        'MAP': ap_sum / n_queries,
        'P@10': p10_sum / n_queries,
        'P@20': p20_sum / n_queries
    }

Common pitfalls

• Using raw retrieval scores instead of the specified reciprocal rank score $score(d)=1/(60+rank(d))$ for fusion.
• Training fusion weights on full official qrels instead of the specified 20% or 50% partial pools selected via fixed-length pooling.
• Failing to split queries into odd/even partitions for the two-fold cross-validation setup.

Evidence (verbatim from paper)

Four metrics were used for evaluation: MAP, RP, P@10, and P@20. MAP and RP are system-oriented metrics, while P@10 and P@20 are user-oriented metrics. The two-fold cross-validation methodology was applied: for all the queries in a data set, we divided them into two partitions: odd-numbered and even-numbered. One partition was used for weights training and the other for testing, and vice versa.

Citation

@misc{xu2023streamlined,
  title={Streamlined Data Fusion: Unleashing the Power of Linear Combination with Minimal Relevance Judgments},
  author={Xu et al. (2023)},
  year={2023},
  note={arXiv:2309.04981}
}

• arXiv: 2309.04981