Ma4div Diversity Eval

ma4div-diversity-eval

MA4DIV: Multi-Agent Reinforcement Learning for Search Result Diversification — Chen et al. (2024) (arXiv:2403.17421, 2024)

What this evaluates

Evaluates a model's ability to rerank search results to maximize topic diversity, ensuring that top-k results cover multiple relevant subtopics for a given query rather than just maximizing single-topic relevance.

Datasets

• TREC 2009~2012 Web Track — total 198; splits: train (-1), test (-1)
• DU-DIV — total 4473; splits: train (3578), test (895)

Metrics

• α-NDCG@10 (primary) — range: [0, 1]
  • Discounted Cumulative Gain that penalizes redundancy by weighting relevance by the fraction of new subtopics covered at each position, normalized by ideal DCG.
• ERR-IA@10 — range: [0, 1]
  • Expected Reciprocal Rank for Information Adequacy, computing the probability that a user finds a relevant subtopic at each rank position.
• S-recall@10 — range: [0, 1]
  • Subtopic Recall, measuring the fraction of all relevant subtopics for a query that appear in the top-k ranked documents.
• α-NDCG@5 — range: [0, 1]
  • Same as α-NDCG@10 but evaluated at cut-off k=5.
• ERR-IA@5 — range: [0, 1]
  • Same as ERR-IA@10 but evaluated at cut-off k=5.
• S-recall@5 — range: [0, 1]
  • Same as S-recall@10 but evaluated at cut-off k=5.

Input / output format

Input: A search query and a candidate list of documents (typically top-15 or subsampled 30) with pre-computed embeddings.

Output: A ranked permutation of the candidate documents.

Scoring recipe

def compute_alpha_ndcg(ranked_docs, doc_subtopics, k):
    covered = set()
    dcg = 0.0
    for i, doc in enumerate(ranked_docs[:k]):
        rel = len(doc_subtopics[doc])
        new_topics = len(doc_subtopics[doc] - covered)
        covered.update(doc_subtopics[doc])
        dcg += (2**rel - 1) / math.log2(i + 2) * (new_topics / rel if rel > 0 else 0)
    # idcg computed similarly on ideal ranking
    return dcg / idcg

Common pitfalls

• The TREC dataset is extremely small (198 queries), causing severe overfitting; the authors subsample to 30 documents per query and use 5-fold cross-validation to mitigate this.
• Embedding models differ across datasets (Doc2vec for TREC vs. BERT for DU-DIV), making cross-dataset metric comparisons potentially confounded by representation quality.
• Statistical significance is only reported for comparisons against MA4DIV (marked with '*'), not for all baseline pairs.

Evidence (verbatim from paper)

Table 3 and 4 report the performances of all baselines and our MA4DIV in terms of the six diversity performance metrics, including α-NDCG@5, α-NDCG@10, ERR-IA@5, ERR-IA@10, S-recall@5, S-recall@10 on the TREC web track datasets and the DU-DIV dataset respectively.

Citation

@misc{chen2024ma4div,
  title={MA4DIV: Multi-Agent Reinforcement Learning for Search Result Diversification},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2403.17421}
}

• arXiv: 2403.17421