Cds Search Eval

cds-search-eval

Benchmarking Clinical Decision Support Search — Nguyen et al. (2018) (arXiv:1801.09322, 2018)

What this evaluates

Evaluates clinical information retrieval systems on their ability to rank relevant medical documents for clinical decision support queries. It probes the effectiveness of query and document processing techniques such as negation detection, concept extraction, and pseudorelevance feedback in a standardized biomedical search setting.

Datasets

• TREC CDS'16 — total ?; splits: test (-1)

Metrics

• infNDCG (primary) — range: [0, 1]
  • Inferred Normalized Discounted Cumulative Gain. Adapts standard NDCG to clinical relevance judgments by incorporating inferred relevance scores rather than binary labels, as defined in the CDS track guidelines.
• infAP — range: [0, 1]
  • Inferred Average Precision. Modifies standard AP to account for inferred clinical relevance grades across the ranked list.
• R-prec — range: [0, 1]
  • Recall-Precision metric computed at the rank equal to the total number of relevant documents in the corpus.
• P@10 — range: [0, 1]
  • Precision at rank 10, measuring the fraction of relevant documents among the top 10 retrieved results.

Input / output format

Input: Clinical decision support queries paired with a corpus of medical documents from the TREC CDS track.

Output: Ranked list of documents for each query.

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions: list of doc_ids ranked by system
    # gold: dict mapping doc_id -> inferred relevance score
    scores = [gold.get(d, 0) for d in predictions]
    
    # P@10
    p10 = sum(1 for s in scores[:10] if s > 0) / 10.0
    
    # R-prec (simplified)
    r = len([s for s in scores if s > 0])
    rprec = sum(1 for s in scores[:r]) / r if r > 0 else 0.0
    
    # infAP & infNDCG follow standard IR formulas but use inferred scores
    # (exact weighting follows CDS track reference [34], [35])
    return {'infNDCG': ndcg(scores), 'infAP': ap(scores), 'R-prec': rprec, 'P@10': p10}

Common pitfalls

• Using standard binary relevance judgments instead of the track-specific inferred relevance scores required for infNDCG and infAP.
• Omitting the paired 2-sample t-test for statistical significance, which the authors explicitly require to validate improvements over baselines.
• Skipping standard preprocessing steps (Porter Stemmer and stopword removal) that are mandated for reproducible evaluation in this benchmark.

Evidence (verbatim from paper)

For evaluations, we used the four metrics proposed in the CDS track: infNDCG (inferred NDCG) [35], infAP (inferred average precision) [34], R-prec (Recall Precision) and P@10 (Precision at rank 10), with infNDCG being the main metric. The significance of improvements over the baseline is tested using a paired 2 sample t-test and is represented in two scales of 95% and 98% confidence.

Citation

@misc{nguyen2018benchmarking,
  title={Benchmarking Clinical Decision Support Search},
  author={Nguyen et al. (2018)},
  year={2018},
  note={arXiv:1801.09322}
}

• arXiv: 1801.09322