Disastir Eval

disastir-eval

DisastIR: A Comprehensive Information Retrieval Benchmark for Disaster Management — Yin et al. (2025) (arXiv:2505.15856, 2025)

What this evaluates

Evaluates information retrieval models on disaster management queries across multiple intents and categories. It measures how well models retrieve relevant passages from a large-scale domain-specific corpus under both exact and approximate nearest neighbor search settings.

Datasets

• DisastIR — total 9600; splits: test (-1); repo https://github.com/KaiYin97/Disaster_IR

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. It measures the quality of the ranked list of retrieved passages by discounting relevance scores logarithmically based on their position, normalized by the ideal DCG.

Input / output format

Input: A user query and a set of candidate passages (either the full corpus for exact search or a pre-filtered candidate set for ANN search).

Output: Top-k ranked list of passages for each query.

Scoring recipe

def compute_ndcg_at_10(relevance_scores, k=10):
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
    ideal_scores = sorted(relevance_scores, reverse=True)
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal_scores[:k]))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Comparing ANN and exact retrieval results directly without accounting for the candidate generation bottleneck in ANN, which can artificially lower scores.
• Aggregating scores across all 48 tasks/intents/categories without reporting per-task performance, as model strengths vary significantly across disaster categories.
• Assuming general-domain embedding models will transfer effectively to disaster queries without domain-specific fine-tuning or prompt adaptation.

Evidence (verbatim from paper)

We evaluate model performance under two retrieval settings, exact and ANN, using Normalized Discounted Cumulative Gain at rank 10 (NDCG@10) as the primary metric, consistent with prior works.

Citation

@misc{yin2025disastir,
  title={DisastIR: A Comprehensive Information Retrieval Benchmark for Disaster Management},
  author={Yin et al. (2025)},
  year={2025},
  note={arXiv:2505.15856}
}

• arXiv: 2505.15856