Wximpactbench Eval

wximpactbench-eval

WXImpactBench: A Disruptive Weather Impact Understanding Benchmark for Evaluating Large Language Models — Yongan Yu et al. (arXiv:2505.20249, 2025)

What this evaluates

Evaluates large language models' ability to understand and classify disruptive weather impacts from historical and modern newspaper articles, and to answer related questions by ranking relevant information. It specifically probes models' capacity to handle climate-related polysemy, extract nuanced societal responses, and correctly identify passages without weather impacts.

Datasets

• WXImpactBench — total 350; splits: test (350); repo https://github.com/Michaelyya/WXImpactBench

Metrics

• F1-score (primary) — range: [0, 1]
  • Macro-averaged F1-score across the six impact categories, historical/modern articles, and context length settings. Computed as the harmonic mean of precision and recall for each label, then averaged.
• row-wise accuracy — range: [0, 1]
  • Strict metric requiring correct classification of all six impact labels for a given article. Formula: (1/N) * sum_{i=1}^{N} prod_{j=1}^{6} I(y_hat_i^j == y_i^j), where I is the indicator function.
• accuracy — range: [0, 1]
  • Standard accuracy averaged across the six impact categories, historical and modern articles, and context lengths.
• Hit@1 — range: [0, 1]
  • Proportion of queries where the top-ranked answer/document matches the ground truth.
• nDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 5, measuring ranking quality with logarithmic discounting for lower positions.
• Recall@5 — range: [0, 1]
  • Proportion of relevant documents/answers found within the top 5 ranked results.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank, averaging the reciprocal of the rank of the first relevant item for each query.

Input / output format

Input: Historical or modern newspaper articles (text). For the mixed-context version, articles are split into ~250-token segments. Prompts are provided in Appendix C.2 (classification) and C.3 (QA).

Output: For classification: a set of 6 binary labels (one per impact category: infrastructural, political, financial, ecological, agricultural, human health). For QA: a ranked list of candidate answers/documents.

Scoring recipe

def row_wise_accuracy(preds, golds, num_labels=6):
    correct = sum(1 for p, g in zip(preds, golds) if all(pi == gi for pi, gi in zip(p, g)))
    return correct / len(golds)

# For classification: compute F1, accuracy, and row-wise accuracy per category/context, then average.
# For QA: compute Hit@1, nDCG@5, Recall@5, MRR on ranked lists.
# All LLMs run with temperature=0; final scores are averaged over 3 runs.

Common pitfalls

• Decomposing multi-label classification into multiple independent binary calls instead of simultaneous prediction (the benchmark requires a single LLM call for all six labels).
• Assuming mixed-context chunk labels are automatically inherited from the original article; they are independently annotated, creating negative examples (chunks with no labels) that must be handled correctly.
• Ignoring the sliding window mechanism used in the QA ranking task, which segments articles into three chunks and ranks independently before merging, to mitigate long-context noise.

Evidence (verbatim from paper)

For multi-label classification task, we use F1-score, accuracy, and row-wise accuracy as evaluation metrics. The evaluation via F1-score and accuracy are averaged across the six impact categories, historical and modern articles, and the effect of different context lengths. Compared to the common F1-score and accuracy, the row-wise accuracy is a strict metric that requires more accurate output as the model should correctly classify all six impact labels for a given article, defined as

where $N$ is the number of samples, $\hat{y}{i}^{j}$ denotes the predicted label for the $j$-th category in the $i$-th sample, $y{i}^{j}$ is the corresponding ground-truth label, and $\mathcal{I}(\cdot)$ is the indicator function.

Citation

@misc{yu2025wximpactbench,
  title={WXImpactBench: A Disruptive Weather Impact Understanding Benchmark for Evaluating Large Language Models},
  author={Yongan Yu et al.},
  year={2025},
  note={arXiv:2505.20249}
}

• arXiv: 2505.20249