Weatherqa Eval

weatherqa-eval

Weather-R1: Logically Consistent Reinforcement Fine-Tuning for Multimodal Reasoning in Meteorology — Wu et al. (2026) (arXiv:2601.14044, 2026)

What this evaluates

Evaluates multimodal reasoning capabilities in the meteorological domain, specifically testing a model's ability to interpret weather maps and answer domain-specific multiple-choice questions. It also measures cross-task generalization and the logical consistency of the model's reasoning chains versus final answers.

Datasets

• WeatherQA — total ?; splits: train (-1), val (-1), test (-1)
• ScienceQA — total 324; splits: test (324)

Metrics

• Multiple-choice accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions out of the total number of questions.
• Self-Contra proportion — range: percent
  • Percentage of instances where the model's generated reasoning path contradicts its final selected answer, as determined by an LLM judge.

Input / output format

Input: Multimodal input consisting of meteorological images (e.g., weather maps, wind vectors) paired with multiple-choice questions.

Output: A single letter corresponding to the selected multiple-choice option (e.g., A, B, C, D).

Scoring recipe

def compute_metrics(predictions, golds, reasoning_paths):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    accuracy = (correct / len(golds)) * 100
    self_contra_count = 0
    for path, answer in zip(reasoning_paths, predictions):
        if judge_model(path, answer) == "contradictory":
            self_contra_count += 1
    self_contra_prop = (self_contra_count / len(predictions)) * 100
    return {"accuracy": accuracy, "self_contra_proportion": self_contra_prop}

Common pitfalls

• The cross-task evaluation protocol requires testing on tasks not seen during training (off-diagonal cells in Fig. 4), which can mask catastrophic forgetting or overfitting to specific meteorological modalities.
• Self-Contra detection relies on an external LLM judge (gpt-oss-20b), meaning reported inconsistency rates are subject to the judge's parsing and logical verification capabilities.
• The ScienceQA OOD benchmark uses a manually curated subset of 324 weather/climate questions rather than the full dataset, limiting direct comparability with standard ScienceQA scores.

Evidence (verbatim from paper)

We use WeatherQA as our training dataset and evaluation benchmark, following its defined cross-task protocol (see Section[2.1]). Additionally, to measure the model’s OOD generalization capability, we curate a test set from ScienceQA*[[16]]*, which consists of 324 multiple-choice questions related to weather and climate. Multiple-choice accuracy is used as the metric for all experiments.

Citation

@misc{wu2026weatherr1,
  title={Weather-R1: Logically Consistent Reinforcement Fine-Tuning for Multimodal Reasoning in Meteorology},
  author={Wu et al. (2026)},
  year={2026},
  note={arXiv:2601.14044}
}

• arXiv: 2601.14044