Radarqa Eval

radarqa-eval

RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts — He et al. (2025) (arXiv:2508.12291, 2025)

What this evaluates

Evaluates multi-modal large language models on weather radar forecast quality analysis, specifically testing their ability to perform quantitative rating of radar frames/sequences and generate qualitative assessment reports. It probes domain-specific meteorological understanding, temporal pattern evolution tracking, and alignment with expert judgment.

Datasets

• RQA-70K — total 70000; splits: train (-1); repo https://github.com/hexmSeeU/RadarQA

Metrics

• accuracy (primary) — range: [0, 1]
  • Computed separately for each general attribute in rating tasks. Measures the percentage of correctly predicted categorical labels (e.g., False Alarm, Miss, High Value, Sharpness) against expert ground truth.
• BERTScore — range: [0, 1]
  • Measures semantic similarity between generated assessment text and ground truth using contextual BERT embeddings.
• BLEU — range: [0, 1]
  • Standard n-gram precision metric evaluating the overlap between generated text and reference ground truth.
• ROUGE_L — range: [0, 1]
  • Longest common subsequence metric measuring text generation overlap and recall.
• METEOR — range: [0, 1]
  • Evaluates text generation based on alignment of synonyms, stems, and exact matches between prediction and reference.
• GPT-4 Score — range: [0, 10]
  • LLM-as-a-judge score where GPT-4 rates the model's response from 0 to 10 based on relevance, accuracy, and level of detail compared to the ground truth.

Input / output format

Input: Multi-modal input consisting of weather radar frames (for frame tasks) or sequences of frames (for sequence tasks), accompanied by task-specific prompts requesting either structured attribute ratings or open-ended quality assessment reports.

Output: For rating tasks: a structured JSON object with predefined keys corresponding to general attributes. For assessment tasks: a free-form textual report describing the radar forecast quality, evolution, and deficiencies.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_gpt4_score(predictions, gold):
    scores = []
    for pred, gt in zip(predictions, gold):
        prompt = f'Rate response (0-10) on relevance, accuracy, detail vs ground truth:\nResponse: {pred}\nGT: {gt}'
        score = call_gpt4_judge(prompt)  # Returns float 0-10
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• Models often struggle to correctly map rainfall intensities to color mappings (High Value Match), leading to artificially low accuracy on that specific attribute.
• Sequence assessment tasks are significantly harder than frame tasks due to the need for temporal coherence; models trained only on static frames fail to capture dynamic evolution.
• GPT-4 judge scores can be sensitive to prompt phrasing and may not perfectly align with human expert ratings on content accuracy and information density.

Evidence (verbatim from paper)

For the rating tasks, we adopt accuracy as the evaluation metric. Specifically, we prompt MLLMs to generate responses in a structured JSON format with predefined keys. Accuracy is then computed separately for each general attributes. For the assessment tasks, we employ standard metrics, including BERTScore, BLEU, ROUGE_L, and METEOR. Following Liu et al. (2023); You et al. (2024a), we also incorporate the GPT-4 score, where the model’s response is rated from 0 to 10 based on relevance, accuracy, and level of detail with respect to the ground truth.

Citation

@misc{he2025radarqa,
  title={RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts},
  author={He et al. (2025)},
  year={2025},
  note={arXiv:2508.12291}
}

• arXiv: 2508.12291