Synopticbench Eval

synopticbench-eval

SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future — Higgins et al. (2026) (arXiv:2604.16451, 2026)

What this evaluates

Evaluates vision-language models' ability to generate physically grounded, spatially accurate weather forecast discussions from Numerical Weather Prediction (NWP) images. It probes the model's capacity to identify and correctly locate synoptic-scale phenomena (e.g., pressure systems) in generated text, revealing limitations of traditional lexical metrics in domain-specific evaluation.

Datasets

• SynopticBench — total 1370000; splits: test (-1)

Metrics

• Bertscore — range: [0, 1]
  • Computes contextual embedding-based similarity between generated and reference text. Scores range from 0 to 1, with 1 indicating perfect semantic match.
• ROUGE-L — range: [0, 1]
  • Measures the longest common subsequence between generated and reference text to capture sentence-level structure. Scores range from 0 to 1.
• METEOR — range: [0, 1]
  • Calculates alignment between generated and reference text using synonymy, stemming, and paraphrase matching. Scores range from 0 to 1.
• F1 — range: [0, 1]
  • Computes token-level F1 score (harmonic mean of precision and recall) between generated and reference text. Scores range from 0 to 1.
• LLM-judge — range: [0, 1]
  • Uses Gemini-2.5-Flash (or Gemini-3.1-Pro) as an automated judge to score generated discussions against reference texts. Scores range from 0 to 1.
• Space-local (primary) — range: [0, 1]
  • Evaluates alignment and coverage of generated text to specific synoptic phenomena (e.g., pressure systems) at a single location. Computed as match scores (s_m) for correct polarity identification and coverage ratios (r_c) for correct spatial location mention. Scores range from 0 to 1.
• Space-aggregate — range: [0, 1]
  • Aggregates Space-local evaluation across all locations at a given time to assess large-scale pressure feature identification. Uses the same match and coverage ratio calculations but over a larger sample size. Scores range from 0 to 1.

Input / output format

Input: Paired Numerical Weather Prediction (NWP) images with corresponding station/location context, prompting the model to generate a forecast discussion.

Output: Natural language text representing a National Weather Service Area Forecast Discussion (AFD).

Scoring recipe

def compute_metrics(predictions, golds):
    # Traditional metrics
    bertscore = compute_bertscore(predictions, golds)
    rouge_l = compute_rouge_l(predictions, golds)
    meteor = compute_meteor(predictions, golds)
    f1 = compute_f1(predictions, golds)
    llm_judge = llm_as_judge(predictions, golds, model='Gemini-2.5-Flash')
    
    # SPACE metrics (example for pressure systems)
    pred_phenomena = extract_phenomena(predictions, target='pressure')
    gold_phenomena = extract_phenomena(golds, target='pressure')
    
    # Local vs Aggregate
    local_match = calculate_polarity_match(pred_phenomena, gold_phenomena, scale='local')
    local_coverage = calculate_spatial_coverage(pred_phenomena, gold_phenomena, scale='local')
    agg_match = calculate_polarity_match(pred_phenomena, gold_phenomena, scale='aggregate')
    agg_coverage = calculate_spatial_coverage(pred_phenomena, gold_phenomena, scale='aggregate')
    
    return {
        'Bertscore': bertscore, 'ROUGE-L': rouge_l, 'METEOR': meteor,
        'F1': f1, 'LLM-judge': llm_judge,
        'Space-local': (local_match, local_coverage),
        'Space-aggregate': (agg_match, agg_coverage)
    }

Common pitfalls

• Traditional metrics (BLEU/ROUGE/Bertscore) often yield high scores for hallucinated or physically irrelevant text, failing to capture domain-specific fidelity and physical reasoning.
• SPACE evaluates one phenomenon type at a time, so it does not penalize hallucinations of other phenomena (e.g., snow) unless specifically targeted in the evaluation.
• Local vs. Aggregate scores differ significantly due to sample size; aggregate scores are inherently higher because evaluating across all locations increases the probability of mentioning relevant phenomena.

Evidence (verbatim from paper)

Space-aggregate uses all locations at a given time to create a large sample size of pressure objects matched to specific locations for evaluation. It is useful for understanding the model’s ability to determine the general large-scale pressure features that occur in the forecast. Space-local uses a single location at a given time to create a much smaller sample size of pressure objects but can be useful for understanding the model’s ability to discuss the features impacting a specific location.

Citation

@misc{higgins2026synopticbench,
  title={SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future},
  author={Higgins et al. (2026)},
  year={2026},
  note={arXiv:2604.16451}
}

• arXiv: 2604.16451