S2sservicebench Eval

s2sservicebench-eval

S2SServiceBench: A Multimodal Benchmark for Last-Mile S2S Climate Services — Chenyue Li et al. (arXiv:2602.14017, 2026)

What this evaluates

Evaluates whether multimodal LLMs and agentic systems can reliably generate actionable decision support from operational S2S climate service products. It probes three core capabilities: actionable signal comprehension, uncertainty-conditioned decision-making handoffs, and evidence-grounded planning under dynamic hazards.

Datasets

• S2SServiceBench — total 1000; splits: test (1000)

Metrics

• Rubric Score (CT, ACT, TTH, EG, FC, UC) (primary) — range: [0, 5]
  • Each dimension is scored on a 0–5 scale based on expert-defined criteria: CT (Context Tailoring), ACT (Actionability), TTH (Trigger Time Horizon), EG (Evidence Grounding), FC (Feasibility & Constraints), UC (Uncertainty & Confidence). The reported value is the mean score across all evaluation items for a given topic and capability level.
• Overall Score (Ovr) — range: [0, 5]
  • The arithmetic mean of the six rubric dimension scores, aggregated across all topics and capability levels.

Input / output format

Input: Multimodal operational S2S climate service products (e.g., plots, data tables) paired with task prompts requiring decision support deliverables at specific capability levels (Level I: signal comprehension; Level II: decision-making handoff; Level III: decision analysis & planning).

Output: Decision support deliverables ranging from short-slot structured completions (Level I) to detailed, uncertainty-conditioned planning and trigger specifications (Levels II–III).

Scoring recipe

def compute_metrics(predictions, gold, rubric):
    dim_scores = {dim: [] for dim in ['CT', 'ACT', 'TTH', 'EG', 'FC', 'UC']}
    for pred, gold_item in zip(predictions, gold):
        for dim in dim_scores:
            dim_scores[dim].append(expert_rubric_score(pred, gold_item, dim, rubric))
    metrics = {dim: mean(scores) for dim, scores in dim_scores.items()}
    metrics['Ovr'] = mean(list(metrics.values()))
    return metrics

Common pitfalls

• Treating the benchmark as a single 'multimodal reasoning' score; performance is highly product- and capability-level-dependent, with strong variance across topics.
• Assuming standardized agentic workflows (e.g., DeepAgent) consistently improve performance; they can degrade decision-facing capabilities at Levels II and III.
• Overestimating operational readiness based on high Evidence Grounding (EG) scores; bottlenecks typically lie in trigger/time clarity, feasibility, and uncertainty handling rather than multimodal grounding.

Evidence (verbatim from paper)

Table 2 Rubric scores for GPT-5.2 at Level 2 and Level 3 (mean±std, where std = sqrt). Rubrics: CT = Context Tailoring; ACT = Actionability; TTH = Trigger Time Horizon; EG = Evidence Grounding; FC = Feasibility & Constraints; UC = Uncertainty & Confidence.

Citation

@misc{li2026s2sservicebench,
  title={S2SServiceBench: A Multimodal Benchmark for Last-Mile S2S Climate Services},
  author={Chenyue Li et al.},
  year={2026},
  note={arXiv:2602.14017}
}

• arXiv: 2602.14017