Scivisagentbench Eval

scivisagentbench-eval

SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents — Ai et al. (2026) (arXiv:2603.29139, 2026)

What this evaluates

Evaluates agentic systems' ability to perform scientific data analysis and visualization workflows. It probes outcome correctness, process behavior, and computational efficiency across real-world scientific domains and tools.

Datasets

• SciVisAgentBench — total ?; splits: test (-1)

Metrics

• outcome correctness (primary) — range: [0, 1]
  • Assessed via a multimodal outcome-centric evaluation pipeline combining LLM-based judging with deterministic verifiers, including image metrics, code checkers, and rule-based validators.
• execution cost — range: other
  • Quantified through time and token usage, as well as benchmark-level computational tractability to ensure repeated evaluations without excessive overhead.

Input / output format

Input: Task specifications grounded in real-world SciVis tools, pipelines, and data formats; agents operate over authentic visualization environments (e.g., ParaView, napari, VMD).

Output: Visualization outcomes, states, or scripts that serve as the reference for evaluation.

Scoring recipe

# Pseudo-code based on the multimodal outcome-centric evaluation pipeline
def evaluate(agent, task):
    outcome = agent.execute(task)
    # Combine LLM judging with deterministic verifiers
    llm_verdict = llm_judge(outcome, task.reference)
    det_verdict = verify(outcome, task.reference, 
                         tools=["image_metrics", "code_checkers", "rule_based_validators"])
    score = combine(llm_verdict, det_verdict)
    cost = measure_time_and_tokens(agent)
    return score, cost

Common pitfalls

• Stochastic model behavior can hinder full determinism, requiring repeated trials and consistency measures to quantify variability.
• Process-level trajectory assessment is currently infeasible due to reproducibility and stability challenges, limiting evaluation to outcome correctness.
• Multimodal reasoning and prompt robustness limitations may cause misalignment between human experts and LLM judges.

Evidence (verbatim from paper)

Outcome correctness serves as the primary evaluation axis in the current release, while process-level analysis remains future work due to reproducibility and stability challenges in trajectory assessment. Efficiency is quantified through execution cost, including time and token usage, as well as benchmark-level computational tractability, ensuring that evaluations can be run repeatedly without excessive overhead.

Citation

@misc{ai2026scivisagentbench,
  title={SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents},
  author={Ai et al. (2026)},
  year={2026},
  note={arXiv:2603.29139}
}

• arXiv: 2603.29139