Scinav Eval

scinav-eval

SciNav: A General Agent Framework for Scientific Coding Tasks — Zhang et al. (2026) (arXiv:2603.20256, 2026)

What this evaluates

Evaluates an autonomous agent's capability to generate executable scientific and data-science code by measuring execution reliability and task-goal satisfaction. It probes the model's ability to navigate constrained search budgets while producing outputs that match predefined success criteria across diverse disciplinary benchmarks.

Datasets

• ScienceAgentBench — total ?; splits: test (-1)
• DA-Code — total 100; splits: test (100)

Metrics

• Valid Execution Rate (VER) — range: percent
  • Binary indicator (1 if the generated program executes without runtime errors, 0 otherwise). Averaged across all tasks.
• Success Rate (SR) (primary) — range: percent
  • Binary indicator (1 if the program output satisfies the task goal, such as passing predefined success criteria, matching expected predictions, or producing a high-quality visualization; 0 otherwise). Averaged across tasks. This is the headline metric.
• DA-Code Evaluation Suite Metrics — range: varies
  • Task-specific scoring methodology customized per output type (tables, charts, machine learning predictions). Each task defines its own required outputs, metrics, and options.
• API Cost (Cost) — range: other
  • Average dollar cost required to complete a single task, calculated by summing API usage charges across all agent steps (generation, debugging, comparison) and dividing by the number of tasks.

Input / output format

Input: Natural language task description specifying a scientific or data-science programming goal, often with constraints or expected output formats.

Output: Python code (program) intended to solve the task, which may produce numerical outputs, tables, charts, or visualizations.

Scoring recipe

def evaluate(program, task_gold):
    # Valid Execution Rate
    try:
        exec(program)
        ver = 1.0
    except Exception:
        ver = 0.0
    # Success Rate
    sr = check_task_goal(program, task_gold)  # matches predictions, passes criteria, or high-quality viz
    # API Cost
    cost = calculate_api_cost(program)  # sum of token/API charges for all steps
    return ver, sr, cost

Common pitfalls

• Success Rate (SR) is not a single universal formula; it relies on task-specific evaluation programs or rubrics defined during benchmark annotation.
• DA-Code does not use a single aggregate score; each task has a uniquely defined output type and customized metric, requiring the evaluator to route to the correct scoring function.
• Reported metrics are heavily constrained by the fixed exploration budget (max 10 steps, 5 initial solutions, 3 debug steps), so results are not directly comparable to unconstrained baselines.

Evidence (verbatim from paper)

For ScienceAgentBench, we use two key metrics. (1) Valid Execution Rate (VER) measures whether a program can execute without errors. (2) Success Rate (SR) assesses whether the output satisfies the specific task goal, such as passing predefined task success criteria, matching expected predictions, or producing a high-quality visualization. These criteria are implemented as task-specific evaluation programs during the benchmark annotation process. Among the reported metrics, SR (Success Rate) is the most important as it directly reflects task success. For DA-Code, we leverage their evaluation suite, which supports multiple tasks through configurable setups, where each task is uniquely identified and defined with its required outputs, metrics, and options. Their tailored scoring methodology assesses agent performance across diverse outputs such as tables, charts, and machine learning predictions, with metrics customized for each output type. (3) API Cost (Cost) reports the average dollar cost required to complete a single task using the agent.

Citation

@misc{zhang2026scinav,
  title={SciNav: A General Agent Framework for Scientific Coding Tasks},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2603.20256}
}

• arXiv: 2603.20256