Clinsql Eval

clinsql-eval

Patient-Similarity Cohort Reasoning in Clinical Text-to-SQL — Shen et al. (2026) (arXiv:2601.09876, 2026)

What this evaluates

Evaluates clinical text-to-SQL capabilities by requiring models to generate executable BigQuery queries that perform multi-table joins, temporal reasoning, and patient-similarity cohort analysis on electronic health record data.

Datasets

• CLINSQL — total ?; splits: val (-1), test (-1)

Metrics

• SQL Score — range: percent
  • Rubric-based metric specifically designed for clinical text-to-SQL tasks to evaluate query correctness, clinical relevance, and adherence to schema constraints.
• Execution Score (primary) — range: percent
  • Execution accuracy percentage; measures whether the generated BigQuery query runs successfully against the target database and returns the expected results.

Input / output format

Input: Natural language clinical question and database schema/context (including table structures, column definitions, and optional schema hints).

Output: A single executable BigQuery SQL query, typically extracted from a fenced code block.

Scoring recipe

def compute_sql_score(pred_sql, gold_sql, rubric):
    return rubric.evaluate(pred_sql, gold_sql)

def compute_exec_score(pred_sql, db_context):
    try:
        result = execute_bigquery(pred_sql, db_context)
        return 1.0 if result_matches_gold(result, gold_sql) else 0.0
    except ExecutionError:
        return 0.0

# Aggregate over dataset
sql_acc = mean([compute_sql_score(p, g, rubric) for p, g in predictions])
exec_acc = mean([compute_exec_score(p, ctx) for p, ctx in predictions])

Common pitfalls

• Models frequently fail on first-pass execution, requiring up to two self-refinement rounds with error feedback to succeed.
• Cohort specification errors are common when models replace explicit ICD or itemid constraints with loose keyword heuristics.
• Schema mismatches and omitted required columns often trigger validation checks, lowering the SQL Score.

Evidence (verbatim from paper)

We evaluate all models on CLINSQL using rubric-based metrics specifically designed for clinical text-to-SQL tasks. Our primary evaluation metrics are the SQL Score and the Execution Score.

Citation

@misc{shen2026clinsql,
  title={Patient-Similarity Cohort Reasoning in Clinical Text-to-SQL},
  author={Shen et al. (2026)},
  year={2026},
  note={arXiv:2601.09876}
}

• arXiv: 2601.09876