Mimicsql Eval

mimicsql-eval

CBR-to-SQL: Rethinking Retrieval-based Text-to-SQL using Case-based Reasoning in the Healthcare Domain — Nguyen et al. (2026) (arXiv:2603.05569, 2026)

What this evaluates

Evaluates a model's ability to generate correct SQL queries from natural language clinical questions in the healthcare domain. It specifically probes retrieval-based reasoning, robustness to noisy or ambiguous medical terminology, and generalization under data scarcity conditions.

Datasets

• MIMICSQL — total 10000; splits: train (8000), val (1000), test (1000)

Metrics

• Execution Accuracy (primary) — range: [0, 1]
  • Proportion of queries where the executed predicted SQL returns results identical to the executed gold SQL. Formula: $N_{EX}/N$.
• Logical Form Accuracy — range: [0, 1]
  • Proportion of queries where the predicted SQL string exactly matches the gold SQL string. Formula: $N_{LF}/N$.
• Brittleness — range: [0, 1]
  • Performance drop when top-ranked retrieved cases are stochastically removed based on rank. Formula: $Acc_{original} - Acc_{drop}$.

Input / output format

Input: Natural language clinical question paired with database context (schema and retrieved case templates/examples from either the Complete Database or Incomplete Database environment).

Output: A single SQL query string.

Scoring recipe

def score(predictions, golds, db):
    correct_exec = 0
    correct_lf = 0
    for pred, gold in zip(predictions, golds):
        if execute_sql(pred, db) == execute_sql(gold, db):
            correct_exec += 1
        if pred == gold:
            correct_lf += 1
    acc_ex = correct_exec / len(predictions)
    acc_lf = correct_lf / len(predictions)
    # Brittleness: re-evaluate after rank-based dropout of retrieved cases
    acc_drop = evaluate_with_dropout(predictions, golds, k=5, p_top=1)
    delta_brittle = acc_ex - acc_drop
    return acc_ex, acc_lf, delta_brittle

Common pitfalls

• The Incomplete Database (IDB) setting uses a clustered subset of 774 training examples, not the full test set, which drastically changes retrieval dynamics and generalization expectations.
• Brittleness calculation depends on the retrieval ranking order and a fixed dropout probability schedule ($p_{top}=1$); altering $k$ or the ranking method invalidates the metric.
• Execution accuracy requires an exact schema match and a functional SQLite instance; minor column name or type mismatches cause false negatives.

Evidence (verbatim from paper)

Execution Accuracy: $Acc_{\text{EX}}$ measures the proportion of queries that produce correct results: $Acc_{\text{EX}}=N_{\text{EX}}/N$, where $N_{\text{EX}}$ is the number of queries that return execution results identical to those of the gold queries, and $N$ is the total number of queries. Logical Form Accuracy: $Acc_{\text{LF}}$ measures the proportion of exact SQL matches: $Acc_{\text{LF}}=N_{\text{LF}}/N$, where $N_{\text{LF}}$ counts exact matches to the gold queries and $N$ is the total number of queries.

Citation

@misc{nguyen2026cbrtosql,
  title={CBR-to-SQL: Rethinking Retrieval-based Text-to-SQL using Case-based Reasoning in the Healthcare Domain},
  author={Nguyen et al. (2026)},
  year={2026},
  note={arXiv:2603.05569}
}

• arXiv: 2603.05569