Parrot Eval

parrot-eval

PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation — Zhou et al. (2025) (arXiv:2509.23338, 2025)

What this evaluates

This benchmark evaluates an LLM's ability to accurately translate SQL queries across different database systems and dialects. It probes the model's capacity to handle system-specific syntax, semantic equivalence, and edge-case safeguards without relying on superficial string matching.

Datasets

• PARROT — total 598; splits: test (598); repo https://github.com/weAIDB/PARROT

Metrics

• Acc_EX (primary) — range: [0, 1]
  • Execution Accuracy: 1 if the translated SQL executes successfully on the target database and returns a result set exactly matching the ground truth SQL's output; 0 otherwise.
• Acc_RES — range: [0, 1]
  • Result Accuracy: Measures whether the execution of the translated SQL yields results consistent with the ground truth, prioritizing semantic correctness over string similarity.

Input / output format

Input: A source SQL query, the target database system/dialect specification, and detailed problem instructions provided via a well-crafted prompt.

Output: A single translated SQL statement written in the target dialect.

Scoring recipe

def evaluate(predictions, golds, db_conn):
    ex_correct = 0
    res_correct = 0
    for pred_sql, gold_sql in zip(predictions, golds):
        try:
            pred_res = db_conn.execute(pred_sql)
            gold_res = db_conn.execute(gold_sql)
            if set(pred_res) == set(gold_res):
                ex_correct += 1
                res_correct += 1
        except Exception:
            pass
    return ex_correct / len(predictions), res_correct / len(predictions)

Common pitfalls

• Relying on lexical/string similarity instead of actual database execution to verify correctness.
• Failing to handle dialect-specific runtime safeguards (e.g., division-by-zero checks, ROLLUP syntax, or NULL handling) which cause execution failures despite syntactically plausible translations.
• Performance degradation on lengthy queries due to hallucination or lost-in-the-middle effects, requiring careful token management or segmentation strategies.

Evidence (verbatim from paper)

We adopt the evaluation metrics (i.e., $Acc_{EX}$ and $Acc_{RES}$ ) defined in Section 4.

Citation

@misc{zhou2025parrot,
  title={PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation},
  author={Zhou et al. (2025)},
  year={2025},
  note={arXiv:2509.23338}
}

• arXiv: 2509.23338