Logiccat Eval

logiccat-eval

LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning — Liu et al. (2025) (arXiv:2505.18744, 2025)

What this evaluates

This benchmark evaluates an LLM's ability to perform complex multi-step logical reasoning and chain-of-thought decomposition to generate correct SQL queries from natural language questions. It probes capabilities in mathematical deduction, physical knowledge integration, and cross-domain analytical querying.

Datasets

• LogicCat — total 4038; splits: test (4038)

Metrics

• Valid Execution Syntax (VES) — range: [0, 1]
  • Measures the proportion of predicted SQL queries that are syntactically valid and can be executed against the database without raising an error.
• Execution Accuracy (EX) (primary) — range: [0, 1]
  • Represents the proportion of predicted SQL queries whose execution results exactly match those of the ground-truth SQL query, measuring semantic and logical correctness.

Input / output format

Input: Natural language question with database schema/context, provided via a standardized zero-shot prompting template.

Output: A single SQL query string generated by the model.

Scoring recipe

def evaluate(predictions, gold_sqls, db_engine):
    ves_scores = []
    ex_scores = []
    for i, pred_sql in enumerate(predictions):
        try:
            pred_result = db_engine.execute(pred_sql)
            ves_scores.append(1)
            gold_result = db_engine.execute(gold_sqls[i])
            ex_scores.append(1 if pred_result == gold_result else 0)
        except Exception:
            ves_scores.append(0)
            ex_scores.append(0)
    return sum(ves_scores)/len(ves_scores), sum(ex_scores)/len(ex_scores)

Common pitfalls

• Models may generate syntactically valid SQL that executes but returns incorrect results, which counts as 1 for VES but 0 for EX.
• The evaluation uses a zero-shot setting with temperature=0; non-deterministic sampling or higher temperatures will break evaluation consistency.
• Missing or incorrectly formatted database schema in the prompt leads to false syntax errors or incorrect execution results.

Evidence (verbatim from paper)

To ensure clarity and consistency across our analysis, we adopt two primary evaluation metrics from the survey by (Qin et al. [2022]): Valid Execution Syntax (VES): This metric measures the proportion of predicted SQL queries that are syntactically valid and can be executed against the database without raising an error. It assesses the model’s ability to generate syntactically correct SQL, regardless of the result’s correctness. Execution Accuracy (EX): This is the stricter metric, representing the proportion of predicted SQL queries whose execution results exactly match those of the ground-truth SQL query. This measures the model’s ability to generate a semantically and logically correct query.

Citation

@misc{liu2025logiccat,
  title={LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2505.18744}
}

• arXiv: 2505.18744