Ambiqt Eval

ambiqt-eval

Benchmarking and Improving Text-to-SQL Generation under Ambiguity — Bhaskar et al. (2023) (arXiv:2310.13659, 2023)

What this evaluates

Evaluates a model's ability to generate diverse, semantically valid SQL queries when a natural language question is ambiguous. It specifically probes whether the model can cover multiple valid interpretations of the same query within a limited set of top-k outputs.

Datasets

• AmbiQT — total ?; splits: test (-1); repo https://github.com/testzer0/AmbiQT
• SPIDER — total ?; splits: dev (-1)
• Kaggle DBQA — total ?; splits: dev (-1)

Metrics

• EitherInTopK — range: percent
  • Percentage of instances where at least one of the top-5 generated SQL queries achieves an Execution Match with any of the gold SQL queries.
• BothInTopK (primary) — range: percent
  • Percentage of instances where both valid gold SQL queries are present in the top-5 generated outputs, evaluated via Execution Match. Also referred to as Coverage.

Input / output format

Input: Natural language question paired with the corresponding database schema.

Output: Top-5 SQL queries generated by the model.

Scoring recipe

def compute_metrics(predictions, golds):
    either_count = 0
    both_count = 0
    for pred_sqls, gold_sqls in zip(predictions, golds):
        # pred_sqls: list of 5 strings, gold_sqls: list of 2 strings
        matches = [execute(p) == execute(g) for p in pred_sqls for g in gold_sqls]
        if any(matches):
            either_count += 1
        if all(any(execute(p) == execute(g) for p in pred_sqls) for g in gold_sqls):
            both_count += 1
    return either_count / len(golds), both_count / len(golds)

Common pitfalls

• Increasing beam width or the number of outputs often reduces coverage because beam search converges to a single dominant interpretation, producing only lexical variations.
• Standard diversity decoding methods (top-k, nucleus, typical sampling) fail to produce meaningful diversity due to skewed token probability distributions, often yielding identical outputs.
• Models may generate syntactically valid but semantically meaningless SQLs (e.g., nested aggregates like max(avg_age)) that do not match either gold query.

Evidence (verbatim from paper)

Evaluation Metrics. We present two types of accuracies (i) EitherInTopK - that checks if either of the gold queries feature in the top-5 outputs (ii) BothInTopK - that checks if both gold queries feature in the top-5. We only report the Execution Match (EXM) accuracies for each.

Citation

@misc{bhaskar2023ambiqtbenchmarking,
  title={Benchmarking and Improving Text-to-SQL Generation under Ambiguity},
  author={Bhaskar et al. (2023)},
  year={2023},
  note={arXiv:2310.13659}
}

• arXiv: 2310.13659