Geo880 Atis Eval

geo880-atis-eval

Learning a Neural Semantic Parser from User Feedback — Iyer et al. (2017) (arXiv:1704.08760, 2017)

What this evaluates

Evaluates a neural semantic parser's ability to map natural language utterances directly to executable SQL queries. It probes compositional generalization and schema grounding by measuring whether predicted queries return the exact same results as gold queries on a target database.

Datasets

• GEO880 — total 880; splits: train (600), test (280)
• ATIS — total 5418; splits: train (4473), dev (497), test (448)

Metrics

• denotation accuracy (primary) — range: [0, 1]
  • Accuracy computed by executing the predicted SQL query on the target database and checking if the returned result set exactly matches the gold result set.

Input / output format

Input: Natural language utterances, with entities anonymized by replacing them with their corresponding types and rare words (frequency=1) replaced with UNK tokens.

Output: A single SQL query string.

Scoring recipe

def compute_denotation_accuracy(predictions, golds, database):
    correct = 0
    for pred_sql, gold_sql in zip(predictions, golds):
        pred_result = execute_sql(pred_sql, database)
        gold_result = execute_sql(gold_sql, database)
        if pred_result == gold_result:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Confusing denotation accuracy (result-set matching) with logical form exact match; the paper explicitly uses denotations while some baselines report logical form accuracy.
• Assuming standard train/test splits apply uniformly; GEO880 uses cross-validation on the training set for hyperparameter tuning, while ATIS uses a fixed development set.

Evidence (verbatim from paper)

We report test set accuracy of our SQL query predictions by executing them on the target database and comparing the result with the true result. Tables 2 and 3 show test accuracies based on denotations for our model on GEO880 and ATIS respectively, compared with previous work.

Citation

@misc{iyer2017learning,
  title={Learning a Neural Semantic Parser from User Feedback},
  author={Iyer et al. (2017)},
  year={2017},
  note={arXiv:1704.08760}
}

• arXiv: 1704.08760