Cfq Eval

cfq-eval

*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task — Tsarkov et al. (arXiv:2012.08266, 2020)

What this evaluates

This benchmark evaluates compositional generalization in semantic parsing by measuring how well models translate anonymized natural language questions into executable SPARQL queries. It specifically probes the ability to generalize to unseen combinations of logical rules (compounds) while maintaining familiarity with individual rules (atoms), using Maximum Compound Divergence splits to ensure fair yet challenging evaluation.

Datasets

• CFQ — total ?; splits: MCD (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted SPARQL queries out of the total number of test instances.

Input / output format

Input: Anonymized natural language question

Output: A SPARQL query string

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip() == gold.strip():
        correct += 1
return correct / len(gold_labels)

Common pitfalls

• Evaluating on splits with high atom divergence rather than Maximum Compound Divergence (MCD) splits fails to isolate compositional generalization.
• Models may exploit entity memorization if anonymization is not strictly enforced, leading to inflated accuracy on unseen compositions.

Evidence (verbatim from paper)

The authors release a number of MCD splits for CFQ, and show that there is a strong negative correlation between the accuracy of three standard sequence-to-sequence architectures and the compound divergence.

Citation

@misc{tsarkov2020cfq,
  title={*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task},
  author={Tsarkov et al.},
  year={2020},
  note={arXiv:2012.08266}
}

• arXiv: 2012.08266