Cogs Eval

cogs-eval

COGS: A Compositional Generalization Challenge Based on Semantic Interpretation — Kim et al. (2020) (arXiv:2010.05465, 2020)

What this evaluates

Probes compositional generalization in semantic parsing by evaluating whether models can correctly map out-of-distribution natural language sentences to their corresponding lambda calculus semantic representations. It specifically tests structural generalizations like argument role reversal, depth generalization, and voice transformation, as well as lexical generalizations.

Datasets

• COGS — total ?; splits: train (-1), dev (-1), test (-1), gen (-1); repo https://github.com/najoungkim/COGS

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact-match accuracy: an output sequence is considered correct only if it exactly matches the gold sequence.

Input / output format

Input: Natural language sentence

Output: Lambda calculus semantic representation with index-based labels

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
    return correct / len(golds)

Common pitfalls

• Models may produce structurally valid but token-mismatched outputs (e.g., wrong indices or extra tokens), which are strictly counted as incorrect under exact-match scoring.
• Performance exhibits high variance across random seeds; evaluating a single run is unreliable and the protocol mandates averaging over five runs.
• Depth generalization failures often stem from early stopping or excessive output length rather than pure compositional inability, complicating error attribution.

Evidence (verbatim from paper)

All architectures performed well on the development and test sets (Table 2), with little variability across runs (Figure 2a, green dots). By contrast, generalization accuracy was low across the board, and was characterized by much higher variance (blue dots). An output sequence is considered correct only if it exactly matches the gold sequence.

Citation

@misc{kim2020cogs,
  title={COGS: A Compositional Generalization Challenge Based on Semantic Interpretation},
  author={Kim et al. (2020)},
  year={2020},
  note={arXiv:2010.05465}
}

• arXiv: 2010.05465