Hans Eval

hans-eval

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference — McCoy et al. (2019) (arXiv:1902.01007, 2019)

What this evaluates

Probes whether neural NLI models rely on superficial syntactic heuristics (e.g., lexical overlap, subsequence matching) rather than genuine logical reasoning by presenting structurally similar counterexamples where heuristics lead to incorrect predictions.

Datasets

• HANS — total ?; splits: test (-1); repo https://github.com/tommccoy1/hans

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy computed over the mapped binary labels. Models output probabilities for entailment, contradiction, and neutral; the highest-scoring label is selected and mapped to entailment or non-entailment before comparison with gold.

Input / output format

Input: A premise sentence and a hypothesis sentence.

Output: A single classification label: 'entailment' or 'non-entailment'.

Scoring recipe

def score(predictions, gold):
    mapped_preds = []
    for pred in predictions:
        if pred in ['contradiction', 'neutral']:
            mapped_preds.append('non-entailment')
        else:
            mapped_preds.append('entailment')
    correct = sum(1 for p, g in zip(mapped_preds, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Mapping contradiction and neutral to a single 'non-entailment' label discards fine-grained error analysis and may mask model confidence differences.
• SPINN evaluation depends on an external constituency parser (Stanford PCFG), so parsing errors can be conflated with model reasoning failures.
• Models are trained on MNLI but evaluated on HANS, so low accuracy reflects poor out-of-distribution generalization to synthetic heuristic traps rather than baseline NLI capability.

Evidence (verbatim from paper)

We trained all models on MNLI. MNLI uses three labels (entailment, contradiction, and neutral). We chose to annotate HANS with two labels only (entailment and non-entailment) because the distinction between contradiction and neutral was often unclear for our cases. For evaluating a model on HANS, we took the highest-scoring label out of entailment, contradiction, and neutral; we then translated contradiction or neutral labels to non-entailment.

Citation

@misc{mccoy2019right,
  title={Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference},
  author={McCoy et al. (2019)},
  year={2019},
  note={arXiv:1902.01007}
}

• arXiv: 1902.01007