Leaderboard Zero Shot Rte Eval

leaderboard-zero-shot-rte-eval

Zero-shot Entailment of Leaderboards for Empirical AI Research — Kabongo et al. (2023) (arXiv:2303.16835, 2023)

What this evaluates

Evaluates whether pre-trained Recognizing Textual Entailment (RTE) models can generalize to unseen task-dataset-metric (TDM) extraction pairs in a zero-shot setting. It probes whether models learn genuine semantic entailment or merely memorize training distribution patterns.

Datasets

• LEADERBOARDS — total ?; splits: test (-1)

Metrics

• macro F1 (primary) — range: percent
  • Unweighted mean of the F1 scores computed for each class independently.
• micro F1 — range: percent
  • Global F1 score calculated by aggregating true positives, false positives, and false negatives across all classes before computing precision and recall.

Input / output format

Input: A document representation (DocTAET T) paired with a Task-Dataset-Metric (TDM) hypothesis for sequence-pair classification.

Output: Classification label indicating whether the document entails the TDM hypothesis.

Scoring recipe

def compute_f1(y_true, y_pred, average='macro'):
    # Calculate per-class or global TP, FP, FN
    # precision = TP / (TP + FP)
    # recall = TP / (TP + FN)
    # f1 = 2 * (precision * recall) / (precision + recall)
    # Return macro average (mean of per-class F1) or micro average (global aggregation)
    return f1_score(y_true, y_pred, average=average) * 100

Common pitfalls

• Models exhibit very high recall but extremely low precision, indicating they classify most hypotheses as true rather than identifying the correct entailment.
• The task is framed as RTE but functions as multi-class classification; models may memorize training patterns instead of learning true semantic entailment.
• Longer document representations significantly degrade performance, suggesting shorter (1-2 sentence) summaries are necessary for effective entailment.

Evidence (verbatim from paper)

ORKG-TDM ${Bert}$ in a non-zero-shot setting reported 90.8% macro F1 and 91.8% micro F1 [14]. The same model in a zero-shot setting reports performances significantly less than chance at 26.7% macro F1 and 21% micro F1. Likewise, the other SOTA model ORKG-TDM ${XLNet}$ in a non-zero-shot setting reported 91.2% macro F1 and 92.4% micro F1. This model in the zero-shot setting reported 22.3% macro F1 and 17.2% micro F1.

Citation

@misc{kabongo2023leaderboard,
  title={Zero-shot Entailment of Leaderboards for Empirical AI Research},
  author={Kabongo et al. (2023)},
  year={2023},
  note={arXiv:2303.16835}
}

• arXiv: 2303.16835