Zest Eval

zest-eval

Learning from Task Descriptions — Weller et al. (2020) (arXiv:2011.08115, 2020)

What this evaluates

Evaluates a model's ability to understand and generalize across unseen NLP tasks based solely on task descriptions, rather than few-shot examples. It probes systematic generalization across variations like paraphrasing, composition, semantic flips, and output structure changes.

Datasets

• ZEST — total ?; splits: dev (-1), test (-1)

Metrics

• Mean (primary) — range: percent
  • Average score across all instances in a split.
• C@75 — range: percent
  • Performance metric at the 75% threshold (likely accuracy or coverage at top-75 predictions).
• C@90 — range: percent
  • Performance metric at the 90% threshold (likely accuracy or coverage at top-90 predictions).

Input / output format

Input: Task description formulated as a question paired with a passage.

Output: Model prediction (exact format not specified in text).

Scoring recipe

Compute mean score across instances.
Calculate C@75 and C@90 by evaluating predictions at the 75th and 90th percentile thresholds respectively.
Report per generalization type and overall.

Common pitfalls

• Metrics are stratified by generalization type (Base, Paraphrase, etc.), so reporting only overall scores hides systematic failures.
• C@75 and C@90 thresholds are not explicitly defined in the provided text, requiring careful interpretation.
• Human performance is reported separately and significantly outperforms models, indicating a large capability gap.

Evidence (verbatim from paper)

Table 7: Detailed T5 results for ZEST with multi-task training.

Citation

@misc{weller2020learning,
  title={Learning from Task Descriptions},
  author={Weller et al. (2020)},
  year={2020},
  note={arXiv:2011.08115}
}

• arXiv: 2011.08115