Depth Eval

depth-eval

DEPTH: Discourse Education through Pre-Training Hierarchically — Bamberger et al. (2024) (arXiv:2405.07788, 2024)

What this evaluates

Evaluates the effectiveness of a hierarchically pre-trained encoder-decoder model (DEPTH) against a standard T5 baseline on discourse understanding, natural language inference, sentiment analysis, grammar checking, and instruction following.

Datasets

• MNLI — total ?; splits: test (-1)
• SST2 — total ?; splits: test (-1)
• CoLA — total ?; splits: test (-1)
• DiscoEval — total ?; splits: test (-1)
• Natural Instructions — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted labels out of the total number of instances. For instruction following tasks, exact-match accuracy is typically used.

Input / output format

Input: Text pairs or single sentences for classification tasks (MNLI, SST2, CoLA, SP, DC); instruction prompts for Natural Instructions.

Output: Class labels for classification tasks; generated text responses for Natural Instructions.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Evaluating intermediate checkpoints without full fine-tuning convergence may misrepresent final performance.
• Comparing From-Scratch and Continuous Pre-Training baselines requires careful accounting of pre-training token counts and initialization states.
• Example packing differences between DEPTH and T5 baselines can confound performance comparisons if not controlled.

Evidence (verbatim from paper)

We evaluate our models on natural language inference (MNLI, Williams et al. ([2018])), sentiment analysis (SST2, Socher et al. ([2013])), and grammar (CoLA, Warstadt et al. ([2019])) within the GLUE benchmark (Wang et al., [2018]). We also use the DiscoEval suite (Chen et al., [2019]) to evaluate models on their understanding of discourse. We use two tasks from DiscoEval: Sentence Permutation (SP) and Discourse Coherence (DC). SP involves identifying the correct position of a removed, while DC involves predicting whether or not a paragraph was coherent. Finally, we measure our model’s generative abilities on the Natural Instructions (NI) dataset (Mishra et al., [2022]), which measures the ability of LMs to follow instructions, and served as a benchmark for NanoT5 (Nawrot, [2023]).

Citation

@misc{bamberger2024depth,
  title={DEPTH: Discourse Education through Pre-Training Hierarchically},
  author={Bamberger et al. (2024)},
  year={2024},
  note={arXiv:2405.07788}
}

• arXiv: 2405.07788