Temporal Degradation Eval

temporal-degradation-eval

A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity — Longpre et al. (2023) (arXiv:2305.13169, 2023)

What this evaluates

Evaluates how temporal misalignment between pretraining/fine-tuning data and evaluation data impacts model performance across classification and summarization benchmarks.

Datasets

• PubCLS — total ?; splits: test (-1)
• NewSum — total ?; splits: test (-1)
• TwiERC — total ?; splits: test (-1)
• AIC — total ?; splits: test (-1)
• PoliAff — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels out of total instances.
• Rouge-L — range: [0, 1]
  • Recall-oriented F-score computed on the longest common subsequence between generated and reference summaries.

Input / output format

Input: Text instances (news articles, tweets, science articles) or prompts for summarization, partitioned by publication year.

Output: Predicted class labels for classification tasks; generated text summaries for NewSum.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type == 'classification':
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return correct / len(golds)
    elif task_type == 'summarization':
        return rouge_l_score(predictions, golds)

Common pitfalls

• Temporal degradation is often conflated with general domain shift; this protocol specifically isolates year-based misalignment.
• Performance drops are persistent and not recoverable via fine-tuning, so evaluating only on aligned years masks the true degradation effect.

Evidence (verbatim from paper)

Table 10: Left: Full results on the PubCLS temporal task splits from (Luu et al., 2021). This task evaluates news article source classification, measured with Accuracy. Right: Full results on the NewSum summarization task temporal splits from (Luu et al., 2021), evaluated in Rouge-L.

Citation

@misc{longpre2023pretrainers,
  title={A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity},
  author={Longpre et al. (2023)},
  year={2023},
  note={arXiv:2305.13169}
}

• arXiv: 2305.13169