Evolvingqa Eval

evolvingqa-eval

Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models — Kim et al. (2023) (arXiv:2311.08106, 2023)

What this evaluates

Evaluates lifelong language models' ability to update outdated world knowledge while retaining new information and avoiding catastrophic forgetting. It specifically probes temporal adaptation, numerical reasoning, and the model's capacity to forget obsolete facts during continual pretraining.

Datasets

• EvolvingQA — total ?; splits: Unchanged (-1), New (-1), Outdated (-1), Updated (-1); repo https://github.com/kimyuji/EvolvingQA_benchmark

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • 1 if the predicted answer exactly matches the ground truth answer, else 0.
• F1 score — range: [0, 1]
  • Calculated by counting the common tokens between the predicted answer and the ground truth answer, averaged over the dataset.

Input / output format

Input: A question about world knowledge, typically presented as an open-book QA prompt. During pretraining, entities are masked, but evaluation uses standard question format.

Output: A predicted answer string.

Scoring recipe

def compute_metrics(predictions, golds):
    em_scores = [1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)]
    f1_scores = []
    for p, g in zip(predictions, golds):
        p_tokens = p.lower().split()
        g_tokens = g.lower().split()
        common = len(set(p_tokens) & set(g_tokens))
        f1 = 2 * common / (len(p_tokens) + len(g_tokens)) if (len(p_tokens) + len(g_tokens)) > 0 else 0.0
        f1_scores.append(f1)
    return {'EM': sum(em_scores) / len(em_scores), 'F1': sum(f1_scores) / len(f1_scores)}

Common pitfalls

• Models often fail to forget outdated knowledge, yielding similar EM/F1 scores for 'Outdated' and 'Updated' questions instead of near-zero for outdated.
• Numerical and temporal answers consistently show near-zero EM scores because updated knowledge produces negligible weight gradients during continual pretraining.

Evidence (verbatim from paper)

We measure Exact Match(EM) and F1 score, and F1 score is calculated by counting the common tokens between predicted answer and ground truth answer.

Citation

@misc{kim2023carpediem,
  title={Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models},
  author={Kim et al. (2023)},
  year={2023},
  note={arXiv:2311.08106}
}

• arXiv: 2311.08106