Captrack Eval

captrack-eval

CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training — Thede et al. (2026) (arXiv:2603.06610, 2026)

What this evaluates

This evaluation framework probes systematic capability drift and forgetting in large language models after post-training. It measures degradation across latent competence (knowledge, reasoning), default behavioral preferences (refusal, verbosity, formatting), and protocol compliance (instruction following, tool use, citation) in legal and medical domains.

Datasets

• CapTrack Evaluation Suite — total ?; splits: test (-1); repo https://github.com/thomsonreuters/captrack

Metrics

• average forgetting (primary) — range: percent
  • Percentage drop in performance from the pre-training baseline to the post-trained model, calculated per capability category. Computed as (baseline_score - post_training_score) / baseline_score * 100.

Input / output format

Input: Task-specific prompts or questions from the CapTrack suite, covering legal and medical domain tasks across competence, behavioral, and protocol capability groups.

Output: Model-generated text response.

Scoring recipe

def compute_forgetting(pre_scores, post_scores):
    forgetting_rates = []
    for pre, post in zip(pre_scores, post_scores):
        if pre > 0:
            forgetting_rates.append((pre - post) / pre * 100)
        else:
            forgetting_rates.append(0.0)
    return sum(forgetting_rates) / len(forgetting_rates)

Common pitfalls

• Forgetting is defined relative to a pre-training baseline, not absolute post-training performance; reporting only post-training scores misses the drift.
• The suite aggregates results across heterogeneous capability groups (CAN, WILL, HOW) that use different evaluation metrics (e.g., accuracy vs. refusal rates vs. formatting checks), so unstratified averaging obscures specific failure modes.

Evidence (verbatim from paper)

We repeat each evaluation with three random seeds and report the average results. We report average forgetting within each capability category, with faint markers indicating benchmark-level variability.

Citation

@misc{thede2026captrack,
  title={CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training},
  author={Thede et al. (2026)},
  year={2026},
  note={arXiv:2603.06610}
}

• arXiv: 2603.06610