Sequential Model Editing Eval

sequential-model-editing-eval

Toward Ultra-Long-Horizon Sequential Model Editing — Mingda Liu et al. (arXiv:2602.02543, 2026)

What this evaluates

Evaluates the stability and performance of sequential knowledge editing methods on large language models over long horizons. It probes whether editing techniques can maintain factual accuracy, preserve general capabilities, and avoid norm blow-up or catastrophic forgetting across thousands of atomic updates.

Datasets

• CounterFact — total 20877; splits: test (20877)
• ZsRE — total 19086; splits: test (19086)
• WikiBigEdit — total ?; splits: test (-1)
• GLUE-style tasks (SST, MRPC, RTE, CoLA, MNLI) — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)

Metrics

• Efficacy (primary) — range: [0, 1]
  • Measures the proportion of edits where the model correctly generates the target fact or paraphrase. Follows standard knowledge editing exact-match or semantic similarity protocols.
• Generalization — range: [0, 1]
  • Evaluates performance on paraphrased or structurally varied prompts for the same edited fact. Reported as accuracy or semantic similarity.
• Specificity — range: [0, 1]
  • Measures the model's ability to avoid altering unrelated facts or control prompts. Calculated as 1 minus the error rate on control generations.
• Fluency — range: other
  • Assesses the naturalness and coherence of generated text post-edit, typically via perplexity or LLM-judge scoring.
• Consistency — range: [0, 1]
  • Measures cross-prompt agreement for the same edited fact, ensuring stable outputs across different phrasings.
• Collapse Point (CP@60) — range: int
  • The number of sequential edits performed before the editing score drops below a threshold (e.g., 60%), characterizing long-horizon failure.

Input / output format

Input: A sequence of atomic editing requests, each containing a source fact and a target replacement fact, fed to the model sequentially with immediate weight updates.

Output: The model's generated text response to each edit prompt (for efficacy/generalization) and to control prompts (for specificity/fluency/consistency).

Scoring recipe

def score_editing_metrics(predictions, golds, control_preds, control_golds):
    efficacy = sum(1 for p, g in zip(predictions, golds) if exact_match(p, g)) / len(golds)
    generalization = semantic_similarity_score(predictions, golds)
    specificity = 1.0 - (sum(1 for p, g in zip(control_preds, control_golds) if exact_match(p, g)) / len(control_golds))
    fluency = perplexity_score(predictions)
    consistency = cross_prompt_agreement(predictions)
    return {'Efficacy': efficacy, 'Generalization': generalization, 'Specificity': specificity, 'Fluency': fluency, 'Consistency': consistency}

Common pitfalls

• Confusing atomic sequential editing (immediate per-fact weight update) with batch or offline editing protocols.
• Overlooking the Collapse Point (CP@60) metric, which is essential for measuring long-horizon stability rather than just final accuracy.
• Assuming high efficacy implies good general capability retention; the paper explicitly shows these metrics can diverge under long horizons.

Evidence (verbatim from paper)

We adopt two widely used knowledge editing benchmarks, CounterFact*(Meng et al., [2023a])* and ZsRE*(Levy et al., [2017])*, and follow standard protocols to report Efficacy, Generalization, Specificity, Fluency, Consistency, and Score.

Citation

@misc{liu2026towardultralonghorizonsequentialmodelediting,
  title={Toward Ultra-Long-Horizon Sequential Model Editing},
  author={Mingda Liu et al.},
  year={2026},
  note={arXiv:2602.02543}
}

• arXiv: 2602.02543