Counterfact Edit Eval

counterfact-edit-eval

Beyond Hard Writes and Rigid Preservation: Soft Recursive Least-Squares for Lifelong LLM Editing — Xinyu Wang et al. (2026) (arXiv:2601.15686, 2026)

What this evaluates

Evaluates the effectiveness and stability of sequential LLM knowledge editing methods. It measures how well a model updates a specific fact while preserving related paraphrases, neighboring facts, generation fluency, and overall general capabilities over thousands of edits.

Datasets

• CounterFact — total ?; splits: test (-1)
• GLUE_MMLU_GSM8K_HumanEval_MBPP — total ?; splits: test (-1)

Metrics

• Efficacy (primary) — range: percent
  • Percentage of generated outputs that successfully rewrite the target fact as instructed.
• Generalization — range: percent
  • Percentage of paraphrased prompts that successfully generate the target fact.
• Specificity — range: percent
  • Percentage of neighborhood prompts that correctly preserve the original fact (i.e., do not leak the new fact).
• Fluency — range: other
  • Generation entropy of the model's output distribution; lower values indicate more fluent or coherent text.
• Consistency — range: percent
  • Reference score measuring similarity between generated text and a ground-truth reference completion.

Input / output format

Input: Prompt containing a factual statement to be edited, typically with a masked target entity or a direct replacement instruction.

Output: Generated text completion following the prompt.

Scoring recipe

def score_edit_eval(predictions, golds, paraphrases, neighbors, references):
    efficacy = sum(1 for p in predictions if matches_target(p, golds)) / len(predictions)
    generalization = sum(1 for p in predictions if matches_target(p, golds)) / len(paraphrases)
    specificity = sum(1 for p in predictions if matches_original(p, neighbors)) / len(neighbors)
    fluency = compute_generation_entropy(predictions)
    consistency = compute_reference_similarity(predictions, references)
    return {'Efficacy': efficacy, 'Generalization': generalization, 'Specificity': specificity, 'Fluency': fluency, 'Consistency': consistency}

Common pitfalls

• Sequential editing requires evaluating at multiple checkpoints (e.g., every 1K edits) to track degradation, not just at the final step.
• Fluency is measured via generation entropy, which may not correlate perfectly with human-perceived naturalness.
• CounterFact's neighborhood prompts must be strictly filtered to ensure they do not inadvertently contain the target fact, which would inflate Specificity scores.

Evidence (verbatim from paper)

Following prior work, we use the CounterFact dataset*(Meng et al., [2022])*. We report Efficacy (rewrite success), Generalization (paraphrase success), Specificity (neighborhood success), Fluency (generation entropy), and Consistency (reference score).

Citation

@misc{wang2026rlsedit,
  title={Beyond Hard Writes and Rigid Preservation: Soft Recursive Least-Squares for Lifelong LLM Editing},
  author={Xinyu Wang et al. (2026)},
  year={2026},
  note={arXiv:2601.15686}
}

• arXiv: 2601.15686