Editeval Eval

editeval-eval

EditEval: An Instruction-Based Benchmark for Text Improvements — Dwivedi-Yu et al. (2022) (arXiv:2209.13331, 2022)

What this evaluates

Evaluates instruction-based text editing capabilities across modular tasks such as simplification, fluency, coherence, paraphrasing, neutralization, and information updating. It measures how well models follow prompts to improve or modify text while preserving intended meaning.

Datasets

• EditEval Benchmark — total ?; splits: test (-1); repo https://github.com/facebookresearch/EditEval

Metrics

• SARI (primary) — range: [0, 100]
  • Standard text editing metric that computes precision, recall, and F1 for added, deleted, and kept words relative to the original text and reference edits. Higher scores indicate better editing performance.
• GLEU — range: [0, 100]
  • N-gram overlap metric used for fluency, clarity, and coherence tasks. Computed as the geometric mean of n-gram precisions against reference texts.
• EM — range: [0, 1]
  • Exact match accuracy used for neutralization tasks. Scores 1 if the prediction exactly matches the reference, 0 otherwise.
• Update-R1 — range: [0, 100]
  • Recall metric for updating tasks, measuring the proportion of correctly updated information spans present in the prediction.

Input / output format

Input: Original text paired with an instruction/prompt specifying the editing task (e.g., 'Simplify this text', 'Neutralize this text', 'Update with new information').

Output: Edited text generated by the model.

Scoring recipe

def compute_sari(prediction, original, references):
    pred_tokens = tokenize(prediction)
    orig_tokens = tokenize(original)
    ref_tokens = [tokenize(r) for r in references]
    # Compute precision, recall, F1 for added, deleted, kept words across 1-4 grams
    p_add, r_add, f_add = compute_ngram_stats(pred_tokens, orig_tokens, ref_tokens, 'add')
    p_del, r_del, f_del = compute_ngram_stats(pred_tokens, orig_tokens, ref_tokens, 'del')
    p_keep, r_keep, f_keep = compute_ngram_stats(pred_tokens, orig_tokens, ref_tokens, 'keep')
    # Weighted F1 (weights typically 0.5 for add, 0.25 for del, 0.25 for keep)
    sari = 0.5 * f_add + 0.25 * f_del + 0.25 * f_keep
    return sari * 100

Common pitfalls

• Variance in model performance across prompts does not necessarily indicate task difficulty; easier tasks can show high variance.
• Prompts optimized for maximum performance often differ from those that yield robust performance across models.
• Metric correlations can be misleading; e.g., ROUGE and SARI are inversely correlated, and high scores on one metric do not guarantee high scores on another.

Evidence (verbatim from paper)

The first numbers for each task are SARI scores; additional metrics are GLEU for fluency, clarity, and coherence, EM for neutralization, Update-R1 for updating.

Citation

@misc{dwivediyu2022editeval,
  title={EditEval: An Instruction-Based Benchmark for Text Improvements},
  author={Dwivedi-Yu et al. (2022)},
  year={2022},
  note={arXiv:2209.13331}
}

• arXiv: 2209.13331