# Editeval Eval

> Evaluates instruction-based text editing capabilities across modular tasks such as simplification, fluency, coherence, paraphrasing, neutralization, and information updating. It measures how well models follow prompts to improve or modify text while preserving intended meaning. Use when the user wants to benchmark on EditEval Benchmark, or asks about evaluating this task. Reports SARI.

- Skill: `qhjqhj00/editeval-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/editeval-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/editeval-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/editeval-eval

---


# editeval-eval

> EditEval: An Instruction-Based Benchmark for Text Improvements — Dwivedi-Yu et al. (2022) (arXiv:2209.13331, 2022)

## What this evaluates

Evaluates instruction-based text editing capabilities across modular tasks such as simplification, fluency, coherence, paraphrasing, neutralization, and information updating. It measures how well models follow prompts to improve or modify text while preserving intended meaning.

## Datasets

- **EditEval Benchmark** — total ?; splits: test (-1); repo https://github.com/facebookresearch/EditEval

## Metrics

- `SARI` **(primary)** — range: [0, 100]
  - Standard text editing metric that computes precision, recall, and F1 for added, deleted, and kept words relative to the original text and reference edits. Higher scores indicate better editing performance.
- `GLEU` — range: [0, 100]
  - N-gram overlap metric used for fluency, clarity, and coherence tasks. Computed as the geometric mean of n-gram precisions against reference texts.
- `EM` — range: [0, 1]
  - Exact match accuracy used for neutralization tasks. Scores 1 if the prediction exactly matches the reference, 0 otherwise.
- `Update-R1` — range: [0, 100]
  - Recall metric for updating tasks, measuring the proportion of correctly updated information spans present in the prediction.

## Input / output format

**Input**: Original text paired with an instruction/prompt specifying the editing task (e.g., 'Simplify this text', 'Neutralize this text', 'Update with new information').

**Output**: Edited text generated by the model.

## Scoring recipe

```python
def compute_sari(prediction, original, references):
    pred_tokens = tokenize(prediction)
    orig_tokens = tokenize(original)
    ref_tokens = [tokenize(r) for r in references]
    # Compute precision, recall, F1 for added, deleted, kept words across 1-4 grams
    p_add, r_add, f_add = compute_ngram_stats(pred_tokens, orig_tokens, ref_tokens, 'add')
    p_del, r_del, f_del = compute_ngram_stats(pred_tokens, orig_tokens, ref_tokens, 'del')
    p_keep, r_keep, f_keep = compute_ngram_stats(pred_tokens, orig_tokens, ref_tokens, 'keep')
    # Weighted F1 (weights typically 0.5 for add, 0.25 for del, 0.25 for keep)
    sari = 0.5 * f_add + 0.25 * f_del + 0.25 * f_keep
    return sari * 100
```

## Common pitfalls

- Variance in model performance across prompts does not necessarily indicate task difficulty; easier tasks can show high variance.
- Prompts optimized for maximum performance often differ from those that yield robust performance across models.
- Metric correlations can be misleading; e.g., ROUGE and SARI are inversely correlated, and high scores on one metric do not guarantee high scores on another.

## Evidence (verbatim from paper)

> The first numbers for each task are SARI scores; additional metrics are GLEU for fluency, clarity, and coherence, EM for neutralization, Update-R1 for updating.

## Citation

```bibtex
@misc{dwivediyu2022editeval,
  title={EditEval: An Instruction-Based Benchmark for Text Improvements},
  author={Dwivedi-Yu et al. (2022)},
  year={2022},
  note={arXiv:2209.13331}
}
```

- arXiv: 2209.13331