Coedit Text Editing Eval

coedit-text-editing-eval

CoEdIT: Text Editing by Task-Specific Instruction Tuning — Raheja et al. (2023) (arXiv:2305.09857, 2023)

What this evaluates

Evaluates a model's ability to follow task-specific and composite text editing instructions across diverse tasks like grammar correction, simplification, coherence, style transfer, and paraphrasing. It also assesses generalization to unseen instructions and human-perceived writing efficiency.

Datasets

• JFLEG — total ?; splits: test (-1)
• TurkCorpus — total ?; splits: test (-1)
• ASSET — total ?; splits: test (-1)
• ITER (Coherence split) — total ?; splits: test (-1)
• DISCOFUSE — total ?; splits: test (-1)
• ITER (Iterative text revision) — total ?; splits: test (-1)
• GYAFC — total ?; splits: test (-1)
• WNC — total ?; splits: test (-1)
• MRPC — total ?; splits: test (-1)
• STS — total ?; splits: test (-1)
• QQP — total ?; splits: test (-1)

Metrics

• evaluation metrics (primary) — range: varies
  • Varies by dataset; includes automatic metrics (e.g., BLEU, METEOR, ROUGE, ASSET score) and human evaluation for writing efficiency. Exact formulas, conventions, and scoring details are provided in Appendix B.

Input / output format

Input: Source text concatenated with a natural language editing instruction specifying the desired transformation.

Output: Edited text generated via greedy decoding.

Scoring recipe

for dataset in [JFLEG, TurkCorpus, ASSET, ITER_Coherence, DISCOFUSE, ITER_Iterative, GYAFC, WNC, MRPC, STS, QQP]:
    predictions = model.generate(input_text + instruction)
    gold = dataset.gold_references
    score = compute_dataset_specific_metric(predictions, gold)  # Details in Appendix B
    aggregate_scores.append(score)
# Human evaluation also conducted for RQ3 (efficiency)

Common pitfalls

• Metrics are not explicitly defined in the main text; researchers must consult Appendix B for exact formulas and conventions.
• Evaluation includes both automatic metrics and human evaluation for writing efficiency, which are reported separately.
• All model outputs are generated using greedy decoding unless otherwise specified.

Evidence (verbatim from paper)

To assess the editing capabilities of CoEdit, we perform evaluations on standard test sets sourced from a variety of text editing task benchmarks, most notably, EDITEVAL (Dwivedi-Yu et al., 2022). ... Detailed descriptions of each dataset and its evaluation metrics are in §B

Citation

@misc{raheja2023coedit,
  title={CoEdIT: Text Editing by Task-Specific Instruction Tuning},
  author={Raheja et al. (2023)},
  year={2023},
  note={arXiv:2305.09857}
}

• arXiv: 2305.09857