Excgex Eval

excgex-eval

EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correction — Ye et al. (2024) (arXiv:2407.00924, 2024)

What this evaluates

This benchmark evaluates a model's ability to jointly perform Chinese grammatical error correction and generate edit-wise explanations. It probes the model's capacity to identify specific error spans, assign severity levels, and provide natural-language justifications with evidence words, linguistic rules, and revision advice.

Datasets

• EXCGEC — total 8216; splits: test (-1); repo https://github.com/THUKElab/EXCGEC

Metrics

• CLEME F0.5 (primary) — range: [0, 1]
  • Precision, Recall, and F0.5 score computed over aligned edits using the CLEME toolkit. F0.5 weights precision twice as much as recall.
• ChERRANT F0.5 — range: [0, 1]
  • Precision, Recall, and F0.5 score computed over aligned edits using the ChERRANT toolkit. F0.5 weights precision twice as much as recall.
• Hit Rate — range: [0, 1]
  • Ratio of hypothesis edits that overlap with reference edits to the total number of hypothesis edits.
• Miss Rate — range: [0, 1]
  • Ratio of reference edits with no overlap to hypothesis edits to the total number of reference edits.
• Accuracy — range: [0, 1]
  • Percentage of correctly predicted error types among hit edits.
• F1 — range: [0, 1]
  • F1 score for error type prediction among hit edits.
• MAE — range: other
  • Mean Absolute Error between predicted and reference severity levels.
• BLEU — range: [0, 1]
  • Standard n-gram based text similarity score for explanation generation.
• METEOR — range: [0, 1]
  • Semantic similarity metric combining precision, recall, and fragmentation penalty for explanation text.
• ROUGE-1/2/L — range: [0, 1]
  • Recall-oriented overlap scores for unigrams, bigrams, and longest common subsequence in explanation text.

Input / output format

Input: Source Chinese sentence containing grammatical errors.

Output: Corrected Chinese sentence followed by a structured explanation containing error type, severity level, natural-language description, evidence words, linguistic rules, and revision advice.

Scoring recipe

def score(predictions, gold):
    hyp_edits = align_edits(predictions.corrected, predictions.source)
    ref_edits = align_edits(gold.corrected, gold.source)
    hit_edits = [e for e in hyp_edits if overlaps(e, ref_edits)]
    miss_edits = [e for e in ref_edits if not overlaps(e, hyp_edits)]
    
    corr_f05 = compute_f05(hyp_edits, ref_edits)  # via CLEME/ChERRANT
    hit_rate = len(hit_edits) / max(len(hyp_edits), 1)
    miss_rate = len(miss_edits) / max(len(ref_edits), 1)
    acc = accuracy(hit_edits.pred_type, hit_edits.ref_type)
    f1 = f1_score(hit_edits.pred_type, hit_edits.ref_type)
    mae = mean_absolute_error(hit_edits.pred_severity, hit_edits.ref_severity)
    text_scores = compute_text_metrics(predictions.explanation, gold.explanation, ['bleu', 'meteor', 'rouge'])
    return {'CLEME_F0.5': corr_f05, 'Hit': hit_rate, 'Miss': miss_rate, 'Acc': acc, 'F1': f1, 'MAE': mae, **text_scores}

Common pitfalls

• Evaluating explanation metrics on all hypothesis edits instead of restricting them to 'hit edits' (edits overlapping with references), which artificially deflates scores and violates the protocol.
• Confusing 'post-explaining' (correct-then-explain) with 'pre-explaining' (explain-then-correct) settings, as the latter suffers from error propagation and yields significantly lower scores across all metrics.
• Using standard ROUGE/METEOR without accounting for the edit-wise alignment constraint, leading to misleading correlation with human judgments for free-text explanations.

Evidence (verbatim from paper)

We obtain the metric results using public toolkits including ROUGE (Lin [2004]), NLTK (Bird and Loper [2004]), and scikit-learn (Pedregosa et al. [2011]). Particularly, we observe many hypothesis edits are not covered by the corresponding reference edits, making it impossible to subsequently evaluate the explanations for these edits. To address this, we introduce two extra indicators, namely Hit and Miss rates. A hypothesis edit overlapping with a reference edit is designated as a hit edit, while a reference edit without any match with hypothesis edits is deemed a miss edit. The hit rate is defined as the ratio of hit edits to all hypothesis edits, and the miss rate as the ratio of miss edits to all reference edits. Only the hit edits are used to calculate the evaluation outcomes for explanations.

Citation

@misc{ye2024excgex,
  title={EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correction},
  author={Ye et al. (2024)},
  year={2024},
  note={arXiv:2407.00924}
}

• arXiv: 2407.00924