Lexical Simplification Eval

lexical-simplification-eval

Enhancing Pre-trained Language Model with Lexical Simplification — Bao et al. (2020) (arXiv:2012.15070, 2020)

What this evaluates

Evaluates the robustness and classification accuracy of pre-trained language models when augmented with rule-based lexical simplification as auxiliary inputs. It probes whether lemmatization and rare-word replacement preserve semantic meaning while mitigating lexical diversity effects on downstream NLU tasks.

Datasets

• SST-2 — total ?; splits: train (-1), test (-1)
• CR — total ?; splits: train (-1), test (-1)
• SUBJ — total ?; splits: train (-1), test (-1)
• MR — total ?; splits: train (-1), test (-1)
• AG — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted class labels out of the total number of instances: (TP + TN) / (TP + TN + FP + FN).
• F1 — range: [0, 1]
  • The harmonic mean of precision and recall, calculated as 2 * (precision * recall) / (precision + recall).

Input / output format

Input: Original text sequence paired with a rule-simplified version of the same text (lemmatized and rare words replaced) as auxiliary input.

Output: Discrete class label corresponding to the dataset's taxonomy (e.g., positive/negative sentiment or one of four news topics).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• The simplified auxiliary input must be processed alongside the original text, not as a replacement.
• Rare word replacement strictly preserves part-of-speech tags; ignoring this constraint alters the evaluation protocol.
• Performance gains are measured relative to specific baselines (BERT-base, ELECTRA-large) under identical training/inference conditions.

Evidence (verbatim from paper)

Experiments on SST-2, MR, CR, SUBJ, and AG show consistent performance gains over baseline models (e.g., +1.1% to +0.8% F1), outperforming synonym replacement, back-translation, and cutoff methods in both training and inference.

Citation

@misc{bao2020enhancing,
  title={Enhancing Pre-trained Language Model with Lexical Simplification},
  author={Bao et al. (2020)},
  year={2020},
  note={arXiv:2012.15070}
}

• arXiv: 2012.15070