Clinical Text Robustness Eval

clinical-text-robustness-eval

Deep learning models are not robust against noise in clinical text — Moradi et al. (2021) (arXiv:2108.12242, 2021)

What this evaluates

Evaluates the robustness of clinical NLP models against real-world input noise across four standard tasks. It probes whether models maintain performance when text contains character- or word-level perturbations (e.g., misspellings, deletions, negations) that remain human-readable.

Datasets

• i2b2 — total ?; splits: test (-1)
• MedSTS — total ?; splits: test (-1)
• MedNLI — total ?; splits: test (-1)

Metrics

• evaluation scores (accuracy/F1) (primary) — range: percent
  • Task-specific accuracy or F1 score calculated as (correct predictions / total predictions) * 100 or standard F1 formula. Reported as percentage.

Input / output format

Input: Clinical text sentences or documents, optionally perturbed with character-level or word-level noise according to a specified Perturbation Per Sample (PPS) parameter.

Output: Task-specific predictions: NER entity tags, relation extraction labels, textual inference labels, or semantic similarity scores.

Scoring recipe

original_score = compute_metric(model.predict(original_text), gold)
perturbed_score = compute_metric(model.predict(perturbed_text), gold)
drop = original_score - perturbed_score
return perturbed_score, drop

Common pitfalls

• Only PPS=1 results are reported in the main table, though PPS up to 4 is tested.
• Human readability does not guarantee model robustness; models fail on noise that humans find understandable.
• Task sensitivity varies significantly: semantic similarity degrades more than relation extraction under identical noise.

Evidence (verbatim from paper)

Table 3 also presents the evaluation scores obtained by the neural language models on the perturbed samples from the clinical NLP datasets. In the experiments, a parameter named Perturbation Per Sample (PPS) controlled the maximum number of perturbations per sample. For brevity reasons, only the results for PPS=1 are reported in Table 3.

Citation

@misc{moradi2021clinicaltextperturbation,
  title={Deep learning models are not robust against noise in clinical text},
  author={Moradi et al. (2021)},
  year={2021},
  note={arXiv:2108.12242}
}

• arXiv: 2108.12242