Plaba Track Eval

plaba-track-eval

Lessons from the TREC Plain Language Adaptation of Biomedical Abstracts (PLABA) track — Ondov et al. (2025) (arXiv:2507.14096, 2025)

What this evaluates

Evaluates NLP systems and large language models on adapting biomedical abstracts to plain language for lay consumers. It probes capabilities in text simplification, term replacement, factual faithfulness, and conciseness while measuring alignment with human expert judgments.

Datasets

• TREC PLABA — total ?; splits: test (-1)

Metrics

• SARI (primary) — range: [0, 1]
  • Measures simplification quality by balancing n-grams kept, inserted, and deleted against multiple reference simplifications. Uses the original Xu et al. (2016) implementation.
• F1 — range: [0, 1]
  • Union F1 score against the combined term identifications of two human annotators. For classification tasks, scores are macro-averaged across five simplification methods to handle class imbalance.
• Manual Final Score — range: [0, 100]
  • Linear interpolation of a 3-point Likert scale (-1, 0, 1) to a 0-100 range. The final score is the average of four axes: Simplicity, Accuracy, Completeness, and Brevity.

Input / output format

Input: A biomedical abstract paired with a consumer question.

Output: A plain language adaptation of the abstract, consisting of rewritten sentences or simplified terms.

Scoring recipe

def compute_sari(predictions, references, source):
    # Use original Xu et al. (2016) implementation
    return sari_score(predictions, references, source)

def compute_f1(predictions, gold_annotations):
    union_gold = set(gold_annotations[0]) | set(gold_annotations[1])
    tp = len(set(predictions) & union_gold)
    return 2 * tp / (2 * tp + len(predictions) - tp + len(union_gold) - tp)

def compute_manual_score(likert_values):
    # Interpolate -1, 0, 1 to 0-100
    interpolated = [(v + 1) * 50 for v in likert_values]
    return sum(interpolated) / len(interpolated)

Common pitfalls

• Using the Huggingface evaluate implementation of SARI instead of the original Xu et al. implementation, which yields significantly different scores.
• Relying solely on automatic metrics like SARI, BLEU, or ROUGE, as they show poor correlation with human judgments and manual evaluation is the gold standard.
• Confusing brevity with sentence simplicity, which previously caused low inter-annotator agreement.
• Failing to align document-level submissions to sentence-level references before evaluation.

Evidence (verbatim from paper)

As the primary metric for automatic, reference-based evaluation, we adopt SARI, a metric specifically designed to assess simplification by including the source and balancing n-grams kept, inserted, and deleted in the references*(Xu et al., [2016])*.

Citation

@misc{ondov2025lessons,
  title={Lessons from the TREC Plain Language Adaptation of Biomedical Abstracts (PLABA) track},
  author={Ondov et al. (2025)},
  year={2025},
  note={arXiv:2507.14096}
}

• arXiv: 2507.14096