Plo Abbreviation Detection Eval

plo-abbreviation-detection-eval

PLOD: An Abbreviation Detection Dataset for Scientific Documents — Zilio et al. (2022) (arXiv:2204.12061, 2022)

What this evaluates

Evaluates sequence labeling models on detecting abbreviations and extracting their corresponding long forms in scientific text. It probes domain-specific NER capabilities under challenges like context dependency, sub-abbreviations, and polysemy.

Datasets

• PLOD — total 7150008; splits: train (-1), val (-1), test (-1)
• SDU@AAAI-22 Shared Task — total ?; splits: test (-1)

Metrics

• F (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: F = 2 * (P * R) / (P + R). Computed separately for abbreviation (AB) and long-form (LF) entity types at the token/span level.

Input / output format

Input: Tokenized text sequences from scientific documents.

Output: I-O-B sequence labels per token: B-AB (Begin Abbreviation), B-LF (Begin Long Form), I-LF (Inside Long Form), O (Outside).

Scoring recipe

def compute_f(predictions, gold):
    # predictions and gold are lists of I-O-B labels per token
    # For each entity type (AB, LF):
    #   TP = count of correctly predicted spans
    #   FP = count of predicted spans not in gold
    #   FN = count of gold spans not predicted
    #   P = TP / (TP + FP)
    #   R = TP / (TP + FN)
    #   F = 2 * P * R / (P + R) if (P + R) > 0 else 0
    # Return macro-averaged F across AB and LF

Common pitfalls

• Models are evaluated on both filtered and unfiltered data, which may yield different performance and should be reported separately.
• The SDU@AAAI-22 test set combines train and dev splits due to unreleased official test data, potentially inflating evaluation scores.
• Abbreviation detection does not distinguish acronyms from other short forms, which may conflate task definitions across datasets.

Evidence (verbatim from paper)

We randomised and split our dataset into 70% instances for training, 15% for validation, and the remaining 15% as test data. ... This resulted in a one-token-per-line training file with the I-O-B annotation which amounted to 7,150,008 annotated tokens. ... The results obtained with the help of our models are presented in Tables 3 and 4. Table 4: Results of the fine-tuning-based abbreviation detection task where Filtered data was used for training and testing. The table also shows results where we used the same trained models, but tested them on the SDU Shared Task dataset. ... Abbreviations P R F ... Long-forms P R F

Citation

@misc{zilio2022plod,
  title={PLOD: An Abbreviation Detection Dataset for Scientific Documents},
  author={Zilio et al. (2022)},
  year={2022},
  note={arXiv:2204.12061}
}

• arXiv: 2204.12061