Diskn Eval

diskn-eval

Probing Pre-Trained Language Models for Disease Knowledge — Israa Alghanmi et al. (2021) (arXiv:2106.07285, 2021)

What this evaluates

Evaluates pre-trained language models' ability to reason about diseases by mapping symptoms, treatments, tests, procedures, and terminology to disease names. It isolates medical reasoning types and uses adversarial negative examples to prevent knowledge leakage.

Datasets

• DisKnE — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/israa-alghanmi/DisKnE

Metrics

• F1 score (primary) — range: percent
  • Macro and weighted average of F1 score for the positive class (entailment), averaged over three runs with different random seeds.

Input / output format

Input: Sentence pairs (premise and hypothesis) derived from MedNLI, where hypotheses are either canonicalized to disease names or kept as full text, used for binary entailment classification.

Output: Binary prediction (entailment or negative) for each sentence pair.

Scoring recipe

def compute_f1(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

macro_f1 = mean(compute_f1(model_preds, gold) for disease in diseases)
weighted_f1 = weighted_mean(compute_f1(model_preds, gold), weights=[num_examples_per_disease])

Common pitfalls

• Hypothesis-only baselines can achieve high scores if hypotheses are not canonicalized, exploiting lexical overlap rather than reasoning.
• Performance varies drastically across diseases; macro and weighted averages can tell conflicting stories about model capability.
• Disease-specific train-test splits are enforced to prevent knowledge leakage, so results are not directly comparable to standard NLI benchmarks.

Evidence (verbatim from paper)

We analyze the results per disease and per category in terms of F1 score for the positive class, reporting results for all diseases that have at least two positive examples for the considered category.

Citation

@misc{alghanmi2021probing,
  title={Probing Pre-Trained Language Models for Disease Knowledge},
  author={Israa Alghanmi et al. (2021)},
  year={2021},
  note={arXiv:2106.07285}
}

• arXiv: 2106.07285