Dti Relation Extraction Eval

dti-relation-extraction-eval

R-BERT-CNN: Drug-target interactions extraction from biomedical literature — Aldahdooh et al. (2021) (arXiv:2111.00611, 2021)

What this evaluates

Evaluates a model's ability to classify drug-target interaction relations from biomedical text into one of ten specific interaction types. It probes multiclass relation extraction under conditions of severe class imbalance.

Datasets

• DrugProt — total ?; splits: test (-1)
• ChemProt — total ?; splits: test (-1)

Metrics

• micro F1-score (primary) — range: [0, 1]
  • Aggregates true positives, false positives, and false negatives across all classes globally. Computed as 2 × (micro_precision × micro_recall) / (micro_precision + micro_recall), where micro_precision = ΣTP / (ΣTP + ΣFP) and micro_recall = ΣTP / (ΣTP + ΣFN).

Input / output format

Input: Biomedical text (PubMed abstracts) containing drug and target mentions.

Output: Single class label from 10 predefined interaction types: ANTAGONIST, INHIBITOR, AGONIST, ACTIVATOR, INDIRECT-UPREGULATOR, INDIRECT-DOWNREGULATOR, PART-OF, DIRECT-REGULATOR, SUBSTRATE, PRODUCT-OF.

Scoring recipe

tp = sum(1 for p, g in zip(preds, gold) if p == g)
fp = sum(1 for p, g in zip(preds, gold) if p != g)
fn = sum(1 for p, g in zip(preds, gold) if p != g)
micro_p = tp / (tp + fp)
micro_r = tp / (tp + fn)
micro_f1 = 2 * micro_p * micro_r / (micro_p + micro_r)

Common pitfalls

• Severe class imbalance causes models to ignore minority classes, inflating micro-averaged scores while failing on rare relations.
• The paper explicitly skips the three least represented classes during training to mitigate imbalance, which deviates from standard full-dataset evaluation protocols.
• Micro-averaging masks per-class performance differences, making it difficult to assess model robustness on underrepresented interaction types.

Evidence (verbatim from paper)

The micro F1-score is used to assess the quality of multiclassification problems by aggregating the contribution of each class. The micro F1, micro recall and micro precision scores are defined using the following equations. ... The result for the model I on the DrugProt test dataset gives 6.3% F1 micro score. ... R-BERT-CNN model achieved better results over both DrugProt and ChemProt test datasets (Table 3) ... we achieved the micro F1 score of 0.56 as shown in Table 3.

Citation

@misc{aldahdooh2021rbertcnn,
  title={R-BERT-CNN: Drug-target interactions extraction from biomedical literature},
  author={Aldahdooh et al. (2021)},
  year={2021},
  note={arXiv:2111.00611}
}

• arXiv: 2111.00611