Microbiorel Re Eval

microbiorel-re-eval

Summarization for Generative Relation Extraction in the Microbiome Domain — El Khettari et al. (2025) (arXiv:2506.08647, 2025)

What this evaluates

Evaluates generative and discriminative models on document-level relation extraction in the microbiome domain. It probes the ability to correctly classify pairwise relations between biomedical entities (species, diseases, chemicals, etc.) under a low-resource setting.

Datasets

• MicrobioRel — total 1994; splits: (unstated); repo https://github.com/Stan8/MicrobioRel-dataset

Metrics

• Weighted F1-score (primary) — range: [0, 1]
  • Weighted average of per-class F1 scores, where weights are proportional to class support (number of instances). Computed as Σ(support_i / total_support × F1_i). Only exact matches with gold labels count as correct predictions.

Input / output format

Input: Paragraphs containing at least two named entities (species, diseases, chemicals, mutations, genes, cell lines). The model receives the text context and entity pairs to predict the relation type.

Output: A single relation class label from the predefined schema (or 'None'). For the summarization step, outputs follow the format 'Entity1 Relation Entity2' in a single sentence.

Scoring recipe

def compute_weighted_f1_exact_match(preds, golds, classes):
    tp = {c: 0 for c in classes}
    fp = {c: 0 for c in classes}
    fn = {c: 0 for c in classes}
    support = {c: 0 for c in classes}
    for p, g in zip(preds, golds):
        support[g] += 1
        if p == g:
            tp[g] += 1
        else:
            fp[p] += 1
            fn[g] += 1
    total = sum(support.values())
    weighted_f1 = 0.0
    for c in classes:
        if support[c] > 0:
            prec = tp[c] / (tp[c] + fp[c]) if (tp[c] + fp[c]) > 0 else 0
            rec = tp[c] / (tp[c] + fn[c]) if (tp[c] + fn[c]) > 0 else 0
            f1_c = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
            weighted_f1 += (support[c] / total) * f1_c
    return weighted_f1

Common pitfalls

• LLM outputs often contain extra text; post-processing is required to extract only the predicted relation class before scoring.
• Partial matches or substring extractions do not count; predictions must exactly match the gold relation label.
• Weighted F1 is calculated by averaging per-class F1 scores, not by computing weighted precision and recall first.

Evidence (verbatim from paper)

Evaluation of the RE task is carried out using weighted precision, recall, and F1-score. Under an exact match criterion, a prediction is considered correct only if it matches the gold relation label. ... In the weighted setting, the precision, recall, and F1-score are computed as the weighted averages of the corresponding per-class scores, where the weights are based on the support of each class (see Equation 1). Note that weighted F1 is derived from individual per-class F1 scores, not from weighted precision and recall. Metrics are computed after post-processing the LLM outputs to remove text beyond the predicted relation class.

Citation

@misc{elkhettari2025summarization,
  title={Summarization for Generative Relation Extraction in the Microbiome Domain},
  author={El Khettari et al. (2025)},
  year={2025},
  note={arXiv:2506.08647}
}

• arXiv: 2506.08647