Care Eval

care-eval

CARE: Extracting Experimental Findings From Clinical Literature — Naik et al. (2023) (arXiv:2311.09736, 2023)

What this evaluates

Evaluates information extraction systems on clinical literature for fine-grained extraction of experimental findings, including entities, attributes, and complex n-ary relations with discontinuous spans and variable arity.

Datasets

• CARE — total 700; splits: train (-1), test (-1); repo https://github.com/aakanksha19/clinical-findings-extraction

Metrics

• entity-level F1 — range: [0, 1]
  • Standard entity-level precision, recall, and F1 score computed over extracted entity spans.
• attribute-level F1 — range: [0, 1]
  • Standard entity-level precision, recall, and F1 score computed over extracted attribute spans.
• relaxed overlap F1 (primary) — range: [0, 1]
  • Aligns predicted relations with gold relations by highest overlap. Computes partial match score as #shared_entities/total_entities, then calculates F1 using this score instead of binary 0/1. Reported in both typed (entity types must match) and untyped (types ignored) settings.

Input / output format

Input: Clinical abstract text. For generative models, input may include gold entity/attribute markers () or few-shot examples selected by SPECTER v2.0 similarity.

Output: JSON format adhering to the CARE schema for entities, attributes, and relations, or span/label predictions for extractive models.

Scoring recipe

def compute_relaxed_overlap_f1(pred_rels, gold_rels):
    prec_scores, rec_scores = [], []
    for pred in pred_rels:
        best_overlap = 0
        for gold in gold_rels:
            shared = len(set(pred.entities) & set(gold.entities))
            total = len(gold.entities)
            overlap = shared / total if total > 0 else 0
            best_overlap = max(best_overlap, overlap)
        prec_scores.append(best_overlap)
    # Recall computed similarly by swapping pred/gold roles
    # F1 = 2 * P * R / (P + R)

Common pitfalls

• Many extractive models cannot handle discontinuous spans, requiring strict or no-discontinuity F1 variants for fair comparison.
• Evaluating relation extraction in isolation requires ignoring entity type mismatches (untyped F1), as mistyping entities should not penalize relation structure prediction.
• N-ary relation extraction with variable arity causes combinatorial explosion, making some baseline systems infeasible on this dataset.

Evidence (verbatim from paper)

Model performance on entity and attribute extraction is evaluated using entity-level F1. Relation extraction performance is evaluated using a relaxed overlap F1 score inspired by Tiktinsky et al. (2022), which assigns partial credit to correctly identified subsets of entities in a relation, even if all identified entities do not match. As with agreement score calculation, predicted relations are first aligned with gold relations by choosing the gold relation with highest overlap per predicted relation. Then a partial match score is computed as #shared_entities/total_entities and used in the F1 computation instead of binary 0/1 score.

Citation

@misc{naik2023care,
  title={CARE: Extracting Experimental Findings From Clinical Literature},
  author={Naik et al. (2023)},
  year={2023},
  note={arXiv:2311.09736}
}

• arXiv: 2311.09736