Nli4pr Eval

nli4pr-eval

Am I eligible? Natural Language Inference for Clinical Trial Patient Recruitment: the Patient's Point of View — Aguiar et al. (2025) (arXiv:2503.15718, 2025)

What this evaluates

Evaluates whether large language models can correctly determine clinical trial eligibility by performing natural language inference between patient profiles and trial criteria. It probes the model's ability to handle imprecise layman medical terminology compared to precise clinical language in a zero-shot setting.

Datasets

• NLI4PR — total ?; splits: test (-1); repo https://github.com/CTInfer/NLI4PR

Metrics

• Macro F1 (primary) — range: percent
  • Macro-averaged F1 score computed across all NLI classes. Calculated as the unweighted mean of the F1 scores for each class (Entailment, Contradiction, Neutral).

Input / output format

Input: A prompt containing a patient profile (written in either lay/patient language or medical language) and clinical trial eligibility criteria, formatted using either a vanilla or persona template.

Output: A single NLI classification label indicating the relationship between the patient profile and the eligibility criteria (e.g., Entailment, Contradiction, Neutral).

Scoring recipe

def compute_macro_f1(predictions, gold_labels):
    classes = ['Entailment', 'Contradiction', 'Neutral']
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores) * 100

Common pitfalls

• Layman terms often lack the precision of medical terminology, making it difficult for models to match vague patient descriptions to strict eligibility criteria.
• Using a persona template does not consistently improve performance and can sometimes degrade it, particularly for smaller models like Flan-T5.
• The majority baseline assumes 'Entailment', which may skew expectations if the dataset is imbalanced or if the task is treated as binary rather than ternary.

Evidence (verbatim from paper)

Table 4 displays the results obtained by the models on the two types of templates. Table 4: Macro F1 score (in %) for the different baselines, using our different prompting templates in a zero-shot setting, on the test set. Lay is patient language, Med is medical doctor’s language, V stands for vanilla template and P stands for persona template. The majority baseline is Entailment.

Citation

@misc{aguiar2025nli4pr,
  title={Am I eligible? Natural Language Inference for Clinical Trial Patient Recruitment: the Patient's Point of View},
  author={Aguiar et al. (2025)},
  year={2025},
  note={arXiv:2503.15718}
}

• arXiv: 2503.15718