Rct Numerical Extraction Eval

rct-numerical-extraction-eval

Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models — Yun et al. (2024) (arXiv:2405.01686, 2024)

What this evaluates

Evaluates LLMs' zero-shot capability to classify outcome types and extract precise numerical values from randomized controlled trial reports. It probes the models' numerical reasoning, information extraction robustness, and suitability for automating meta-analysis pipelines.

Datasets

• RCT Numerical Extraction Dataset — total ?; splits: test (-1), val (-1); repo https://github.com/hyesunyun/llm-meta-analysis

Metrics

• exact_match_accuracy (primary) — range: [0, 1]
  • Proportion of instances where all required fields (e.g., IE, IGS, CE, CGS for binary; IM, ISD, IGS, CM, CSD, CGS for continuous) exactly match the reference values. Calculated as correct_predictions / total_instances.
• mse_log_odds_ratio — range: other
  • Mean standardized error of the log odds ratios (for binary outcomes) or standardized mean differences (for continuous outcomes) computed using the extracted numerical data versus the reference data.

Input / output format

Input: Full-text XML of randomized controlled trial reports.

Output: Categorical label ('binary' or 'continuous') for outcome type; YAML format containing intervention/comparator events and group sizes (binary) or means, standard deviations, and group sizes (continuous).

Scoring recipe

def compute_exact_match(predictions, gold):
    correct = 0
    for pred, ref in zip(predictions, gold):
        if pred == ref:
            correct += 1
    return correct / len(gold)

def compute_mse(predictions, gold):
    # Extract numerical fields to compute effect sizes
    effect_sizes_pred = [calculate_effect_size(p) for p in predictions]
    effect_sizes_gold = [calculate_effect_size(g) for g in gold]
    # Standardize errors and take mean
    errors = [(e_p - e_g) / std_dev for e_p, e_g in zip(effect_sizes_pred, effect_sizes_gold)]
    return mean(errors)

Common pitfalls

• Models frequently fail to extract negative numbers when reported with double hyphens ('--') instead of a minus sign.
• LLMs often confuse medians with means, or standard errors with standard deviations.
• Failure to perform simple arithmetic to infer group sizes from total participant counts.
• Extracting values from the wrong timepoint or intervention/comparator group when multiple are reported.

Evidence (verbatim from paper)

GPT-4 has the highest overall exact match accuracy of 0.655, followed by GPT-3.5 with 0.298. Similar to the task of inferring outcome types, the open, smaller models vary substantially in terms of their match accuracies.

Citation

@misc{yun2024automaticallyextracting,
  title={Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models},
  author={Yun et al. (2024)},
  year={2024},
  note={arXiv:2405.01686}
}

• arXiv: 2405.01686