Relaxed Perplexity

relaxed-perplexity

Automatic Evaluation of Healthcare LLMs Beyond Question-Answering — Arias-Duart et al. (2025) (arXiv:2502.06666, 2025)

What this evaluates

This protocol evaluates the reliability, consistency, and inter-correlation of various open-ended and close-ended evaluation metrics on healthcare LLM outputs. It specifically probes how well metrics capture factual coherence and content quality while being robust to output rephrasing and sampling variations.

Datasets

• CareQA-Open — total ?; splits: test (-1)

Metrics

• Relaxed Perplexity (primary) — range: other
  • A novel metric for open-ended responses designed to capture factual coherence without relying on brittle n-gram overlaps or strict probabilistic perplexity measures.
• Perplexity — range: other
  • Probabilistic prediction-based metric encompassing Word Perplexity, Bits per Byte, and Byte Perplexity to measure information efficiency.
• BERTScore — range: [0, 1]
  • Semantic similarity metric that evaluates content and overall text quality by comparing contextual embeddings of predictions and gold text.
• ROUGE — range: [0, 1]
  • N-gram-based metric relying on surface-level word matching (ROUGE1, ROUGE2, ROUGEL) to measure overlap between generated and reference text.
• Prometheus — range: [0, 1]
  • An LLM judge metric that scores model outputs based on correctness, with high sensitivity to punctuation and formatting.

Input / output format

Input: Medical prompts requiring open-ended generation (e.g., clinical note-taking, summarization, question entailment) or multiple-choice questions (MCQA).

Output: Model-generated text responses for open-ended tasks, or selected answer options for MCQA.

Scoring recipe

def compute_metric(predictions, golds, metric_name):
    if metric_name == 'relaxed_perplexity':
        return relaxed_perplexity_score(predictions, golds)
    elif metric_name in ['rouge1', 'rouge2', 'rougel', 'bleu']:
        return ngram_overlap(predictions, golds)
    elif metric_name in ['bertscore', 'bleurt', 'moverscore']:
        return semantic_similarity(predictions, golds)
    elif metric_name == 'perplexity':
        return probabilistic_prediction_score(predictions, golds)
    elif metric_name == 'prometheus':
        return llm_judge_score(predictions, golds)
    elif metric_name == 'mcqa':
        return exact_match_accuracy(predictions, golds)

Common pitfalls

• Metrics like Prometheus and semantic similarity (BERTScore, BLEURT) are highly sensitive to rephrasing and formatting, leading to unreliable scores if outputs are slightly altered.
• Sampling strategies (top_p, top_k) cause significant variance in metric scores across runs, reducing self-consistency for non-perplexity metrics.
• Assuming high correlation between different task types (e.g., MCQA vs. summarization) is misleading, as they measure distinct capabilities and often show weak or negative correlation.

Evidence (verbatim from paper)

To assess whether the metrics used in the open evaluation are correlated among themselves, and to simplify future analyses for practitioners, we conduct a correlation analysis for each of the metrics detailed in §[2.2] across all implemented open-ended benchmarks... Results show that most n-gram-based metrics (i.e., ROUGE1, ROUGE2, ROUGEL and BLEU) are resilient to rephrasing... In contrast, Prometheus (LLM judge) is the most affected by rewording, which is reasonable considering that, for this evaluation, correct punctuation and formatting in the answers greatly improve scores.

Citation

@misc{ariasduart2025automatic,
  title={Automatic Evaluation of Healthcare LLMs Beyond Question-Answering},
  author={Arias-Duart et al. (2025)},
  year={2025},
  note={arXiv:2502.06666}
}

• arXiv: 2502.06666