Spearman Correlation

spearman-correlation

G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment — Liu et al. (2023) (arXiv:2303.16634, 2023)

What this evaluates

Measures the rank-order agreement between an NLG evaluator's predicted scores and human reference judgments. It probes the evaluator's ability to capture task-specific quality dimensions such as fluency, coherence, consistency, and groundedness across summarization and dialogue generation.

Datasets

• SummEval — total ?; splits: test (-1)
• Topical-Chat — total ?; splits: test (-1)
• QAGS — total ?; splits: test (-1)

Metrics

• Spearman correlation (primary) — range: [-1, 1]
  • Non-parametric measure of rank correlation between the predicted scores from the NLG evaluator and the human reference scores. Calculated on the ranks of the two score vectors. Ranges from -1 to 1, where 1 indicates perfect agreement in ranking.

Input / output format

Input: Source document/context, generated summary or dialogue response, and a set of evaluation criteria (e.g., fluency, coherence, consistency, relevance, naturalness, engagingness, groundedness).

Output: A numerical score (typically on a 1–5 scale) for each specified criterion, generated via a chain-of-thought form-filling prompt.

Scoring recipe

def compute_spearman(predictions, gold):
    # predictions: list of evaluator scores per instance
    # gold: list of human scores per instance
    from scipy.stats import spearmanr
    corr, _ = spearmanr(predictions, gold)
    return corr

Common pitfalls

• GPT-4 lacks native token probability output; the paper estimates probabilities by sampling n=20 times with temperature=1, introducing sampling variance.
• LLM-based evaluators exhibit a strong bias toward LLM-generated text over human-written content, which can artificially inflate scores in self-reinforcement training loops.
• Correlation is computed separately for each evaluation aspect (e.g., coherence, groundedness) rather than as a single aggregate metric.

Evidence (verbatim from paper)

We adopt the same approach as Zhong et al. (2022) to evaluate different summarization metrics using summary-level Spearman and Kendall-Tau correlation.

Citation

@misc{liu2023geval,
  title={G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2303.16634}
}

• arXiv: 2303.16634