Gptscore Eval

gptscore-eval

GPTScore: Evaluate as You Desire — Jinlan Fu et al. (2023) (arXiv:2302.04166, 2023)

What this evaluates

Evaluates the correlation between automated scoring functions (GPTScore variants) and human judgments across multiple text generation tasks. It probes the ability of instruction-based LLMs to serve as training-free, customizable evaluators that align with human preference.

Datasets

• SummEval — total ?; splits: test (-1)
• RealSumm — total ?; splits: test (-1)
• NEWSROOM — total ?; splits: test (-1)
• QXSUM — total ?; splits: test (-1)
• MQM-2020 — total ?; splits: test (-1)
• BAGEL — total ?; splits: test (-1)
• SFRES — total ?; splits: test (-1)
• FED — total ?; splits: test (-1)

Metrics

• Spearman correlation (primary) — range: [-1, 1]
  • Standard rank correlation coefficient measuring the monotonic relationship between the model's predicted scores (conditional generation probabilities) and human judgment scores. Computed as ρ = 1 - (6Σd_i²)/(n(n²-1)) for tied ranks, or via standard library rankdata.

Input / output format

Input: Source text, reference text, and generated hypothesis (or system output).

Output: A continuous score representing the conditional generation probability of the reference given the source and hypothesis, or vice versa, depending on the prompt configuration.

Scoring recipe

def compute_spearman(pred_scores, human_scores):
    # pred_scores: list of GPTScore probabilities for each instance
    # human_scores: list of human judgment scores for each instance
    rank_pred = rankdata(pred_scores)
    rank_human = rankdata(human_scores)
    n = len(rank_pred)
    d_sq = sum((r1 - r2)**2 for r1, r2 in zip(rank_pred, rank_human))
    rho = 1 - (6 * d_sq) / (n * (n**2 - 1))
    return rho

Common pitfalls

• GPTScore outputs conditional probabilities, not direct human-like scores; evaluation requires computing correlation against human judgments rather than direct accuracy.
• The paper reports correlation coefficients, so lower scores indicate worse alignment with human preference, not worse generation quality.
• Instruction and demonstration settings (VAL, IST, IDM) significantly impact results; vanilla prompts often underperform and should not be used for final reporting.

Evidence (verbatim from paper)

Fig. 3 shows the evaluation results of five GPT3 variant models on four text summarization datasets, where QXSUM uses the Pearson correlation and other datasets use the Spearman correlation metric.

Citation

@misc{fu2023gptscore,
  title={GPTScore: Evaluate as You Desire},
  author={Jinlan Fu et al. (2023)},
  year={2023},
  note={arXiv:2302.04166}
}

• arXiv: 2302.04166