Paired T Test Evaluation

paired-t-test-evaluation

Eliciting Informative Text Evaluations with Large Language Models — Lu et al. (2024) (arXiv:2405.15077, 2024)

What this evaluates

Tests whether a peer prediction mechanism's scoring function is sensitive to report quality by verifying that replacing high-quality reports with degraded or LLM-generated low-quality reports leads to a statistically significant decrease in expected scores.

Datasets

• ICLR2020 OpenReview — total ?; splits: evaluation (-1); repo https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models

Metrics

• paired difference t-test (p-value) (primary) — range: [0, 1]
  • Computes the mean difference d_bar between original scores s_+ and degraded scores s_-. The t-statistic is t = d_bar / (sigma_d / sqrt(K)), where sigma_d is the standard deviation of differences and K is the number of pairs. The p-value is p = Pr[X > t] for X ~ t_{K-1}. A p-value < 0.05 rejects the null hypothesis that the mean difference is zero.

Input / output format

Input: A dataset containing items with associated text reports, a peer prediction mechanism M that takes two reports as input, and a degradation process D that generates low-quality reports.

Output: Two lists of scores s_+ and s_- of length K, followed by the computed t-statistic and p-value for the paired differences.

Scoring recipe

s_plus = []
s_minus = []
for k in range(K):
    z = draw_random_item(dataset)
    x_i, x_j = draw_random_reports(z)
    s_plus.append(M(x_i, x_j))
    x_i_l = D(x_i)
    s_minus.append(M(x_i_l, x_j))
d = [s_plus[k] - s_minus[k] for k in range(K)]
d_bar = mean(d)
sigma_d = std(d, ddof=1)
t_stat = d_bar / (sigma_d / sqrt(K))
p_value = 1 - t_cdf(t_stat, df=K-1)
return t_stat, p_value

Common pitfalls

• Using an independent t-test instead of a paired t-test, which ignores the matched s_+ and s_- pairs and reduces statistical power.
• Assuming LLM-generated reviews are direct substitutes for the original signal; they depend on the item z and test quality differentiation rather than untruthful reporting.
• Failing to verify that the difference distribution is approximately normal before applying the t-test, as the paper explicitly notes this assumption.

Evidence (verbatim from paper)

Specifically, We employ the t-test, which is able to identify whether there is a significant decrease in scores from s_+ to s_-. ... The test statistic for the Paired Difference t-test is calculated as follows: t = d_bar / (sigma_d / sqrt(K)) ... The p-value is obtained by p = Pr[X > t], where X is a random variable following the t-distribution with the calculated degrees of freedom K-1, and t is the calculated t-statistic above. Typically, a p-value threshold of 0.05 is used, where a p-value lower than 0.05 leads to the rejection of the null hypothesis, indicating that there’s a statistically significant difference.

Citation

@misc{lu2024eliciting,
  title={Eliciting Informative Text Evaluations with Large Language Models},
  author={Lu et al. (2024)},
  year={2024},
  note={arXiv:2405.15077}
}

• arXiv: 2405.15077