# Paired T Test Evaluation

> Tests whether a peer prediction mechanism's scoring function is sensitive to report quality by verifying that replacing high-quality reports with degraded or LLM-generated low-quality reports leads to a statistically significant decrease in expected scores. Use when the user has predictions and gold and needs to compute paired difference t-test (p-value).

- Skill: `qhjqhj00/paired-t-test-evaluation` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/paired-t-test-evaluation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/paired-t-test-evaluation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/paired-t-test-evaluation

---


# paired-t-test-evaluation

> Eliciting Informative Text Evaluations with Large Language Models — Lu et al. (2024) (arXiv:2405.15077, 2024)

## What this evaluates

Tests whether a peer prediction mechanism's scoring function is sensitive to report quality by verifying that replacing high-quality reports with degraded or LLM-generated low-quality reports leads to a statistically significant decrease in expected scores.

## Datasets

- **ICLR2020 OpenReview** — total ?; splits: evaluation (-1); repo https://github.com/yx-lu/Eliciting-Informative-Text-Evaluations-with-Large-Language-Models

## Metrics

- `paired difference t-test (p-value)` **(primary)** — range: [0, 1]
  - Computes the mean difference d_bar between original scores s_+ and degraded scores s_-. The t-statistic is t = d_bar / (sigma_d / sqrt(K)), where sigma_d is the standard deviation of differences and K is the number of pairs. The p-value is p = Pr[X > t] for X ~ t_{K-1}. A p-value < 0.05 rejects the null hypothesis that the mean difference is zero.

## Input / output format

**Input**: A dataset containing items with associated text reports, a peer prediction mechanism M that takes two reports as input, and a degradation process D that generates low-quality reports.

**Output**: Two lists of scores s_+ and s_- of length K, followed by the computed t-statistic and p-value for the paired differences.

## Scoring recipe

```python
s_plus = []
s_minus = []
for k in range(K):
    z = draw_random_item(dataset)
    x_i, x_j = draw_random_reports(z)
    s_plus.append(M(x_i, x_j))
    x_i_l = D(x_i)
    s_minus.append(M(x_i_l, x_j))
d = [s_plus[k] - s_minus[k] for k in range(K)]
d_bar = mean(d)
sigma_d = std(d, ddof=1)
t_stat = d_bar / (sigma_d / sqrt(K))
p_value = 1 - t_cdf(t_stat, df=K-1)
return t_stat, p_value
```

## Common pitfalls

- Using an independent t-test instead of a paired t-test, which ignores the matched s_+ and s_- pairs and reduces statistical power.
- Assuming LLM-generated reviews are direct substitutes for the original signal; they depend on the item z and test quality differentiation rather than untruthful reporting.
- Failing to verify that the difference distribution is approximately normal before applying the t-test, as the paper explicitly notes this assumption.

## Evidence (verbatim from paper)

> Specifically, We employ the t-test, which is able to identify whether there is a significant decrease in scores from s_+ to s_-. ... The test statistic for the Paired Difference t-test is calculated as follows: t = d_bar / (sigma_d / sqrt(K)) ... The p-value is obtained by p = Pr[X > t], where X is a random variable following the t-distribution with the calculated degrees of freedom K-1, and t is the calculated t-statistic above. Typically, a p-value threshold of 0.05 is used, where a p-value lower than 0.05 leads to the rejection of the null hypothesis, indicating that there’s a statistically significant difference.

## Citation

```bibtex
@misc{lu2024eliciting,
  title={Eliciting Informative Text Evaluations with Large Language Models},
  author={Lu et al. (2024)},
  year={2024},
  note={arXiv:2405.15077}
}
```

- arXiv: 2405.15077