Peerprism Eval

peerprism-eval

PeerPrism: Peer Evaluation Expertise vs Review-writing AI — Sadeghian et al. (2026) (arXiv:2604.14513, 2026)

What this evaluates

Evaluates the ability of various LLM text detection methods to distinguish between human-written and AI-generated peer reviews, while also assessing their robustness to hybrid (human idea + AI text) workflows.

Datasets

• PeerPrism — total 20690; splits: test (-1); repo https://github.com/Reviewerly-Inc/PeerPrism

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard binary classification accuracy: the proportion of correctly classified reviews (Human vs. AI) out of the total evaluated instances.

Input / output format

Input: Peer review texts labeled as original human-written, fully synthetic LLM-generated, or hybrid/transformed.

Output: Binary classification label (Human or AI) or probability score, depending on the detector.

Scoring recipe

# For each detector:
scores = detector.predict(reviews)
if scores are probabilities:
    threshold = calibrate_threshold(scores, held_out_subset)
    preds = [1 if s >= threshold else 0 for s in scores]
else:
    preds = scores
accuracy = sum(pred == gold for pred, gold in zip(preds, gold_labels)) / len(gold_labels)

Common pitfalls

• Hybrid reviews (human ideas + AI text) are explicitly excluded from threshold calibration and standard accuracy computation, but used separately for robustness checks.
• Detectors are evaluated in their original off-the-shelf configurations without fine-tuning on the PeerPrism dataset.
• Score-based methods require threshold calibration on a balanced held-out subset before evaluation, as they lack predefined thresholds.

Evidence (verbatim from paper)

For binary evaluation, we define two strict ground-truth classes: original human-written reviews (Human) and fully synthetic LLM-generated reviews (AI). Hybrid regimes are excluded from threshold calibration and standard accuracy computation and are instead used to assess robustness under mixed-provenance conditions. For detectors that output probabilities, we follow the thresholds recommended in their original implementations. For score-based methods without prescribed thresholds (Anchor and Lastde++), we calibrate the decision boundary on a balanced held-out subset and fix it for all subsequent experiments. We report accuracy and confusion matrices separately for each provenance regime.

Citation

@misc{sadeghian2026peerprism,
  title={PeerPrism: Peer Evaluation Expertise vs Review-writing AI},
  author={Sadeghian et al. (2026)},
  year={2026},
  note={arXiv:2604.14513}
}

• arXiv: 2604.14513