Detecting LLM Peer Reviews Eval

detecting-llm-peer-reviews-eval

Detecting LLM-Generated Peer Reviews — Rao et al. (2025) (arXiv:2503.15772, 2025)

What this evaluates

Evaluates the efficacy of covert watermarking techniques embedded in manuscript PDFs to force LLM-generated peer reviews to contain specific hidden markers. It also tests the robustness of these watermarks against common reviewer defenses like paraphrasing, detection prompts, and page cropping, as well as the performance of cryptic prompt injection via gradient-based optimization.

Datasets

• ICLR 2024 submissions — total 100; splits: test (100); repo https://github.com/Vishisht-rao/detecting-llm-written-reviews
• ICLR 2021 submissions — total 10022; splits: test (10022); repo https://github.com/Vishisht-rao/detecting-llm-written-reviews
• ICLR 2024 submissions (control) — total 28028; splits: test (28028); repo https://github.com/Vishisht-rao/detecting-llm-written-reviews
• NSF Grant Proposals — total 52; splits: test (52); repo https://github.com/Vishisht-rao/detecting-llm-written-reviews
• PRC 2022 abstracts — total 20; splits: test (20); repo https://github.com/Vishisht-rao/detecting-llm-written-reviews
• PeerRead papers — total 20; splits: test (20); repo https://github.com/Vishisht-rao/detecting-llm-written-reviews

Metrics

• Accuracy (primary) — range: [0, 1]
  • Fraction of generated reviews that contain the exact embedded watermark string. Reported with 95% confidence intervals via bootstrap resampling (10,000 replicates).
• High-probability success rate (HPSR) — range: [0, 1]
  • Proportion of papers/abstracts where the watermark appears in at least 8 out of 10 independently generated reviews. Provides a binary assessment per paper.
• Overall success rate (OSR) — range: [0, 1]
  • Proportion of all generated reviews across all papers that contain the watermark. Captures fine-grained efficacy at the individual review level.
• False Positive Rate (FPR) / True Positive Rate (TPR) — range: [0, 1]
  • FPR: fraction of unwatermarked control reviews flagged by the statistical test. TPR: fraction of watermarked reviews correctly flagged. Evaluated at varying threshold k/|W|.

Input / output format

Input: PDF manuscript with covert prompt injection (watermark) embedded via white text, foreign language, font manipulation, or cryptic tokens; or an LLM-generated review for paraphrasing/cropping defenses.

Output: Natural language peer review text generated by the LLM.

Scoring recipe

def compute_metrics(predictions, gold_watermarks, num_trials=10):
    # Accuracy: fraction of reviews containing the exact watermark
    accuracy = sum(1 for p in predictions if gold_watermarks in p) / len(predictions)
    
    # HPSR: fraction of papers where watermark appears in >=8/10 trials
    hpsr = sum(1 for p in predictions if p.count(gold_watermarks) >= 8) / len(predictions)
    
    # OSR: overall proportion of reviews containing watermark
    osr = sum(p.count(gold_watermarks) for p in predictions) / (len(predictions) * num_trials)
    
    # FPR/TPR based on statistical test against watermark set W
    fpr = sum(1 for p in predictions if test_statistic(p) > threshold) / len(negative_set)
    tpr = sum(1 for p in predictions if test_statistic(p) > threshold) / len(positive_set)
    return accuracy, hpsr, osr, fpr, tpr

Common pitfalls

• Watermark detection accuracy drops significantly when using foreign language injection due to translation issues, which may be mistaken for method failure rather than a known limitation.
• Control conditions show high baseline matches for 'Technical Term' watermarks due to common academic phrasing, requiring careful filtering to avoid false positives.
• HPSR requires generating 10 independent reviews per paper; failing to control for temperature or sampling variance can skew results.

Evidence (verbatim from paper)

We define the following quantitative metrics to evaluate the detection performance of the cryptic prompt injections: 1. High-probability success rate (HPSR): We measure the number of abstracts or papers for which the watermark is detected with high probability. Specifically, an abstract or a paper is considered successfully watermarked if the predefined watermark appears in at least 8 out of the 10 independently generated reviews. 2. Overall success rate (OSR): We calculate the proportion of generated reviews containing the watermark across all abstracts/papers and all review samples.

Citation

@misc{rao2025detectingllm,
  title={Detecting LLM-Generated Peer Reviews},
  author={Rao et al. (2025)},
  year={2025},
  note={arXiv:2503.15772}
}

• arXiv: 2503.15772