Counterfactual Rhetorical Score

counterfactual-rhetorical-score

Counterfactual LLM-based Framework for Measuring Rhetorical Style — Qiu et al. (2025) (arXiv:2512.19908, 2025)

What this evaluates

This metric quantifies the promotional or visionary framing of a machine learning paper's abstract, isolating rhetorical style from the underlying technical content. It uses counterfactual abstracts generated by diverse LLM personas and pairwise comparisons to produce a calibrated continuous score of rhetorical strength.

Datasets

• ICLR Submissions (2017-2025) — total 8485; splits: test (8485)

Metrics

• Rhetorical Score (primary) — range: other
  • A continuous score derived from a Bradley-Terry model aggregating pairwise comparisons between the original abstract and 30 counterfactual abstracts generated by distinct LLM personas. Higher values indicate stronger promotional/visionary framing.

Input / output format

Input: Original abstract text (Y) and extracted methods/results sections (X) for each paper.

Output: A single continuous floating-point score representing the paper's rhetorical strength.

Scoring recipe

# 1. Extract substantive content X and abstract Y for each paper.
# 2. Generate 30 counterfactual abstracts Y_i using distinct LLM personas (length ±15 words of Y).
# 3. For each Y_i, prompt an LLM judge to compare Y vs Y_i and select which makes stronger claims.
# 4. Aggregate all pairwise comparison outcomes (original vs personas) using a Bradley-Terry model to estimate the latent rhetorical strength score for the original abstract.

Common pitfalls

• Conflating rhetorical style with substantive technical content, which biases direct LLM rating baselines.
• Assuming higher rhetorical scores correlate with better peer-review ratings; the paper finds a null correlation with reviewer scores.
• Interpreting the Bradley-Terry score as an absolute quality metric rather than a relative framing intensity measure.

Evidence (verbatim from paper)

Table 1 shows that higher rhetorical scores significantly predict both citations and media attention. For example, a one-unit increase in the rhetorical score is on average associated with 24 additional citations, 3 more media posts, and 2 more tweets.

Citation

@misc{qiu2025counterfactual,
  title={Counterfactual LLM-based Framework for Measuring Rhetorical Style},
  author={Qiu et al. (2025)},
  year={2025},
  note={arXiv:2512.19908}
}

• arXiv: 2512.19908