Corrected P Value

corrected_p_value

The Autocorrelation Blind Spot: Why 42% of Turn-Level Findings in LLM Conversation Analysis May Be Spurious — Schessl (2026) (arXiv:2604.14414, 2026)

What this evaluates

Evaluates whether turn-level conversational metrics in LLM interactions suffer from temporal autocorrelation that inflates statistical significance. It compares naive pooled hypothesis testing against cluster-robust corrections to measure false positive rates and classify metric robustness.

Datasets

• LLM Conversation Turn-Level Dataset — total ?; splits: train (-1), test (-1); repo https://github.com/ferdinandschessl-boop/autocorrelation-correction

Metrics

• pooled_point_biserial_correlation — range: [-1, 1]
  • Pearson correlation coefficient between the turn-level metric values and the target label, computed naively across all turns assuming independence.
• corrected_p_value (primary) — range: [0, 1]
  • The maximum of the Chelton-corrected p-value (adjusted for effective degrees of freedom) and the block bootstrap p-value. Used to determine statistical significance after accounting for temporal autocorrelation.
• robustness_status — range: categorical
  • Binary classification derived from the corrected p-value: 'robust' if corrected_p_value < 0.05, otherwise 'weak'.

Input / output format

Input: Turn-level feature values (e.g., embedding velocity, differential, compression-based scores, interaction terms) paired with target labels across sequential conversation turns.

Output: Per metric-label pair: observed correlation, pooled p-value, mean lag-1 autocorrelation, effective degrees of freedom, corrected p-value, and a robustness classification (robust/weak).

Scoring recipe

for each metric_label_pair:
  r_obs = pearson_correlation(metric_values, labels)
  p_pooled = t_test_pvalue(r_obs, n_turns)
  rho1 = autocorrelation(metric_values, lag=1)
  n_eff = chelton_effective_dof(n_turns, rho1)
  p_chelton = t_test_pvalue(r_obs, n_eff)
  p_boot = block_bootstrap_pvalue(metric_values, labels, block_size=optimal)
  p_final = max(p_chelton, p_boot)
  status = 'robust' if p_final < 0.05 else 'weak'
  return {r_obs, p_pooled, rho1, n_eff, p_final, status}

Common pitfalls

• Ignoring temporal dependence in turn-level data inflates effective sample size and causes false positives.
• Naive pooled t-tests assume independence, violating the sequential structure of conversation data.
• Non-memoryless metrics (e.g., cumulative or rolling windows) exhibit high lag-1 autocorrelation, making them highly susceptible to inflation compared to memoryless metrics.

Evidence (verbatim from paper)

Table 6 presents all 81 metric–label pairs that are significant under naive pooled testing across all labels, along with their autocorrelation profiles and correction outcomes. $r_{\text{obs}}$: pooled point-biserial correlation. $p_{\text{final}}=\max(p_{\text{Chelton}},p_{\text{boot}})$. Status: robust if $p_{\text{final}}<0.05$, weak otherwise.

Citation

@misc{schessl2026autocorrelation,
  title={The Autocorrelation Blind Spot: Why 42% of Turn-Level Findings in LLM Conversation Analysis May Be Spurious},
  author={Schessl (2026)},
  year={2026},
  note={arXiv:2604.14414}
}

• arXiv: 2604.14414