Publish And Perish Eval

publish-and-perish-eval

Publish and Perish: How AI-Accelerated Writing Without Proportional Verification Investment Degrades Scientific Knowledge — Kwon (2026) (arXiv:2604.05714, 2026)

What this evaluates

Evaluates a dynamical systems model of scientific publishing by simulating the interplay between AI-accelerated manuscript writing and peer review throughput. It measures how queue pressure drives AI adoption in review, degrades verification quality, and ultimately impacts net scientific knowledge output over a 20-year horizon.

Datasets

• NeurIPS main track submissions — total ?; splits: test (-1)
• ICLR submissions — total ?; splits: test (-1)
• arXiv monthly submissions — total ?; splits: test (-1)
• bioRxiv annual preprints — total ?; splits: test (-1)

Metrics

• normalized knowledge output (K/K₀) (primary) — range: ratio
  • Ratio of simulated net scientific knowledge output K(t) to baseline pre-AI knowledge output K₀. Computed as K(t) = R(t) * q(t), where R(t) is review throughput and q(t) is verification quality. Values >1 indicate net knowledge gain; values <1 indicate degradation.
• submission growth rate (CAGR) — range: percent
  • Compound annual growth rate of manuscript submissions over specified intervals, used to validate model predictions against empirical venue data.
• verification quality (q) — range: [0, 1]
  • Normalized score representing the average quality of peer review and verification, degrading as queue pressure increases and AI review adoption rises without proportional human oversight.

Input / output format

Input: Initial system parameters (baseline queue Q₀, initial review capacity q₀, AI writing acceleration γ, AI review acceleration δ, quality floor q_min) and discrete time steps t over a 20-year simulation horizon.

Output: Time-series trajectories for submissions S(t), review throughput R(t), queue size Q(t), verification quality q(t), and net knowledge output K(t), along with steady-state values and parameter-space heatmaps mapping (γ, δ) to long-term K/K₀.

Scoring recipe

def evaluate_model(params, empirical_data):
    # Run dynamical system simulation
    S, R, Q, q, K = simulate_dynamics(params, horizon=20)
    
    # Compute primary metric
    k_ratio = K[-1] / params['K0']
    
    # Validate against empirical submission growth
    model_cagr = compute_cagr(S)
    empirical_cagr = empirical_data['cagr']
    
    # Check qualitative consistency
    if abs(model_cagr - empirical_cagr) / empirical_cagr < 0.5:
        validation_pass = True
    else:
        validation_pass = False
        
    return {'K_over_K0': k_ratio, 'validation_pass': validation_pass}

Common pitfalls

• Confounding factors like community expansion, broadened conference scopes, and industry participation heavily influence submission growth, making it difficult to isolate AI writing tools as the sole causal driver.
• Empirical review AI adoption rates (φᵣ) rely on GPT-text detectors with known false positive/negative rates, meaning reported adoption figures are order-of-magnitude estimates rather than precise counts.
• The model assumes deterministic dynamics and uniform AI adoption, whereas real-world peer review adoption is highly stochastic, venue-specific, and constrained by editorial policies.

Evidence (verbatim from paper)

Despite the growing queue, knowledge output K(t) initially increases, peaking at 1.10K₀ at t = 3.5 yr (circa year 2026; Figure 1d). This honeymoon occurs because early throughput gains (R rises +22% ) outpace quality degradation (q falls only to 0.90). All conventional metrics including publications, throughput, turnaround appear healthy while verification debt accumulates invisibly. In the second phase, the paradox onset and sustained decline (t > 6 yr, post-2028). Knowledge output crosses below K₀ at t = 6.1 yr (year 2028), marking paradox onset (Figure 1d, red marker). Thereafter, K declines monotonically as quality erosion overwhelms throughput gains. By t = 20 yr (year 2042), K/K₀ = 0.68 (Figure 1d). ... The quality loss exceeds the throughput gain, yielding a net 32% knowledge loss. This confirms the core thesis: the problem is quality erosion driven by queue pressure, not throughput insufficiency.

Citation

@misc{kwon2026publishandperish,
  title={Publish and Perish: How AI-Accelerated Writing Without Proportional Verification Investment Degrades Scientific Knowledge},
  author={Kwon (2026)},
  year={2026},
  note={arXiv:2604.05714}
}

• arXiv: 2604.05714