Statistics & Reproducibility (sci-statistics)
When to trigger
- Results report P values but not effect sizes or n.
- "Three independent experiments" is claimed but replication is unclear.
- Multiple comparisons are run with no correction.
- A reviewer is likely to ask "were analyses pre-specified?" and there's no answer.
The reporting backbone (every quantitative claim)
Each claim needs: effect size + uncertainty + n + test + what n means.
Replication and design
- Distinguish biological replication (independent samples) from technical replication (re-measurement). Science cares about the former.
- State how the sample size was chosen (power analysis or rationale), not post-hoc.
- Report randomization of subjects/treatments and blinding of measurement/analysis where applicable, or state why not.
- Report inclusion/exclusion criteria and any data excluded, with reasons, decided in advance.
Avoid the classic reviewer kills
- Pseudoreplication: treating technical replicates / cells from one animal as independent n.
- HARKing / p-hacking: presenting exploratory findings as confirmatory. If exploratory, label them.
- "Representative" images with no quantification across replicates.
- Bar chart + SEM masking a tiny, variable n.
- Comparing two effects by their significance ("significant here, not there") instead of testing the difference.
Reproducibility package
- Analysis code in a repository (see
sci-data), with a README and environment/versions.
- A reproducibility / reporting summary if requested by the journal — list software, versions, seeds.
- Deterministic where possible; report random seeds for simulations/ML.
Pre-registration & transparency (where relevant)
- For confirmatory studies (especially with human subjects/behavioral work), note pre-registration (OSF/AsPredicted) if done.
- Separate pre-specified analyses from post-hoc exploration explicitly in the text.
Statistics pass for Science
Use this as a second-pass capability check. First lock the broad discovery claim, decisive evidence, uncertainty/limitations, and why the result belongs in a general-science weekly; then test whether the manuscript addresses general-science reviewers and editors who ask whether the result changes a broad field, is technically decisive, and can be understood outside the subdiscipline.
- Primary move: Check estimand, denominator, uncertainty, multiplicity, missing data, sensitivity, and reporting standard before interpreting any result.
- Decision ledger: return
claim / evidence / blocker / next edit rows so the next pass can patch the manuscript directly.
- Neighbor test: compare against Nature for similar broad-scope novelty, PNAS for academy-wide breadth, specialist journals when the claim is field-internal; if the neighboring outlet has the stronger audience claim, recommend re-routing before polishing.
- Verification floor: before submission-ready advice, re-open
resources/official-source-map.md for volatile rules and name the one unresolved fact that could change the recommendation.
Output format
【Per-claim backbone】 effect+CI / n / unit-of-n / test / assumptions → list gaps
【Replication】 biological vs technical clear? yes/no
【Sample-size rationale】 power/justification present? yes/no
【Randomization & blinding】 reported / N/A-justified / missing
【Multiplicity】 corrected? method
【Reproducibility】 code + versions + seeds present? yes/no
【Next】 sci-data
Anti-patterns
- Do not report P without effect size and n.
- Do not count technical replicates as independent observations.
- Do not infer "no effect" from a non-significant test on an underpowered sample.
- Do not present post-hoc subgroup findings as if pre-specified.
Source: brycewang-stanford/Awesome-Journal-Skills → Science-Skills/skills/sci-statistics/SKILL.md
1---2name: sci-statistics3description: Use to enforce Science's statistics and reproducibility reporting — n and replication, test choice and assumptions, effect sizes with uncertainty, multiple-comparison control, randomization/blinding, and pre-registration where relevant.4---5
6
7# Statistics & Reproducibility (sci-statistics)
8
9## When to trigger
10
11- Results report P values but not effect sizes or n.
12- "Three independent experiments" is claimed but replication is unclear.
13- Multiple comparisons are run with no correction.
14- A reviewer is likely to ask "were analyses pre-specified?" and there's no answer.
15
16## The reporting backbone (every quantitative claim)
17
18Each claim needs: **effect size + uncertainty + n + test + what n means.**
19
20- [ ] **n** stated, with the unit of replication (biological vs technical replicates; cells vs animals vs experiments).
21- [ ] **Effect size** with **95% CI** (preferred) or SD/SEM clearly labeled — not P alone.
22- [ ] **Exact P values** (e.g., P = 0.013), not "P < 0.05", unless extremely small.
23- [ ] **Test named and justified** (and its assumptions checked: normality, variance homogeneity, independence).
24- [ ] **Multiple comparisons** corrected (Bonferroni/Holm/FDR) when many tests are run.
25
26## Replication and design
27
28- Distinguish **biological replication** (independent samples) from **technical replication** (re-measurement). Science cares about the former.
29- State **how the sample size was chosen** (power analysis or rationale), not post-hoc.
30- Report **randomization** of subjects/treatments and **blinding** of measurement/analysis where applicable, or state why not.
31- Report **inclusion/exclusion criteria** and any data excluded, with reasons, decided in advance.
32
33## Avoid the classic reviewer kills
34
35- **Pseudoreplication**: treating technical replicates / cells from one animal as independent n.
36- **HARKing / p-hacking**: presenting exploratory findings as confirmatory. If exploratory, label them.
37- **"Representative" images** with no quantification across replicates.
38- **Bar chart + SEM** masking a tiny, variable n.
39- Comparing two effects by their **significance** ("significant here, not there") instead of testing the **difference**.
40
41## Reproducibility package
42
43- Analysis code in a repository (see `sci-data`), with a README and environment/versions.
44- A reproducibility / reporting summary if requested by the journal — list software, versions, seeds.
45- Deterministic where possible; report random seeds for simulations/ML.
46
47## Pre-registration & transparency (where relevant)
48
49- For confirmatory studies (especially with human subjects/behavioral work), note **pre-registration** (OSF/AsPredicted) if done.
50- Separate pre-specified analyses from post-hoc exploration explicitly in the text.
51
52
53## Statistics pass for Science
54
55Use this as a second-pass capability check. First lock the broad discovery claim, decisive evidence, uncertainty/limitations, and why the result belongs in a general-science weekly; then test whether the manuscript addresses general-science reviewers and editors who ask whether the result changes a broad field, is technically decisive, and can be understood outside the subdiscipline.
56
57- **Primary move:** Check estimand, denominator, uncertainty, multiplicity, missing data, sensitivity, and reporting standard before interpreting any result.
58- **Decision ledger:** return `claim / evidence / blocker / next edit` rows so the next pass can patch the manuscript directly.
59- **Neighbor test:** compare against Nature for similar broad-scope novelty, PNAS for academy-wide breadth, specialist journals when the claim is field-internal; if the neighboring outlet has the stronger audience claim, recommend re-routing before polishing.
60- **Verification floor:** before submission-ready advice, re-open `resources/official-source-map.md` for volatile rules and name the one unresolved fact that could change the recommendation.
61
62## Output format
63
64```
65【Per-claim backbone】 effect+CI / n / unit-of-n / test / assumptions → list gaps
66【Replication】 biological vs technical clear? yes/no
67【Sample-size rationale】 power/justification present? yes/no
68【Randomization & blinding】 reported / N/A-justified / missing
69【Multiplicity】 corrected? method
70【Reproducibility】 code + versions + seeds present? yes/no
71【Next】 sci-data
72```
73
74## Anti-patterns
75
76- **Do not** report P without effect size and n.
77- **Do not** count technical replicates as independent observations.
78- **Do not** infer "no effect" from a non-significant test on an underpowered sample.
79- **Do not** present post-hoc subgroup findings as if pre-specified.
80
81---
82
83**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `Science-Skills/skills/sci-statistics/SKILL.md`
thedixitjain