Alterlab Results Transparency

Results Transparency — The Reporting Discipline Gate

Skill type: DISCIPLINE-ENFORCING. This skill is not a how-to for running statistics. It is a gate that stands between analyses you ran and the sentence you are about to write in the Results section. It exists to stop the single most common integrity failure in quantitative write-ups: reporting the analyses that "worked" and quietly dropping the rest.

It orchestrates, it does not reimplement:

• REQUIRED BACKGROUND — alterlab-statistical-analysis runs the tests, checks assumptions (its alterlab-statistical-analysis/scripts/assumption_checks.py), computes effect sizes and CIs, and produces the APA-formatted numbers. This skill checks that those outputs are all present and disclosed before a claim is written.
• REQUIRED BACKGROUND — alterlab-preregistration-discipline owns the frozen plan. This skill checks that confirmatory claims match it and that deviations are labelled.
• alterlab-open-science owns the disclosure mechanics (TOP Guidelines, registration links, data/code availability).
• alterlab-scientific-thinking owns judging evidence quality; this skill only enforces reporting completeness.

The Iron Law

NO RESULTS CLAIM WITHOUT REPORTING EVERY ANALYSIS RUN

Violating the letter of full reporting is violating the spirit of the science. "Reporting every analysis" means every test you actually executed against the data for this hypothesis — including the ones that returned p > .05, the specification you abandoned, and the outlier rule you tried first. A Results section is a census of what you did, not a highlight reel of what confirmed you.

When to Use This Skill

Invoke it the moment a results claim is being written or defended:

• About to write a sentence in the Results / Findings section.
• Stating "we found that…", "X significantly predicted Y", "there was no effect".
• Reporting a p-value, a coefficient, a group difference, or a correlation.
• Defending a finding drawn from a subset of the analyses that were run.
• Tempted to drop outliers, switch a test, or add a covariate after seeing the result.
• Writing the abstract's results sentence or a figure caption that asserts a finding.

If you are choosing which test to run, checking assumptions, or computing an effect size, that is alterlab-statistical-analysis — come back here when you move from computing numbers to claiming findings.

Does NOT Trigger — route these elsewhere

The Gate Function

Before any results claim is written, run these steps in order. This mirrors the verification-before-completion gate (obra/superpowers): evidence before assertions, always.

1. IDENTIFY — State the exact claim sentence you intend to write.
2. LIST — Enumerate every test actually run against this hypothesis, including the ones that did not "work" (p > .05, abandoned specs, alternative exclusion rules). If the list has more entries than the write-up mentions, STOP — the gap is selective reporting.
3. CHECK assumptions reported — For each reported test, the assumption results from alterlab-statistical-analysis/scripts/assumption_checks.py (normality, homogeneity, linearity, outliers) must appear in the write-up, with the remediation taken if any were violated.
4. CHECK effect size + CI present — The claim carries an effect size with a 95% confidence interval, not a bare p-value. A p-value alone never clears the gate (see the sufficiency table).
5. CHECK pre-registration deviations disclosed — Every difference from the frozen plan (alterlab-preregistration-discipline) is named and labelled confirmatory vs. exploratory.
6. ONLY THEN — Write the sentence, with the test name, statistic, df, exact p, effect size + CI, and the confirmatory/exploratory label.

A self-contained checker that enforces steps 2–5 over a small JSON manifest of the claim is scripts/reporting_gate.py (stdlib only; see Quick Start).

What is sufficient vs. not sufficient

Analogous to the verification-gap table: the left column does not clear the gate; the right column is the actual requirement.

Excuse vs. Reality

Seeded from the rationalizations that precede selective reporting. When you hear yourself think the left column, the right column is the truth.

Red Flags — STOP

If you catch yourself thinking any of these, STOP. You are about to exploit a researcher degree of freedom. Either report it in full or label it exploratory.

• "Let me just report the analyses that worked."
• "I'll drop these outliers and rerun before writing it up."
• "The effect is there if I add this one covariate."
• "Let me try a different test — this one isn't quite significant."
• "We can leave the non-significant models out of the Results."
• "This subgroup is interesting, I'll write it as a finding."
• "Close enough to p < .05 — I'll call it a trend and move on."
• "No need to mention we changed the primary outcome."
• "I'll round the p-value down and skip the effect size for now."

All of these mean: STOP. Run the Gate Function. Report everything, or label it exploratory.

Escalation Gate (multiple comparisons / test-shopping)

Mirrors the systematic-debugging "3 failures = wrong approach" rule:

Ran 3+ tests on the same hypothesis searching for significance? STOP. This is multiple comparisons / test-shopping. Either correct for all of them (Bonferroni / FDR — via alterlab-statistical-analysis) or declare the analysis exploratory. Do not run test #4 to find p < .05.

Quick Start

# 1. Build a tiny manifest of the claim and what was actually run (see references).
# 2. Run the gate; non-zero exit = claim is not clear to write.
uv run python skills/methodology/alterlab-results-transparency/scripts/reporting_gate.py claim.json

The script is a deterministic transparency linter, not a statistics engine: it checks that the manifest lists all runs, that an effect size + CI accompany every reported claim, that assumption results are present, and that deviations are labelled. It never computes a p-value or an effect size — that is alterlab-statistical-analysis's job.

Deeper Detail

• references/gate_function.md — the full gate walkthrough, the claim-manifest schema, and a worked pass/fail example.
• references/transparency_standards.md — what a complete Results report contains (CONSORT/STROBE results items, TOP Guidelines reporting standards, APA results requirements) and how this skill maps onto sibling skills.

Self-Check Before Reporting

• Did you LIST every test run, or only the ones in the draft?
• Does every claim carry an effect size + 95% CI, not a bare p-value?
• Are the assumption-check results from alterlab-statistical-analysis reported?
• Is every deviation from the frozen plan labelled confirmatory vs. exploratory?
• Ran 3+ tests on one hypothesis? Corrected, or declared exploratory?

Part of the AlterLab Academic Skills suite.