Rigorous Experiments
Run statistical experiments on observational/personal time-series data that
survive scrutiny. Distilled from a 54-experiment n-of-1 program in which
sampled permutation tests, missing-data artifacts, app-categorization bugs
and collinear mechanisms repeatedly manufactured — and then destroyed —
"findings". Every rule here exists because its absence once produced a
wrong conclusion.
Modes
Pick the mode matching the request; chain them for a full study.
| Mode |
When |
Reference |
| design |
New hypothesis or study |
references/design.md |
| conduct |
Implementing + running the experiment |
references/statistics.md |
| validate-data |
Before trusting ANY new data source |
references/data-validation.md |
| cross-validate |
Findings worth defending; code review; external model review (e.g. GPT Pro) |
references/cross-validation.md |
| investigate-leads |
A sweep/run produced leads (p<0.06, not FDR-confirmed) |
references/lead-investigation.md |
| audit |
Re-examining past claims, registries of findings |
references/statistics.md §Audit |
Non-negotiable core (all modes)
- Pre-register before computing. Hypotheses, exact tests, family size
m, and the acceptance threshold go in the script docstring BEFORE the
first run. Post-hoc tests are reported as descriptive, never promoted.
- Exact permutation, never sampled, on small n. A session sequence of
n=19 has 18 circular shifts: the minimum honest p is ~1/19≈0.05.
Sampling 2000 shifts with replacement fabricates precision (this killed
a flagship "q=0.028" finding). Use
scripts/perm_stats.py.
- Permute over the full calendar, not the compressed series. Shifting
a gap-compressed series breaks the timeline; keep missingness as NaN
masks re-applied per shift. Event indicators must be pure 0/1 with no
gaps — missingness lives only in the outcome series.
- BH with FIXED family size m, a LITERAL CONSTANT declared at design
time — never
len(tests) (that defeats pre-registration; the linter
rejects it). Assert the run matches the declared m. Confirmatory
families small and separate from exploratory sweeps; pooling everything
into one BH buries true effects, cherry-picking families manufactures
them. Plain BH assumes independent/positively-dependent tests; for
strongly dependent lag families use BH-Yekutieli or maxT resampling.
- Stationarity check before correlating trending series. Exact
circular shift on a trending series is "exactly, reproducibly wrong":
report prewhitened-r (AR1 residuals) and stationary bootstrap alongside.
- Stratify before pooling (Simpson check): within group (e.g.
therapy/coaching) and within regime (pre/post known breaks). A pooled
r=−0.25 once hid therapy −0.64 vs coaching +0.53.
- Controls can re-describe a finding, not just kill it. When a control
collapses an effect, check collinearity of control and predictor —
r(self-focus, session-length)=0.79 meant "mechanism ambiguous", not
"effect fake". Report the decomposition.
- Honest statuses: confirmed (q<0.10 exact) ≠ lead (p<0.06) ≠ null ≠
descriptive. Status flips are recorded, never silently edited. Nulls
with adequate power are findings. Robust ≠ significant: a lead
surviving leave-one-out at small n is still underpowered — a candidate
for prospective test, not a finding.
8b. Series scope is part of the test. A lagged "[t+1]" means the next
unit in the series the hypothesis is about, not the next pooled row;
define scope before lagging (it once flipped a sign). When recomputing
a prior result, reproduce a stored artifact on that scope first.
- Privacy: raw text/audio never enters output files or external
uploads — statistics, rates and embedding-derived scores only.
- Plain-language reporting: every statistic carries its practical
meaning inline; define r/p/q/n once per report; no untranslated jargon
calques. Narrative first, numbers as support.
Workflow (full study)
validate-data gate on any new source (see reference — the checklist
has caught: zero-vs-missing conflation, dedup semantics, substring
category bugs, rolling purge windows, timezone conventions).
design: pre-registered hypotheses + family + power sanity.
conduct: implement with scripts/perm_stats.py; run; write results
JSON with tests, statuses, and caveats including known limitations.
cross-validate: adversarial code review (e.g. Codex read-only) BEFORE
trusting results; fix findings; re-run. For major claims, external
model review with a privacy-screened archive.
investigate-leads on anything that surfaced as a lead (not at the
same scale — the triage battery: LOO, directionality, detrend-vs-step,
within-cycle, prewhiten+bootstrap; consolidate same-direction leads
into one composite). Mark diagnostic runs descriptive_only: true.
- Verdicts in honest prose (mixed/rejected allowed); report; registry
update with status provenance.
Viewing results
Launch the bundled explorer over any directory of results JSONs:
python3 scripts/explorer.py <results_dir> [--port 8799] [--pattern "exp*.json"] [--sort newest|oldest]
Generates explorer.html in the directory, starts (or reuses) a loopback
http server on the port, and opens the browser: experiment list with
confirmed/lead badges, filter, sortable test tables color-coded by
status, verdicts, caveats, raw JSON. The page fetches result files live —
re-running experiments updates the view; re-run the script only when new
result files appear. Serve over localhost, never file:// (CDN fonts) and
never on a non-loopback interface (results may contain personal
statistics).
Evals
Run python3 evals/run_evals.py (from the skill directory) to lint an
experiment script/results pair against the standards (pre-registration
present, fixed literal m, exact perm usage, caveats, no raw text in
outputs). A diagnostic/triage run that intentionally mints no new tests
sets descriptive_only: true in its results JSON to satisfy the
"has tests" check. Eval cases in evals/cases/ document expected
pass/fail examples.
1---
2name: rigorous-experiments
3description: This skill should be used when designing, running, validating, or auditing statistical experiments on personal or observational time-series data (health metrics, speech/text corpora, behavioral logs, diaries, n-of-1 self-tracking). It enforces pre-registration, exact permutation tests, FDR discipline, data-validation gates, adversarial code review, and cross-validation with external models. Triggers on "design an experiment", "test this hypothesis on my data", "is this correlation real", "audit these findings", "pre-register", "validate this dataset", or any n-of-1 / quantified-self analysis request.
4metadata:
5 source: https://skillmd.com/skills/glebis/rigorous-experiments
6---
7
8# Rigorous Experiments
9
10Run statistical experiments on observational/personal time-series data that
11survive scrutiny. Distilled from a 54-experiment n-of-1 program in which
12sampled permutation tests, missing-data artifacts, app-categorization bugs
13and collinear mechanisms repeatedly manufactured — and then destroyed —
14"findings". Every rule here exists because its absence once produced a
15wrong conclusion.
16
17## Modes
18
19Pick the mode matching the request; chain them for a full study.
20
21| Mode | When | Reference |
22|------|------|-----------|
23| **design** | New hypothesis or study | `references/design.md` |
24| **conduct** | Implementing + running the experiment | `references/statistics.md` |
25| **validate-data** | Before trusting ANY new data source | `references/data-validation.md` |
26| **cross-validate** | Findings worth defending; code review; external model review (e.g. GPT Pro) | `references/cross-validation.md` |
27| **investigate-leads** | A sweep/run produced leads (p<0.06, not FDR-confirmed) | `references/lead-investigation.md` |
28| **audit** | Re-examining past claims, registries of findings | `references/statistics.md` §Audit |
29
30## Non-negotiable core (all modes)
31
321. **Pre-register before computing.** Hypotheses, exact tests, family size
33 m, and the acceptance threshold go in the script docstring BEFORE the
34 first run. Post-hoc tests are reported as descriptive, never promoted.
352. **Exact permutation, never sampled, on small n.** A session sequence of
36 n=19 has 18 circular shifts: the minimum honest p is ~1/19≈0.05.
37 Sampling 2000 shifts with replacement fabricates precision (this killed
38 a flagship "q=0.028" finding). Use `scripts/perm_stats.py`.
393. **Permute over the full calendar, not the compressed series.** Shifting
40 a gap-compressed series breaks the timeline; keep missingness as NaN
41 masks re-applied per shift. Event indicators must be pure 0/1 with no
42 gaps — missingness lives only in the outcome series.
434. **BH with FIXED family size m**, a LITERAL CONSTANT declared at design
44 time — never `len(tests)` (that defeats pre-registration; the linter
45 rejects it). Assert the run matches the declared m. Confirmatory
46 families small and separate from exploratory sweeps; pooling everything
47 into one BH buries true effects, cherry-picking families manufactures
48 them. Plain BH assumes independent/positively-dependent tests; for
49 strongly dependent lag families use BH-Yekutieli or maxT resampling.
505. **Stationarity check before correlating trending series.** Exact
51 circular shift on a trending series is "exactly, reproducibly wrong":
52 report prewhitened-r (AR1 residuals) and stationary bootstrap alongside.
536. **Stratify before pooling** (Simpson check): within group (e.g.
54 therapy/coaching) and within regime (pre/post known breaks). A pooled
55 r=−0.25 once hid therapy −0.64 vs coaching +0.53.
567. **Controls can re-describe a finding, not just kill it.** When a control
57 collapses an effect, check collinearity of control and predictor —
58 r(self-focus, session-length)=0.79 meant "mechanism ambiguous", not
59 "effect fake". Report the decomposition.
608. **Honest statuses**: confirmed (q<0.10 exact) ≠ lead (p<0.06) ≠ null ≠
61 descriptive. Status flips are recorded, never silently edited. Nulls
62 with adequate power are findings. **Robust ≠ significant**: a lead
63 surviving leave-one-out at small n is still underpowered — a candidate
64 for prospective test, not a finding.
658b. **Series scope is part of the test.** A lagged "[t+1]" means the next
66 unit in the series the hypothesis is about, not the next pooled row;
67 define scope before lagging (it once flipped a sign). When recomputing
68 a prior result, reproduce a stored artifact on that scope first.
699. **Privacy**: raw text/audio never enters output files or external
70 uploads — statistics, rates and embedding-derived scores only.
7110. **Plain-language reporting**: every statistic carries its practical
72 meaning inline; define r/p/q/n once per report; no untranslated jargon
73 calques. Narrative first, numbers as support.
74
75## Workflow (full study)
76
771. `validate-data` gate on any new source (see reference — the checklist
78 has caught: zero-vs-missing conflation, dedup semantics, substring
79 category bugs, rolling purge windows, timezone conventions).
802. `design`: pre-registered hypotheses + family + power sanity.
813. `conduct`: implement with `scripts/perm_stats.py`; run; write results
82 JSON with tests, statuses, and caveats including known limitations.
834. `cross-validate`: adversarial code review (e.g. Codex read-only) BEFORE
84 trusting results; fix findings; re-run. For major claims, external
85 model review with a privacy-screened archive.
865. `investigate-leads` on anything that surfaced as a lead (not at the
87 same scale — the triage battery: LOO, directionality, detrend-vs-step,
88 within-cycle, prewhiten+bootstrap; consolidate same-direction leads
89 into one composite). Mark diagnostic runs `descriptive_only: true`.
906. Verdicts in honest prose (mixed/rejected allowed); report; registry
91 update with status provenance.
92
93## Viewing results
94
95Launch the bundled explorer over any directory of results JSONs:
96
97```bash
98python3 scripts/explorer.py <results_dir> [--port 8799] [--pattern "exp*.json"] [--sort newest|oldest]
99```
100
101Generates `explorer.html` in the directory, starts (or reuses) a loopback
102http server on the port, and opens the browser: experiment list with
103confirmed/lead badges, filter, sortable test tables color-coded by
104status, verdicts, caveats, raw JSON. The page fetches result files live —
105re-running experiments updates the view; re-run the script only when new
106result files appear. Serve over localhost, never file:// (CDN fonts) and
107never on a non-loopback interface (results may contain personal
108statistics).
109
110## Evals
111
112Run `python3 evals/run_evals.py` (from the skill directory) to lint an
113experiment script/results pair against the standards (pre-registration
114present, fixed literal m, exact perm usage, caveats, no raw text in
115outputs). A diagnostic/triage run that intentionally mints no new tests
116sets `descriptive_only: true` in its results JSON to satisfy the
117"has tests" check. Eval cases in `evals/cases/` document expected
118pass/fail examples.