# Scientific Rigor

> Design, run, and evaluate controlled empirical experiments with one variable changed at a time, explicit hypotheses, reproducible protocols, statistical significance checks such as t-tests where appropriate, and conclusions backed by measured evidence. Use when the agent is planning or judging performance tests, latency experiments, A/B comparisons, benchmark runs, optimization claims, production-vs-baseline comparisons, or any investigation where assumptions must be minimized and results must be statistically defensible.

- Skill: `dreamers-laboratory/scientific-rigor` (Agent Skill)
- Install (CLI): `npx skillmds@latest add dreamers-laboratory/scientific-rigor`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dreamers-laboratory/scientific-rigor/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: dreamers-laboratory (https://skillmd.com/u/dreamers-laboratory)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/dreamers-laboratory/scientific-rigor

---


# Scientific Rigor

Use this skill to make experimental claims narrow, testable, repeatable, and
empirically grounded. Treat unmeasured explanations as hypotheses, not
conclusions.

## Workflow

1. State the decision the experiment is meant to inform.
2. Define the hypothesis and null hypothesis before running tests.
3. Identify the single independent variable being changed.
4. Hold confounders fixed: hardware, location, workload, inputs, versions,
   flags, time window, data source, and operator actions.
5. Define dependent metrics before observing results. Include primary metrics
   and guardrail metrics.
6. Establish a baseline with the same measurement path as the treatment.
7. Run enough repetitions or samples to characterize variance.
8. Use appropriate statistical checks when sample size and data shape justify
   them.
9. Report confidence, uncertainty, and residual risk separately from the
   conclusion.

## Variable Discipline

Change one variable per experiment. If multiple changes already happened,
split the analysis into:

- known changes
- unknown or uncontrolled changes
- measurements that can still isolate one factor
- measurements that cannot support causal claims

Do not attribute an improvement or regression to a change if another plausible
change occurred in the same window.

## Measurement Design

Prefer direct measurements over proxies. When proxies are unavoidable, name the
inference explicitly.

When numeric measurements or effect comparisons are involved, use
the `get-stats` skill as the required statistics contract. At minimum, every measured
variable needs sample count, invalid/dropped count, min, max, mean, median,
standard deviation, and standard error when estimating or comparing means. For
latency and other heavy-tailed distributions, include p50, p95, and p99. For
two-sample effects, include Welch's t-test when its assumptions are reasonable,
or state why a different test is required.

For latency or throughput experiments, capture:

- exact code revision, config, feature flags, and host
- start/end times and timezone
- workload definition and sample selection
- warmup behavior and whether warmup samples are excluded
- raw per-event observations, not only aggregates
- p50, p95, p99, mean, standard deviation, min, max, and sample count where
  meaningful
- failure rate, timeout rate, dropped samples, and retry behavior

Keep raw data and commands sufficient for another operator to reproduce the
analysis.

## Statistical Checks

Use statistics to test a narrow question, not to decorate a weak design.

Use Welch's t-test when comparing two sets of roughly independent continuous
measurements with unequal variance, such as two latency-delta samples. Report
the test used, p-value, effect size, sample sizes, and whether the practical
effect matters.

Use non-parametric or distribution-aware methods when data is highly skewed,
heavy-tailed, censored, or dominated by outliers. For win/loss counts or
first-detection share, use count-based tests such as chi-squared or Fisher's
exact test when appropriate.

Never treat statistical significance as production significance. A small
p-value with a tiny effect size may not justify a change.

## Conclusion Standard

Use this hierarchy:

- **Confirmed**: measured, reproduced, and statistically/practically meaningful.
- **Likely**: measured with some uncertainty or limited sample size.
- **Inconclusive**: signal is weak, confounded, or underpowered.
- **Hypothesis**: plausible but not yet tested.

State what would falsify the conclusion and what the next clean experiment is.

## Production-Sensitive Experiments

For live systems, preserve safety before elegance:

- prefer passive observers and read-only probes
- cap runtime and output volume
- pin sidecars away from hot cores when relevant
- avoid packet capture, logging, or polling that can perturb latency unless the
  perturbation is part of the experiment
- do not restart or reconfigure production processes unless the experiment
  explicitly includes a maintenance window

If the measurement tool can alter the system being measured, call that out and
design a lower-impact measurement first.