Scientific Rigor

Scientific Rigor

Use this skill to make experimental claims narrow, testable, repeatable, and empirically grounded. Treat unmeasured explanations as hypotheses, not conclusions.

Workflow

1. State the decision the experiment is meant to inform.
2. Define the hypothesis and null hypothesis before running tests.
3. Identify the single independent variable being changed.
4. Hold confounders fixed: hardware, location, workload, inputs, versions, flags, time window, data source, and operator actions.
5. Define dependent metrics before observing results. Include primary metrics and guardrail metrics.
6. Establish a baseline with the same measurement path as the treatment.
7. Run enough repetitions or samples to characterize variance.
8. Use appropriate statistical checks when sample size and data shape justify them.
9. Report confidence, uncertainty, and residual risk separately from the conclusion.

Variable Discipline

Change one variable per experiment. If multiple changes already happened, split the analysis into:

• known changes
• unknown or uncontrolled changes
• measurements that can still isolate one factor
• measurements that cannot support causal claims

Do not attribute an improvement or regression to a change if another plausible change occurred in the same window.

Measurement Design

Prefer direct measurements over proxies. When proxies are unavoidable, name the inference explicitly.

When numeric measurements or effect comparisons are involved, use the get-stats skill as the required statistics contract. At minimum, every measured variable needs sample count, invalid/dropped count, min, max, mean, median, standard deviation, and standard error when estimating or comparing means. For latency and other heavy-tailed distributions, include p50, p95, and p99. For two-sample effects, include Welch's t-test when its assumptions are reasonable, or state why a different test is required.

For latency or throughput experiments, capture:

• exact code revision, config, feature flags, and host
• start/end times and timezone
• workload definition and sample selection
• warmup behavior and whether warmup samples are excluded
• raw per-event observations, not only aggregates
• p50, p95, p99, mean, standard deviation, min, max, and sample count where meaningful
• failure rate, timeout rate, dropped samples, and retry behavior

Keep raw data and commands sufficient for another operator to reproduce the analysis.

Statistical Checks

Use statistics to test a narrow question, not to decorate a weak design.

Use Welch's t-test when comparing two sets of roughly independent continuous measurements with unequal variance, such as two latency-delta samples. Report the test used, p-value, effect size, sample sizes, and whether the practical effect matters.

Use non-parametric or distribution-aware methods when data is highly skewed, heavy-tailed, censored, or dominated by outliers. For win/loss counts or first-detection share, use count-based tests such as chi-squared or Fisher's exact test when appropriate.

Never treat statistical significance as production significance. A small p-value with a tiny effect size may not justify a change.

Conclusion Standard

Use this hierarchy:

• Confirmed: measured, reproduced, and statistically/practically meaningful.
• Likely: measured with some uncertainty or limited sample size.
• Inconclusive: signal is weak, confounded, or underpowered.
• Hypothesis: plausible but not yet tested.

State what would falsify the conclusion and what the next clean experiment is.

Production-Sensitive Experiments

For live systems, preserve safety before elegance:

• prefer passive observers and read-only probes
• cap runtime and output volume
• pin sidecars away from hot cores when relevant
• avoid packet capture, logging, or polling that can perturb latency unless the perturbation is part of the experiment
• do not restart or reconfigure production processes unless the experiment explicitly includes a maintenance window

If the measurement tool can alter the system being measured, call that out and design a lower-impact measurement first.