yogsoth-ai Tactic: Reproducibility Protocol
Orchestration Pattern
- Assess Reproducibility Requirements → Determine level needed (exact, statistical, conceptual)
- seed-protocol-design → Design random seed strategy for all stochastic components
- environment-specification → Lock all software, hardware, and configuration dependencies
- Define Verification Plan → How to confirm reproducibility (re-run subset, cross-machine test)
- Document Non-Determinism → Identify and document unavoidable sources of variance
Decision Criteria
| Reproducibility Level | Requirement | When to Use |
|---|---|---|
| Exact (bit-for-bit) | Same hardware + deterministic ops | Debugging, verification |
| Statistical (within CI) | Same distribution of results | Standard research |
| Conceptual (same conclusion) | Same qualitative findings | Cross-platform validation |
Quality Checks
- Are all random seeds documented and controllable?
- Is the full software environment captured (versions, dependencies)?
- Are hardware-specific non-determinisms identified (GPU atomics, cuDNN)?
- Is there a verification protocol (re-run N times, check variance)?
- Are data preprocessing steps deterministic or seeded?
- Is the order of data loading controlled?
- Are model initialization seeds separate from data shuffling seeds?
- Is there a plan for cross-machine reproducibility testing?
Available SOPs
Optional, no fixed order; the final leaf is always a sop.
| SOP | When to use |
|---|---|
| environment-specification | SOP: define complete experiment environment specification |
| seed-protocol-design | SOP: design random seed strategy for reproducibility |