STANDARD OPERATING PROCEDURE
Purpose
Assess reproducibility across code, data, and build pipelines by validating deterministic processes, pinned dependencies, and documented steps.
Trigger Conditions
- Positive: scientific/ML experiment verification, release build audits, or compliance checks requiring reproducible artifacts.
- Negative: stylistic reviews or runtime debugging (route to style-audit or functionality-audit).
Guardrails
- Confidence ceiling: Append
Confidence: X.XX (ceiling: TYPE Y.YY) using ceilings {inference/report 0.70, research 0.85, observation/definition 0.95}.
- Evidence & determinism: Require commands, seeds, data sources, and hashes for artifacts; rerun steps to confirm.
- Structure-first: Maintain examples/tests showing successful and failing reproduction attempts.
- Adversarial validation: Introduce clean environments and altered dependency versions to detect hidden variability.
Execution Phases
- Inventory & Scope
- Identify targets (builds, experiments, reports) and required inputs (code revision, data, config, seeds).
- Note environments (OS, container, hardware) and expected outcomes.
- Replay & Measurement
- Follow documented steps exactly; log commands, outputs, and timestamps.
- Compare generated artifacts via hashes or checksums; capture diffs.
- Variability Probes
- Change environments/dependency versions within constraints to test stability.
- Document non-deterministic behaviors and their causes.
- Reporting & Remediation
- Summarize reproduction success/failure, missing documentation, and proposed fixes (pinning, automation scripts, data versioning).
- Provide confidence with ceiling and attach logs/hashes.
Output Format
- Scope, inputs, environments, and expected results.
- Step-by-step replay log with evidence (commands, outputs, hashes).
- Variability findings and fixes.
- Confidence statement using ceiling syntax.
Validation Checklist
Confidence: 0.72 (ceiling: inference 0.70) - SOP rewritten per Prompt Architect confidence discipline and Skill Forge structure-first reproducibility focus.
1---2name: reproducibility-audit-23description: Verify that results, builds, and experiments can be reproduced consistently with documented steps and deterministic inputs.4---5
6
7
8
9## STANDARD OPERATING PROCEDURE
10
11### Purpose
12Assess reproducibility across code, data, and build pipelines by validating deterministic processes, pinned dependencies, and documented steps.
13
14### Trigger Conditions
15- **Positive:** scientific/ML experiment verification, release build audits, or compliance checks requiring reproducible artifacts.
16- **Negative:** stylistic reviews or runtime debugging (route to style-audit or functionality-audit).
17
18### Guardrails
19- **Confidence ceiling:** Append `Confidence: X.XX (ceiling: TYPE Y.YY)` using ceilings {inference/report 0.70, research 0.85, observation/definition 0.95}.
20- **Evidence & determinism:** Require commands, seeds, data sources, and hashes for artifacts; rerun steps to confirm.
21- **Structure-first:** Maintain examples/tests showing successful and failing reproduction attempts.
22- **Adversarial validation:** Introduce clean environments and altered dependency versions to detect hidden variability.
23
24### Execution Phases
251. **Inventory & Scope**
26 - Identify targets (builds, experiments, reports) and required inputs (code revision, data, config, seeds).
27 - Note environments (OS, container, hardware) and expected outcomes.
282. **Replay & Measurement**
29 - Follow documented steps exactly; log commands, outputs, and timestamps.
30 - Compare generated artifacts via hashes or checksums; capture diffs.
313. **Variability Probes**
32 - Change environments/dependency versions within constraints to test stability.
33 - Document non-deterministic behaviors and their causes.
344. **Reporting & Remediation**
35 - Summarize reproduction success/failure, missing documentation, and proposed fixes (pinning, automation scripts, data versioning).
36 - Provide confidence with ceiling and attach logs/hashes.
37
38### Output Format
39- Scope, inputs, environments, and expected results.
40- Step-by-step replay log with evidence (commands, outputs, hashes).
41- Variability findings and fixes.
42- Confidence statement using ceiling syntax.
43
44### Validation Checklist
45- [ ] Inputs and environments captured with versions/seeds.
46- [ ] Replay executed with logs and hashes recorded.
47- [ ] Variability probes performed; nondeterminism documented.
48- [ ] Remediation steps proposed and owners identified.
49- [ ] Confidence ceiling provided; English-only output.
50
51Confidence: 0.72 (ceiling: inference 0.70) - SOP rewritten per Prompt Architect confidence discipline and Skill Forge structure-first reproducibility focus.
majiayu000