Crypto Experiment Reproducibility
Make empirical and computational claims independently inspectable. Availability, functionality, and reproduced results are separate evidence grades.
Scope
In scope: experiment design, claim-to-artifact mapping, environment/version capture, fair baselines, statistical analysis, replay, raw evidence, portability, artifact packaging, anonymity, ethics/privacy/disclosure, and limitation reporting.
Out of scope: fabricating runs/results, calling code availability reproducibility, or releasing sensitive artifacts without authorization and risk review.
Required Inputs
- claims/tables/figures to support;
- source/data/configuration and exact versions;
- environment, hardware, parameters, seeds, workload, baselines, and expected outputs;
- sharing, anonymity, licensing, privacy, ethics, and disclosure constraints.
Success Criteria
- Every empirical/concrete claim maps to exact inputs, command/workflow, raw output, processing, and tolerance.
- Environment and dependencies can be recreated or limitations are explicit.
- Baselines use compatible security/parameters/hardware/batch/resource rules.
- Statistical design, repetitions, seeds, uncertainty, exclusions, and nondeterminism are documented.
- Artifact availability, functionality, and result reproduction receive evidence-based statuses.
- Sensitive/unshareable components have justified safe alternatives or remain limitations.
Failure Modes
- Missing raw evidence/version/config: mark claim unverified and identify minimum rerun.
- Environment cannot be recreated: provide forensic provenance and block reproducibility grade.
- Long/expensive run: use a justified evaluator-scale run without replacing full evidence.
- Sensitive/dual-use/privacy/legal constraint: stop release and require human ethics/disclosure decision.
- Result deviation: investigate before changing tolerance or narrative.
Never invent commands run, outputs, seeds, statistical significance, artifact badges, or reproducibility.
Workflow
- Apply protocol.md to build claim-to-experiment and provenance maps.
- Audit/package using artifact-checklist.md.
- Run available smoke/full/scaled experiments, retaining raw outputs and deviations; otherwise mark not executed.
- Return evidence grades, blockers, and exact reproduction instructions.
- Use evaluation-scenarios.md for future independent evaluation.
Output Contract
Return: claim-to-artifact matrix; provenance/environment manifest; experiment/baseline/statistical plan; executed runs and raw evidence locations; deviations/tolerances; availability/functionality/reproduction statuses; ethics/privacy/disclosure constraints; blockers; artifact README/open-science content.
1---2name: crypto-experiment-reproducibility3description: Use when designing, auditing, reproducing, or packaging cryptography/ZKP/lattice experiments, benchmarks, estimators, proof checkers, datasets, code artifacts, tables, figures, or claim-to-artifact evidence.4---56# Crypto Experiment Reproducibility78Make empirical and computational claims independently inspectable. Availability, functionality, and reproduced results are separate evidence grades.910## Scope1112In scope: experiment design, claim-to-artifact mapping, environment/version capture, fair baselines, statistical analysis, replay, raw evidence, portability, artifact packaging, anonymity, ethics/privacy/disclosure, and limitation reporting.1314Out of scope: fabricating runs/results, calling code availability reproducibility, or releasing sensitive artifacts without authorization and risk review.1516## Required Inputs1718- claims/tables/figures to support;19- source/data/configuration and exact versions;20- environment, hardware, parameters, seeds, workload, baselines, and expected outputs;21- sharing, anonymity, licensing, privacy, ethics, and disclosure constraints.2223## Success Criteria2425- Every empirical/concrete claim maps to exact inputs, command/workflow, raw output, processing, and tolerance.26- Environment and dependencies can be recreated or limitations are explicit.27- Baselines use compatible security/parameters/hardware/batch/resource rules.28- Statistical design, repetitions, seeds, uncertainty, exclusions, and nondeterminism are documented.29- Artifact availability, functionality, and result reproduction receive evidence-based statuses.30- Sensitive/unshareable components have justified safe alternatives or remain limitations.3132## Failure Modes3334- Missing raw evidence/version/config: mark claim unverified and identify minimum rerun.35- Environment cannot be recreated: provide forensic provenance and block reproducibility grade.36- Long/expensive run: use a justified evaluator-scale run without replacing full evidence.37- Sensitive/dual-use/privacy/legal constraint: stop release and require human ethics/disclosure decision.38- Result deviation: investigate before changing tolerance or narrative.3940Never invent commands run, outputs, seeds, statistical significance, artifact badges, or reproducibility.4142## Workflow43441. Apply [protocol.md](references/protocol.md) to build claim-to-experiment and provenance maps.452. Audit/package using [artifact-checklist.md](references/artifact-checklist.md).463. Run available smoke/full/scaled experiments, retaining raw outputs and deviations; otherwise mark not executed.474. Return evidence grades, blockers, and exact reproduction instructions.485. Use [evaluation-scenarios.md](references/evaluation-scenarios.md) for future independent evaluation.4950## Output Contract5152Return: claim-to-artifact matrix; provenance/environment manifest; experiment/baseline/statistical plan; executed runs and raw evidence locations; deviations/tolerances; availability/functionality/reproduction statuses; ethics/privacy/disclosure constraints; blockers; artifact README/open-science content.53
RiaoMike