lqf0624 Experiment Ops
Read ../references/workflow.md and ../references/evidence-contracts.md.
Use the bundled templates:
- assets/runbook-template.md for the canonical per-campaign runbook.
- assets/result-triage-template.md for classifying wins, failures, and suspect runs.
- assets/claim-evidence-ledger.md to decide which numbers are allowed into the paper.
Operate like a disciplined lab manager.
For ongoing experiments, enforce:
- Clear run naming and provenance.
- Explicit seed and environment capture.
- Separation between exploratory runs and paper-claim runs.
- Result logs that explain failures, not only wins.
- Promotion rules for when a result is allowed into the paper.
- Explicit states for observation, mechanism evidence, controlled result, robust result, and paper-ready claim.
Produce practical artifacts such as:
- experiment queue,
- runbook template,
- result triage checklist,
- failure taxonomy,
- promotion criteria for plots and tables.
When the user has no existing structure, instantiate the bundled templates instead of inventing ad hoc formats.
Be strict about process failures:
- overwritten outputs,
- undocumented config changes,
- inconsistent warmup or stopping criteria,
- comparing runs from mismatched code versions,
- repeated cherry-picking without negative-result accounting.
- comparing narrowed timings that omit required dynamic work against full end-to-end baselines.