Design Model Experiment
Outcome
Produce an experiment manifest that another operator can run, audit, and compare without reconstructing unstated choices.
Workflow
- Write one falsifiable hypothesis and the decision it informs.
- Define the experimental unit: model revision, adapter, prompt set, intervention, runtime, or harness.
- Name exactly one primary controlled variable per comparison. Record every intentional difference.
- Select a baseline and explain why it is a fair comparator.
- Pin model, tokenizer, dataset, code, dependency, template, evaluator, and seed provenance.
- Define primary metrics, guardrail metrics, uncertainty treatment, and failure thresholds before the run.
- Estimate compute, storage, time, and paid cost. Set smoke-test and full-run stop conditions.
- Define raw and derived artifacts, retention, and sensitive-data handling.
- Copy
assets/experiment-manifest.json, fill it, then run:
dotnet fsi scripts/validate-experiment-manifest.fsx path/to/experiment.json
- Run the smallest experiment capable of detecting configuration or pipeline failure before spending the full budget.
Evidence Contract
Keep configuration validation, smoke-run evidence, and final experimental evidence separate. A successful process exit proves execution, not model quality. Report deviations from the manifest before interpreting results.
Resources
assets/experiment-manifest.json: managed experiment template.references/experiment-design.md: field semantics and comparison rules.scripts/validate-experiment-manifest.fsx: deterministic structural validation.
1---2name: design-model-experiment3description: Design a reproducible model experiment with hypotheses, controls, baselines, metrics, budgets, provenance, artifacts, and stop conditions. Use before model work whose results must support a decision.4---56# Design Model Experiment78## Outcome910Produce an experiment manifest that another operator can run, audit, and compare without reconstructing unstated choices.1112## Workflow13141. Write one falsifiable hypothesis and the decision it informs.152. Define the experimental unit: model revision, adapter, prompt set, intervention, runtime, or harness.163. Name exactly one primary controlled variable per comparison. Record every intentional difference.174. Select a baseline and explain why it is a fair comparator.185. Pin model, tokenizer, dataset, code, dependency, template, evaluator, and seed provenance.196. Define primary metrics, guardrail metrics, uncertainty treatment, and failure thresholds before the run.207. Estimate compute, storage, time, and paid cost. Set smoke-test and full-run stop conditions.218. Define raw and derived artifacts, retention, and sensitive-data handling.229. Copy `assets/experiment-manifest.json`, fill it, then run:2324```bash25dotnet fsi scripts/validate-experiment-manifest.fsx path/to/experiment.json26```272810. Run the smallest experiment capable of detecting configuration or pipeline failure before spending the full budget.2930## Evidence Contract3132Keep configuration validation, smoke-run evidence, and final experimental evidence separate. A successful process exit proves execution, not model quality. Report deviations from the manifest before interpreting results.3334## Resources3536- `assets/experiment-manifest.json`: managed experiment template.37- `references/experiment-design.md`: field semantics and comparison rules.38- `scripts/validate-experiment-manifest.fsx`: deterministic structural validation.
gaelic-ghost