STANDARD OPERATING PROCEDURE
Purpose
- Develop, refine, and validate novel methods anchored to baselines and constraints.
- Apply constraint hygiene and explicit ceilings to prevent premature claims.
- Keep structure-first artifacts current for handoff and reproducibility.
Trigger Conditions
- Positive: creating or adapting algorithms/pipelines; designing ablations; exploring new research ideas.
- Negative: pure replication (use
baseline-replication) or publication packaging (research-publication).
Guardrails
- HARD / SOFT / INFERRED constraint buckets (compute, data, metrics, ethics) with sources.
- Two-pass refinement on designs: structure vs. baselines, then epistemic/risks.
- Require baseline parity before claiming improvements; document variance sources.
- Confidence ceilings enforced per claim.
Inputs
- Problem statement and success metrics.
- Baselines to beat and constraints (data, compute, deadlines).
- Risk tolerances and evaluation protocols.
Workflow
- Problem Framing: Capture objectives, constraints, and baselines; confirm INFERRED assumptions.
- Design Options: Propose candidates with expected impact; map to constraints.
- Experiment Plan: Define ablations, datasets, metrics, and stopping rules.
- Run & Observe: Execute experiments, log configs/seeds; compare to baselines.
- Validate & Iterate: Analyze results, run adversarial checks, and refine or stop.
- Package: Summarize findings, risks, and next steps; store artifacts and update references/examples.
Validation & Quality Gates
- Baseline beat or variance explained; claims tied to evidence with ceilings.
- Ablations cover key hypotheses; failures documented.
- Reproducibility assets stored (configs, logs, seeds).
Response Template
**Objective & Constraints**
- HARD / SOFT / INFERRED.
**Design Candidates**
- Option → rationale → expected impact.
**Experiment Status**
- Runs, metrics vs. baseline, issues.
**Next Steps**
- Iterate, stop, or expand.
Confidence: 0.80 (ceiling: research 0.85) - based on current evidence and validation checks.
Confidence: 0.80 (ceiling: research 0.85) - reflects validated comparisons to baselines and logged experiments.
1---2name: method-development3description: Design and iterate on new research methods with structured checkpoints, baselines, and validation.4---5
6
7
8## STANDARD OPERATING PROCEDURE
9
10### Purpose
11- Develop, refine, and validate novel methods anchored to baselines and constraints.
12- Apply constraint hygiene and explicit ceilings to prevent premature claims.
13- Keep structure-first artifacts current for handoff and reproducibility.
14
15### Trigger Conditions
16- **Positive:** creating or adapting algorithms/pipelines; designing ablations; exploring new research ideas.
17- **Negative:** pure replication (use `baseline-replication`) or publication packaging (`research-publication`).
18
19### Guardrails
20- HARD / SOFT / INFERRED constraint buckets (compute, data, metrics, ethics) with sources.
21- Two-pass refinement on designs: structure vs. baselines, then epistemic/risks.
22- Require baseline parity before claiming improvements; document variance sources.
23- Confidence ceilings enforced per claim.
24
25### Inputs
26- Problem statement and success metrics.
27- Baselines to beat and constraints (data, compute, deadlines).
28- Risk tolerances and evaluation protocols.
29
30### Workflow
311. **Problem Framing**: Capture objectives, constraints, and baselines; confirm INFERRED assumptions.
322. **Design Options**: Propose candidates with expected impact; map to constraints.
333. **Experiment Plan**: Define ablations, datasets, metrics, and stopping rules.
344. **Run & Observe**: Execute experiments, log configs/seeds; compare to baselines.
355. **Validate & Iterate**: Analyze results, run adversarial checks, and refine or stop.
366. **Package**: Summarize findings, risks, and next steps; store artifacts and update references/examples.
37
38### Validation & Quality Gates
39- Baseline beat or variance explained; claims tied to evidence with ceilings.
40- Ablations cover key hypotheses; failures documented.
41- Reproducibility assets stored (configs, logs, seeds).
42
43### Response Template
44```
45**Objective & Constraints**
46- HARD / SOFT / INFERRED.
47
48**Design Candidates**
49- Option → rationale → expected impact.
50
51**Experiment Status**
52- Runs, metrics vs. baseline, issues.
53
54**Next Steps**
55- Iterate, stop, or expand.
56
57Confidence: 0.80 (ceiling: research 0.85) - based on current evidence and validation checks.
58```
59
60Confidence: 0.80 (ceiling: research 0.85) - reflects validated comparisons to baselines and logged experiments.
majiayu000