jhhuh Experiment Set Design
Core Principle
Compliance doesn't prove quality. Quality requires blind comparison against baseline.
The Baseline Requirement
NON-NEGOTIABLE: Every experiment needs a baseline -- same prompt, same task, no skill loaded. No baseline = invalid experiment.
Task Selection Checklist
Each task must satisfy ALL of:
- Skill's guidance should produce different behavior on this task
- Complex enough that architecture decisions matter (not a one-liner)
- Multiple valid approaches exist (skill's approach is distinguishable)
- Covers a different domain or language than other tasks in the set
Aim for 3-5 tasks per phase. Start small, add tasks when overfitting is suspected.
Three-Phase Progression
Phase 1: COMPLIANCE Phase 2: STRESS Phase 3: QUALITY
"Does it follow?" "Does it follow under "Does following help?"
pressure?"
PASS/FAIL per rule PASS/FAIL per rule Blind review (use
+ competing instructions blind-skill-assessment)
+ edge cases Winner per dimension
| | |
v v v
All PASS?─── no ──> Fix skill, re-run phase
|
yes
|
v
Advance to next phase
Phase 1 -- Compliance. Check each rule. Binary PASS/FAIL.
Phase 2 -- Stress. Same checks + competing instructions ("just write the whole thing"), time pressure, conflicting edge cases.
Phase 3 -- Quality. Run with and without skill. Blind-assess using blind-skill-assessment. If skill wins on aggregate, advance. If baseline wins consistently, the skill is counterproductive — rethink or abandon it rather than forcing further iterations.
Advancement: All PASS before advancing. Any FAIL: improve skill, re-run current phase.
Task Set Diversity
Vary to prevent overfitting:
- Domain: CLI tool, web handler, data pipeline, algorithm
- Complexity: Single function, multi-module, concurrent
- Language: At least 2 if the skill is language-agnostic
After 2+ improvement cycles on the same task set, add new tasks.
Revision Tracking
For each run, record: skill version (git hash), tasks run, phase, results, what changed since last run.
Example: HDD Skill Phases
| Phase | Tasks | Method | Result |
|---|---|---|---|
| 1: Compliance | 5 tasks (Python, Haskell, Go) | Transcript check: holes visible? One-at-a-time? | 4/5 PASS, 1 FAIL |
| 1b: Re-run | Same 5 after skill edit | Same checks | 5/5 PASS |
| 2: Stress | 5 tasks + competing instructions | Same checks under pressure | 5/5 PASS |
| 3: Quality | 5 tasks, A vs B (blinded) | Blind 3-persona review | A won 4/5 → decode: A=skill |
Red Flags -- STOP
- Designing experiments with no baseline comparison
- All tasks at the same difficulty or in the same domain
- Jumping to Phase 3 without passing Phase 1
- Same task set for 3+ improvement cycles without adding tasks
- No record of which skill version produced which results
If you catch yourself doing any of these: STOP. Add baselines. Vary your tasks. Track your versions.