richfrem Skill Continuous Improvement: RED-GREEN-REFACTOR
Adapts the RED-GREEN-REFACTOR cycle from software testing to skill authoring. The key insight: a skill is a testable contract. The failure to follow the contract is observable. Always observe the failure BEFORE writing the fix.
Integrated with:
os-eval-runner-- runseval_runner.pyas the GREEN verification stepos-improvement-loop-- uses this methodology to gate every proposed skill patchevals/evals.json+results.tsv-- autoresearch eval format for longitudinal tracking
The TDD Mapping
| Software TDD | Skill Authoring Equivalent |
|---|---|
| Test case | Pressure scenario: a user prompt that should trigger the skill |
| RED phase | Run a baseline WITHOUT the skill. Observe: does the agent violate the intended protocol? |
| GREEN phase | Write the skill. Run os-eval-runner. KEEP only if score >= baseline. |
| REFACTOR phase | Identify loopholes from eval failures. Patch frontmatter or examples. Re-eval. |
Iron Law: Run a RED Scenario BEFORE Writing
Never write a new skill without first observing a failure.
The RED scenario is the evidence that the skill is needed. Without it you cannot know the
specific failure being fixed, cannot do a before/after comparison, and examples become
generic rather than addressing real failure modes. Full steps for running a RED scenario are
in references/detailed-reference.md.
Required before any mutation
- Skill State Snapshot — run
eval_runner.py --skill <experiment-dir> --snapshotto see current score, iteration history, and whether the dominant problem is PRECISION (too many false positives — don't add more keywords) or RECALL (don't add adversarial examples without also adding trigger phrases). Full detail inreferences/detailed-reference.md. - Hypothesis Block — output failure mode, root cause, change, expected effect, and named
regression risk before editing any file. Format and worked example in
references/detailed-reference.md. Vague hypotheses ("description too vague") are not acceptable — they produce random mutations and early plateau.
Phase 1: Frontier (What failure does this skill fix?)
Before writing a single line of SKILL.md:
- Define the pressure scenario (one concrete user request that should trigger this skill).
- Define the failure the agent exhibits WITHOUT the skill.
- Define the acceptance criterion — what specific behavior proves the skill is working.
- Check the test registry (
context/memory/tests/registry.md) for prior falsified hypotheses. - Add a row to the test registry as IN PROGRESS before writing any SKILL.md content.
Phase 2: GREEN -- Write the Skill
Write the frontmatter (name, description as the primary routing signal, trigger phrases,
allowed-tools), at least two <example> blocks (standard use + adversarial non-trigger), and
the body (When to Use, Iron Law, Step-by-Step Protocol, Common Failures table, References).
Exact templates and anti-patterns are in references/detailed-reference.md.
Phase 3: GREEN Verification -- os-eval-runner
After writing the SKILL.md, run the eval gate via eval_runner.py --skill path/to/new/SKILL.md.
Do not apply the skill without a KEEP verdict. KEEP (score >= baseline) applies the skill;
BASELINE (first run) records the score and requires an eval scenario before shipping;
DISCARD (same or lower score) goes to Phase 4.
Phase 4: REFACTOR & Loop Integration
If eval returns DISCARD: identify the loophole input, add an <example> block, sharpen trigger
descriptions, and re-eval until KEEP. When os-improvement-loop proposes patches, it MUST
generate a RED scenario, verify it fails without the skill, apply the patch, and verify via
os-eval-runner. Proposals skipping the RED scenario must be rejected. Anti-patterns in
references/detailed-reference.md.
References
- os-eval-runner -- eval_runner.py, KEEP/DISCARD logic
- skill_optimization_guide.md -- routing accuracy patterns
- test-registry-protocol.md -- test scenario documentation
- Skill Types Reference and Directory Structure policy are in
references/detailed-reference.md.