Skill Continuous Improvement: RED-GREEN-REFACTOR
Adapts the RED-GREEN-REFACTOR cycle from software testing to skill authoring.
The key insight: a skill is a testable contract. The failure to follow the contract
is observable. Always observe the failure BEFORE writing the fix.
Integrated with:
os-eval-runner -- runs eval_runner.py as the GREEN verification stepos-improvement-loop -- uses this methodology to gate every proposed skill patchevals/evals.json + results.tsv -- autoresearch eval format for longitudinal tracking
The TDD Mapping
| Software TDD |
Skill Authoring Equivalent |
| Test case |
Pressure scenario: a user prompt that should trigger the skill |
| RED phase |
Run a baseline WITHOUT the skill. Observe: does the agent violate the intended protocol? |
| GREEN phase |
Write the skill. Run os-eval-runner. KEEP only if score >= baseline. |
| REFACTOR phase |
Identify loopholes from eval failures. Patch frontmatter or examples. Re-eval. |
Iron Law: Run a RED Scenario BEFORE Writing
Never write a new skill without first observing a failure.
The RED scenario is the evidence that the skill is needed. Without it you cannot know the
specific failure being fixed, cannot do a before/after comparison, and examples become
generic rather than addressing real failure modes. Full steps for running a RED scenario are
in references/detailed-reference.md.
Required before any mutation
- Skill State Snapshot — run
eval_runner.py --skill <experiment-dir> --snapshot to see
current score, iteration history, and whether the dominant problem is PRECISION (too many
false positives — don't add more keywords) or RECALL (don't add adversarial examples without
also adding trigger phrases). Full detail in references/detailed-reference.md.
- Hypothesis Block — output failure mode, root cause, change, expected effect, and named
regression risk before editing any file. Format and worked example in
references/detailed-reference.md. Vague hypotheses ("description too vague") are not
acceptable — they produce random mutations and early plateau.
Phase 1: Frontier (What failure does this skill fix?)
Before writing a single line of SKILL.md:
- Define the pressure scenario (one concrete user request that should trigger this skill).
- Define the failure the agent exhibits WITHOUT the skill.
- Define the acceptance criterion — what specific behavior proves the skill is working.
- Check the test registry (
context/memory/tests/registry.md) for prior falsified hypotheses.
- Add a row to the test registry as IN PROGRESS before writing any SKILL.md content.
Phase 2: GREEN -- Write the Skill
Write the frontmatter (name, description as the primary routing signal, trigger phrases,
allowed-tools), at least two <example> blocks (standard use + adversarial non-trigger), and
the body (When to Use, Iron Law, Step-by-Step Protocol, Common Failures table, References).
Exact templates and anti-patterns are in references/detailed-reference.md.
Phase 3: GREEN Verification -- os-eval-runner
After writing the SKILL.md, run the eval gate via eval_runner.py --skill path/to/new/SKILL.md.
Do not apply the skill without a KEEP verdict. KEEP (score >= baseline) applies the skill;
BASELINE (first run) records the score and requires an eval scenario before shipping;
DISCARD (same or lower score) goes to Phase 4.
Phase 4: REFACTOR & Loop Integration
If eval returns DISCARD: identify the loophole input, add an <example> block, sharpen trigger
descriptions, and re-eval until KEEP. When os-improvement-loop proposes patches, it MUST
generate a RED scenario, verify it fails without the skill, apply the patch, and verify via
os-eval-runner. Proposals skipping the RED scenario must be rejected. Anti-patterns in
references/detailed-reference.md.
References
- os-eval-runner -- eval_runner.py, KEEP/DISCARD logic
- skill_optimization_guide.md -- routing accuracy patterns
- test-registry-protocol.md -- test scenario documentation
- Skill Types Reference and Directory Structure policy are in
references/detailed-reference.md.
1---2name: os-skill-improvement3version: 1.0.04description: >5 Continuously improves an existing agent skill based on eval results using the6 RED-GREEN-REFACTOR cycle. Apply when a skill's routing accuracy is low, trigger7 descriptions need sharpening, or os-eval-runner scores are below target.8 (1) run a RED baseline to observe the failure mode,9 (2) apply a focused patch and verify with os-eval-runner (GREEN),10 (3) refactor to close loopholes until score meets threshold.11 Integrates with os-eval-runner as the objective eval gate.12 NOT for scaffolding new skills — use create-skill (agent-scaffolders) for that.13trigger: improve a skill, improve skill routing, fix routing accuracy, skill is not triggering,14 skill triggers too often, improve trigger description, update a skill trigger, skill patch,15 improve triggers, route a skill, routing precision, fix skill description, skill scoring low,16 eval score low, skill improvement, continuous skill improvement, refactor skill triggers,17 tdd for documentation, skill not routing correctly18allowed-tools: Read, Write, Edit, Bash19---2021# Skill Continuous Improvement: RED-GREEN-REFACTOR2223Adapts the RED-GREEN-REFACTOR cycle from software testing to skill authoring.24The key insight: a skill is a testable contract. The failure to follow the contract25is observable. Always observe the failure BEFORE writing the fix.2627**Integrated with**:28- `os-eval-runner` -- runs `eval_runner.py` as the GREEN verification step29- `os-improvement-loop` -- uses this methodology to gate every proposed skill patch30- `evals/evals.json` + `results.tsv` -- autoresearch eval format for longitudinal tracking3132## The TDD Mapping3334| Software TDD | Skill Authoring Equivalent |35|---|---|36| Test case | Pressure scenario: a user prompt that should trigger the skill |37| RED phase | Run a baseline WITHOUT the skill. Observe: does the agent violate the intended protocol? |38| GREEN phase | Write the skill. Run `os-eval-runner`. KEEP only if score >= baseline. |39| REFACTOR phase | Identify loopholes from eval failures. Patch frontmatter or examples. Re-eval. |4041## Iron Law: Run a RED Scenario BEFORE Writing4243**Never write a new skill without first observing a failure.**4445The RED scenario is the evidence that the skill is needed. Without it you cannot know the46specific failure being fixed, cannot do a before/after comparison, and examples become47generic rather than addressing real failure modes. Full steps for running a RED scenario are48in `references/detailed-reference.md`.4950## Required before any mutation51521. **Skill State Snapshot** — run `eval_runner.py --skill <experiment-dir> --snapshot` to see53 current score, iteration history, and whether the dominant problem is PRECISION (too many54 false positives — don't add more keywords) or RECALL (don't add adversarial examples without55 also adding trigger phrases). Full detail in `references/detailed-reference.md`.562. **Hypothesis Block** — output failure mode, root cause, change, expected effect, and named57 regression risk before editing any file. Format and worked example in58 `references/detailed-reference.md`. Vague hypotheses ("description too vague") are not59 acceptable — they produce random mutations and early plateau.6061## Phase 1: Frontier (What failure does this skill fix?)6263Before writing a single line of SKILL.md:64651. Define the pressure scenario (one concrete user request that should trigger this skill).662. Define the failure the agent exhibits WITHOUT the skill.673. Define the acceptance criterion — what specific behavior proves the skill is working.684. Check the test registry (`context/memory/tests/registry.md`) for prior falsified hypotheses.695. Add a row to the test registry as IN PROGRESS before writing any SKILL.md content.7071## Phase 2: GREEN -- Write the Skill7273Write the frontmatter (name, description as the primary routing signal, trigger phrases,74allowed-tools), at least two `<example>` blocks (standard use + adversarial non-trigger), and75the body (When to Use, Iron Law, Step-by-Step Protocol, Common Failures table, References).76Exact templates and anti-patterns are in `references/detailed-reference.md`.7778## Phase 3: GREEN Verification -- os-eval-runner7980After writing the SKILL.md, run the eval gate via `eval_runner.py --skill path/to/new/SKILL.md`.81**Do not apply the skill without a KEEP verdict.** `KEEP` (score >= baseline) applies the skill;82`BASELINE` (first run) records the score and requires an eval scenario before shipping;83`DISCARD` (same or lower score) goes to Phase 4.8485## Phase 4: REFACTOR & Loop Integration8687If eval returns DISCARD: identify the loophole input, add an `<example>` block, sharpen trigger88descriptions, and re-eval until KEEP. When `os-improvement-loop` proposes patches, it MUST89generate a RED scenario, verify it fails without the skill, apply the patch, and verify via90`os-eval-runner`. Proposals skipping the RED scenario must be rejected. Anti-patterns in91`references/detailed-reference.md`.9293## References9495- [os-eval-runner](../os-eval-runner/SKILL.md) -- eval_runner.py, KEEP/DISCARD logic96- [skill_optimization_guide.md](references/operations/skill_optimization_guide.md) -- routing accuracy patterns97- [test-registry-protocol.md](references/testing/test-registry-protocol.md) -- test scenario documentation98- Skill Types Reference and Directory Structure policy are in `references/detailed-reference.md`.
richfrem