Skill Refiner

Skill Refiner: Iterative Self-Improvement Loop

Adaptive evaluation loop for AI skill collections, inspired by Karpathy's AutoResearch. Orchestrates repeated score-improve-verify cycles using skill-creator as the engine and mandatory peer review as an adversarial check (cross-model when distinct model identity is verified, fresh-context self-review as the minimum fallback).

When to use

• Batch-improving the entire skill collection after a period of manual edits
• Targeted improvement of one named skill when the user explicitly asks for skill-refiner, multiple iterations, a score target, or "no ceiling" polish
• Running quality sweeps before a release or publish
• Triggering a self-improvement cycle where skills bootstrap each other
• After adding several new skills that need polish and consistency alignment
• When cross-model perspective would catch single-model blind spots
• Periodic maintenance: scheduled improvement runs to keep skills current

When NOT to use

• Creating a new skill from scratch - use skill-creator (Mode 1)
• Single-pass review of one skill without iteration, scoring, or peer review - use skill-creator (Mode 2)
• One-off collection audit without iteration - use skill-creator (Mode 3)
• Full codebase review (code, not skills) - use full-review
• Style/slop audit on application code - use anti-slop

Configuration

skill-refiner [--iterations N] [--mode MODE] [--secondary HARNESS] [--threshold N] [--plateau N]

Environment override: SKILL_REFINER_SECONDARY=<harness> (CLI flag takes precedence)

Checkpoint Modes

circuit-breaker (default): runs autonomously, auto-pauses on score regression, contested major flags, or plateau. Always pauses before phase 2.

auto: fully autonomous through phase 1. Still pauses before phase 2 and on contested major flags (non-configurable).

step: pauses after every iteration for manual review. Best for first run or learning.

Performance

• Batch similar edits and validations to reduce repeated full-collection scans.
• Prioritize low-scoring or recently changed skills before polishing already-healthy ones.
• Use focused diffs and line-count checks after each batch to avoid late cleanup churn.

Best Practices

• Snapshot evaluation criteria before editing the skills that define the criteria.
• Revert changes that add complexity without improving behavior.
• Keep run history factual and free of unverifiable score inflation.
• Deduct behavioral points only for a named failed quality signal or verified defect. Do not reserve points merely because a case was simulated or a live runtime was unavailable.
• Treat composite deltas as noisy when different scorer instances run across iterations: a few points of swing is judge variance, not real change. Anchor keep/revert decisions on the structural gate, on whether the specific targeted weakness was fixed, and on peer-review NO_FLAGS - not on small composite moves. Use a consistent scoring approach within a single before/after comparison.
• Leave externally-maintained version, CVE, and EOL pins out of scope. When a collection has a freshness routine (or equivalent) that owns version currency, do not edit those pins during a run and do not score them as "unverifiable" failures; flag only internal contradictions.
• Audit offensive or security skills (privilege escalation, exploit research) in-loop rather than through web-researching subagents, which can trip platform safeguards. Keep their scoring and edits in the main session.
• Treat cross-harness review as a data export. Before sending private or sensitive repository content to another provider, require explicit user authorization for that destination; a request to run skill-refiner alone is not authorization. Otherwise use the fresh local-reviewer fallback and record why.

Workflow

Phase 0: Setup

1. Create feature branch: skill-refiner/YYYY-MM-DD-HHMMSS from current HEAD. Preserve dirty worktrees; branching isolates the run, it does not imply cleanup. If already on a run branch for this sweep, record it instead of nesting another branch. Do not mix unrelated dirty files into refiner commits; isolate them with a path-limited stash only when authorized.
2. Load run history: read .refiner-runs.json from the repository root, beside .refiner-ledger.md (if it exists). Repository root, not the skills/ directory: a second history file in skills/ splits the log and hides prior baselines. Use previous run data for: baseline score comparison (detect regressions from external changes), model/harness change detection (flag if the primary or secondary model changed since last run - new model = new baseline, not a comparable delta), and skip analysis (don't re-attempt improvements that were already tried and reverted in a recent run).
3. Build skill inventory: list all skills, exclude phase-2 targets (skill-creator, skill-refiner) from the improvement pool
4. Record evaluator identity: capture actual provider, resolved model, effective effort, harness and version for primary and reviewer evaluations, with redacted runtime/config evidence per references/harness-detection.md. Record unavailable fields as unknown; requested flags and skill effort metadata do not prove effective runtime settings.
5. Probe for secondary harness: run three-step validation (PATH check, config check, smoke test) per references/harness-detection.md. Announce result. Before sending a review payload, classify the source as public, private, or sensitive and verify that the user authorized sharing it with that harness/provider.
6. If no authorized secondary is available: always fall back to self-review. Spawn a fresh agent on the current harness with the review prompt template from references/harness-detection.md. Label as "same-model fresh-context review" only when identity is verified; otherwise use "unknown-model fresh-context review". Both weight at 3% instead of 5% (composite becomes gate/40/55/3, renormalize the missing 2% proportionally to AI Self-Check and Behavioral). Only a verified distinct model receives 5%, including on the same harness. Different harnesses alone do not establish model diversity. Skipping review entirely is not an option - a fresh-context self-review is the minimum bar. If the harness doesn't support subagents, run the review prompt as a separate CLI invocation (claude -p, codex exec, gemini -p, etc.).

Phase 1: Regular Iterations

7. Iteration 1 - full sweep: score every skill in the pool using the four-component model from references/evaluation-criteria.md
   • Structural: run lint-skills.sh + validate-spec.sh
   • AI Self-Check: invoke skill-creator review mode on each skill
   • Behavioral: run test prompts from references/test-cases.md. For skills without pre-written test cases, auto-generate 2-3 test prompts from the skill's "When to use" section and quality signals from its AI Self-Check. Log a warning that generated tests are lower quality than hand-written ones. Optionally save generated tests to a references/test-cases-local.md file alongside references/test-cases.md so they accumulate across runs.
   • Cross-model: skip on first iteration (no diff to review yet)
8. Log baseline scores: record per-skill and aggregate scores in a score ledger before any edits. The ledger must include structural gate (G), AI Self-Check (A), behavioral score (B), cross-model review (X), composite score, test source, evaluator identity and evidence for each evaluation, reviewer classification and applied weight, and timestamp. After this step, if the ledger is missing, incomplete, or only records lint/spec status, pause and backfill scoring before applying changes. In headless mode, halt the run and report the missing score data.
9. Iteration 2+: enter adaptive focus mode. For a user-requested single-skill run, treat that skill as the whole phase-1 pool, run at least the requested iteration count, and keep iterating until the explicit score target is reached, quality plateaus, or a circuit breaker fires.
10. Select targets: identify skills scoring below the focus threshold
11. For each targeted skill, run the improvement cycle: a. Read current SKILL.md and all reference files b. Invoke skill-creator review mode - collect findings c. Run behavioral test - score current output quality d. Propose targeted improvements based on findings (not random changes) e. Apply changes to SKILL.md (and references if needed) f. Re-score structural, AI Self-Check, and behavioral components; keep the change provisional g. Send the minimum necessary diff to an authorized peer reviewer or the fresh local fallback h. Process flags per references/harness-detection.md verification protocol i. If secondary flags major issue and primary agrees: revert j. If secondary flags major issue and primary disagrees: escalate to circuit breaker k. Karpathy gate: compute the final composite including verified peer-review deductions and the applicable 5% or 3% weight. Keep only if it improved; otherwise revert.
12. Commit iteration: one commit with all improvements from this iteration Format: refactor(skill-refiner): iteration N - skill1(+X), skill2(+Y)
13. Log iteration summary:

    --- iteration N / max -------------------------------------------
    improved:  skill1 (72 > 80 | G:pass A:76 B:78 X:90), skill2 (68 > 73 | G:pass A:70 B:72 X:100)
    gated:     skillZ (lint/spec failed - excluded from scoring)
    skipped:   M skills above threshold
    reverted:  skill3 (proposed change scored -2, rolled back | G:pass A:74 B:69 X:100)
    contested: skill4 (secondary flagged major, primary disagreed)
    plateau:   yes/no (max delta: +X)
    -----------------------------------------------------------------
    

    Also append the same data to the score ledger. Keep/reject decisions must point to numeric before/after scores, not reviewer impressions or passing lint/spec checks.
14. Check termination conditions (phase 1 always flows into phase 2 on termination, except on circuit-breaker pauses which wait for user input first):
    • Plateau detected (max delta < plateau threshold)? Terminate phase 1.
    • All skills above focus threshold? Bump threshold by 5 and continue. If threshold is already at max (95) and all skills still clear it, terminate phase 1.
    • Iteration cap reached? Terminate phase 1.
    • Circuit breaker triggered? Pause for user input.
15. Repeat from step 10 until terminated

Phase 2: Meta-Improvement

16. Announce: "Entering phase 2 - meta-improvement. This always requires human review."
17. Snapshot evaluation criteria:
    • Copy skill-creator's AI Self-Check section to a temp location
    • Copy references/evaluation-criteria.md to a temp location
    • Copy skill-creator's conventions.md reference to a temp location These snapshots are the evaluation baseline for phase 2.
18. Improve skill-creator: run the improvement cycle (steps 11a-11k) using the snapshot as the evaluation criteria, not skill-creator's live version
19. Improve skill-refiner: same process, using the snapshot
    • Compare every public skills/*/SKILL.md directory with the canonical ### <skill-name> headings in references/test-cases.md. Exclude the format-template heading.
    • Promote stable generated or local cases into the canonical catalog for every gap, then verify there are no missing, duplicate, or orphan headings. This edit is phase-2-only.
20. Improve lint scripts (lint-skills.sh, validate-spec.sh):
    • Capture baseline: run both scripts, save full output
    • Propose improvements
    • Apply changes
    • Run regression: compare output to baseline
    • If false positives or false negatives introduced: revert
    • If clean: keep
21. Commit phase 2: one commit per target Format: refactor(skill-refiner): meta - improve <target> (+N)
22. Pause for human review: display phase 2 changes, wait for approval. This checkpoint is non-configurable - it fires even in --mode auto. A direct user approval such as "continue" or "proceed" counts as approval to resume.

Phase 3: Summary

23. Final report: write a human-readable report first, then machine-readable run history. Include branch, pool, config, every changed skill, score before/after, delta, files changed, verification commands, peer-review flags, reverted changes, private-skill handling, and skipped checks. If scoring was reconstructed after the fact, label it retroactive and state which components were not captured during the live loop. Do not output only JSON.

    === skill-refiner run complete ===================================
    Branch:     skill-refiner/YYYY-MM-DD-HHMMSS
    Primary:    <harness> <version> (<provider>/<resolved model>, effective effort: <level>)
    Secondary:  <same identity fields> | none (baseline only)
    Evidence:   <redacted runtime/config references; unknown fields and reasons>
    Review:     <verified cross-model | same-model | unknown-model>, weight: <5% | 3%>
                <baseline: no diff, review omitted>
    Pool:       N skills (skill-creator, skill-refiner excluded)
    Config:     iterations=M, threshold=T, mode=MODE, plateau=P
    
    Iterations: N (of max M)
    Terminated: plateau / threshold / cap / user
    
    Score changes:
      skill1:  62 > 88 (+26)  [G:pass A:84 B:86 X:90]
      skill2:  71 > 85 (+14)  [G:pass A:82 B:79 X:100]
      ...
      skill-creator: 80 > 84 (+4)  [G:pass A:82 B:81 X:100] [meta]
      skill-refiner: 78 > 83 (+5)  [G:pass A:80 B:79 X:100] [meta]
    
    Aggregate:  avg X.X | min X.X | max X.X
    Reverted:   X changes across Y iterations
    Contested:  Z flags escalated to human
    =================================================================
    

24. Write run history: append this run's metadata to .refiner-runs.json at the repository root, the same file read in Phase 0 step 2. Include: run_id, branch, date, primary/secondary provider+resolved model+effective effort+harness+version and redacted runtime/config evidence per evaluation, reviewer classification and applied weight, config, pool size, termination reason, peer-review flag counts, before/after per-skill scores (component breakdown + composite, or clearly labeled estimates if the run used a targeted manual rubric instead of the full automated sweep), and a changes summary. When updating an existing history file, append the new object without reserializing the whole file; do not normalize or rewrite old entries just because a JSON writer changes escaping, commas, or whitespace. Commit with the phase 3 summary.
25. Announce branch: remind user to review and merge when ready

AI Self-Check

Before committing any skill modification, verify:

• Lint passes: lint-skills.sh exits 0 for the modified skill
• Spec valid: validate-spec.sh exits 0 for the modified skill
• Score improved: composite score is strictly higher than before the change
• No content regression: change does not remove critical sections, warnings, or cross-references without replacement
• Simplicity maintained: change does not add unnecessary complexity for marginal gains
• Cross-references intact: all skill names in bold still resolve to existing skills
• Target ~500 lines: modified SKILL.md stays near 500 lines. Hard max 600
• ASCII only: no non-ASCII characters introduced (except allowed emoji indicators)
• Immutability respected: no phase-1 modification to evaluation criteria, test cases, lint scripts, skill-creator, or skill-refiner
• Current source checked: dated versions, CLI flags, API names, and support windows are verified against primary docs before repeating them
• Hidden state identified: local config, credentials, caches, contexts, branches, cluster targets, or previous runs are made explicit before acting
• Verification is real: final checks exercise the actual runtime, parser, service, or integration point instead of only linting prose or happy paths
• Score discipline kept: changes are kept only when they improve measured quality or fix a verified defect
• Reviewer identity verified: weight is 5% only for verified distinct models; same or unknown model identity uses fresh context at 3%, with runtime/config evidence recorded
• Score ledger present: baseline, iteration, and final component scores exist before reporting completion
• Canonical test coverage complete: phase 2 compares public skill directories with the canonical test headings and leaves no missing, duplicate, or orphan skill section
• Local-only scope respected: public and private skills are separated before commits or release notes
• Review export authorized: private or sensitive source is sent to another harness/provider only with explicit user authorization; otherwise the fresh local fallback is used

Output Contract

See references/output-contract.md for the full contract.

• Skill name: SKILL-REFINER
• Deliverable bucket: audits
• Mode: conditional. When invoked to analyze, review, audit, or improve existing repo content outside the refiner workflow, emit the full contract - monospace inline header, severity-grouped inline summary, linked Markdown deliverable, and concise monospace conclusion - and write the deliverable to docs/local/audits/skill-refiner/<YYYY-MM-DD>-<slug>.md. When invoked to run the refiner workflow (its primary mode), use the existing Phase 3 "Final report" format described in the workflow; that build-mode output is unchanged by this contract.
• Severity scale: P0 | P1 | P2 | P3 | info (see shared contract; only used in audit/review mode).

Related Skills

• skill-creator - the evaluation and improvement engine. skill-refiner invokes skill-creator's review mode (Mode 2) for scoring and its improve mode for generating changes. skill-creator handles individual skill quality; skill-refiner handles iteration, prioritization, and orchestration. Primary dependency.
• full-review - one-off collection audit across code-review, anti-slop, security-audit, and update-docs. Use full-review for a single pass over application code; use skill-refiner for iterative improvement of skill files.
• anti-slop - code quality patterns. skill-refiner may invoke anti-slop principles through skill-creator during improvement, but does not call anti-slop directly. Different domain: anti-slop audits application code, skill-refiner audits skill files.

Rules

1. Immutability in phase 1: never modify references/evaluation-criteria.md, references/test-cases.md, lint-skills.sh, validate-spec.sh, skill-creator, or skill-refiner during phase 1. Violation = abort the run.
2. Karpathy gate: only directional improvements survive. If a change does not improve the composite score, revert it. No exceptions, no "it looks better."
3. Verify flags: never take cross-model flags at face value. Primary reviews every flag independently. Disagreements on major flags go to human.
4. Snapshot before meta: always snapshot evaluation criteria before phase 2. Evaluate against the snapshot, never the live version being modified.
5. Phase 2 always pauses: even in --mode auto. Non-configurable.
6. Contested major flags always pause: even in --mode auto. Non-configurable.
7. Simplicity criterion: all else being equal, simpler is better. Deletions that maintain score are preferred over additions that marginally improve it.
8. One commit per iteration: bundle improvements, include score deltas in message.
9. Branch isolation: all work on a feature branch. Never modify main directly.
10. Human-readable report required: every run ends with a report that names changes, before/after scores, verification, peer-review flags, skipped checks, and next action.
11. Read before edit: always read the full skill before proposing changes. Never edit from memory or assumption.
12. No score laundering: do not call a run scored unless component scores were recorded. Retroactive scoring is allowed only when clearly labeled.
13. No unapproved review export: do not send private or sensitive repository content to a secondary harness/provider without explicit user authorization for that destination.