SkillOpt
A candidate lands only when it adds at least one held-out pass and loses
none. No scored eval set means no gate, and no gate means no edit.
SkillOpt treats a skill document as the trainable state of a frozen model.
You run rollouts and reflect on failures. The bundled scripts decide whether
an edit may exist: the edit budget, the patch applier, the protected-section
guard, the rejected-edit buffer, and the acceptance gate are deterministic.
Do not edit the target by hand. Route every edit through the scripts so the
gate is real. Method: Yang et al., arXiv:2605.23904. The paper's reported
gains are its claims; this skill does not reproduce them.
Triggers
| Phrase |
Action |
optimize this skill |
Run the Process below against a scored eval set |
make this skill better |
Same, after the Preconditions pass |
compress these skills safely |
Declare a resource task up front, then gate |
tune this prompt against an eval |
Same loop with a prompt as the target |
| A skill keeps failing the same way |
Mine sessions, confirm the defect, then gate |
Boundaries
| Need |
Owner |
| Create a skill, find one, or route a request to one |
skillforge |
| Decide skill vs passive context, or compress markdown with no eval |
context-optimizer |
| Measured improvement of an existing skill, agent file, or prompt |
this skill |
Held-out optimization of agents, rules, and hooks inside the rjmurillo/ai-agents checkout |
optimize-artifact.py in that repository's eval tooling (not shipped) |
Preconditions (hard stop if unmet)
- Target artifact. A
SKILL.md, agent file, or prompt.
- Scored eval set. A nonempty held-out
sel split with per-task pass or
fail verifiers (references/eval-set-format.md). Without one, stop and say
so. score and pair-gate exit 2 on a missing or empty split.
- Canonical owner. Name the versioned source, its generated mirrors, and
the live install path before any candidate runs. A file in an eval
workspace is not deployed behavior.
- Two isolated runners. The candidate author and the
sel runner must
be separate: a fresh subagent or another session. The sel runner writes
and holds the full eval.json. The author gets a train-only copy that
keeps the sel task ids but drops the sel task bodies; apply reads
only those ids. In every step the author must not write, run, or see the
sel prompts, verifiers, or per-task results. Only the gate verdict goes
back. If the harness cannot provide both, do not run skillopt. Report that the gate cannot be
independent. Replacement holdouts written by the same agent are still
exposed.
Freeze the train and sel task lists before authoring any candidate. Keep
sel prompts, verifiers, and outcomes away from the candidate author for the
whole run (Precondition 4). If they were exposed, list them under exposed in
the manifest, move them to train, and add untouched holdout tasks written
by a runner who has not seen the candidate (Precondition 4).
pair-gate exits 2 while an exposed or invalid task sits in the sel split.
Deterministic tasks taken from a written spec before any candidate exists
are frozen once their list and verifiers stop changing. Freezing is not
isolation. If the agent that wrote them also writes the candidate, it has
seen every verifier, so list them under exposed and keep them in train.
They still catch regressions. They count as sel only when a different
author or session wrote them and kept them from the candidate author.
Read references/target-selection.md before you fund rollouts for a target,
and when a patch fails to apply. It covers confirming the defect, splitting
after baseline, anchor rules, and library-wide passes.
Scripts
Every subcommand takes explicit file paths, so it runs from any working
directory and needs no clone of the source repository. Pure standard library.
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/skillopt/scripts/skillopt.py" --help
python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/skillopt/scripts/mine_sessions.py" --help
SKILLOPT_DIR="${SKILLOPT_DIR:-${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/skillopt/scripts}"
If neither COPILOT_PLUGIN_ROOT nor CLAUDE_PLUGIN_ROOT is set (for example
Codex, or a plain checkout under PowerShell), use the directory that holds
this SKILL.md. Set SKILLOPT_DIR to the scripts folder beside it:
SKILLOPT_DIR="<folder that holds this SKILL.md>/scripts"
python3 "$SKILLOPT_DIR/skillopt.py" --help
On Windows PowerShell, no Bash required:
if (-not $SkillOptDir) {
$Root = if ($env:COPILOT_PLUGIN_ROOT) { $env:COPILOT_PLUGIN_ROOT } else { $env:CLAUDE_PLUGIN_ROOT }
if (-not $Root) { throw 'Set COPILOT_PLUGIN_ROOT or CLAUDE_PLUGIN_ROOT, or set $SkillOptDir to the scripts folder beside this SKILL.md' }
$SkillOptDir = Join-Path $Root 'skills\skillopt\scripts'
}
$SkillOpt = Join-Path $SkillOptDir 'skillopt.py'
py -3 $SkillOpt --help
Below, skillopt.py means that resolved path. References use $SKILLOPT_DIR
($SkillOptDir in PowerShell) for the same folder.
| Command |
Exit codes |
skillopt.py budget, apply, score, buffer-add |
0 success, 2 usage or validation error |
skillopt.py gate, pair-gate |
0 accept, 1 reject, 2 usage or validation error |
skillopt.py efficiency-gate |
0 ACCEPT_EFFICIENCY, 1 reject, 2 usage or validation error |
skillopt.py report |
0 report printed, 2 usage or validation error. Never gates |
skillopt.py run |
0 results written, 1 results written with a task UNVERIFIED, 2 refused or invalid (no results written) |
skillopt.py hash PATH... |
0 digests printed; a directory hashes as one whole bundle |
skillopt.py hash PATH --expect SHA |
0 match, 1 MISMATCH, 2 usage or validation error |
skillopt.py buffer-check |
0 KNOWN-BAD (discard), 1 NOVEL (proceed), 2 usage or validation error |
mine_sessions.py sources, rank, tasks, collect, merge |
0 success, 2 usage or validation error |
buffer-check is inverted on purpose: exit 0 means the edit was already
rejected. Branch on the printed word or the exact code, never on &&.
Modules named _skillopt_*.py and _mine_*.py are internal helpers the
two entry scripts import. Do not call them directly.
Process
One optimization step:
- Baseline. Read
references/rollout-validity.md before the first
rollout. Split the roles first. The candidate author sees only
train tasks and their results. A separate runner, such as a fresh
subagent or another session (Precondition 4), holds the sel prompts,
verifiers, and results in every step, and reports only the verdict back.
Each runner preflights its own tasks: prompt, verifier, inputs, fixtures,
and permitted scope. Run the rollout under exactly the scope the task grants.
A rollout with the wrong artifact or a scope change is invalid, not a pass
or a fail; mark the task "status": "invalid" and replace it. Grade the
complete persisted output against every clause of the verifier. A runner's
self-check or summary is not proof. A clipped or missing answer is
"UNVERIFIED" in the results file, which the gate refuses to count.
Record {"task_id": true|false} to incumbent_results.json and hash the
incumbent: skillopt.py hash target.md. For tasks with an argv command,
skillopt.py run --eval eval.json --bundle DIR --role incumbent --out-dir results/ runs them and writes results/incumbent_results.json. It
replaces the whole file, so run it first and add your grades for tasks with
no command after it finishes. Read the security note in
references/eval-set-format.md first, because eval.json commands run any
binary on PATH. When the target is a skill directory with scripts, pass
the directory to hash and apply; it hashes as one whole bundle. When the
manifest sets
"scorer": "nondeterministic", the sel runner also reruns the
byte-identical incumbent into null_results.json now (see Before
promotion, step 2).
- Budget.
skillopt.py budget --step T --total N prints L_t, the
cosine-decayed cap on atomic patches this step. T is 0-based, from 0 to
N-1; --step N exits 2 with out of range.
- Reflect. Failures become corrective rules; successes become behaviors
to preserve. Write at most L_t patches to
patches.json. Use only train
evidence.
- Buffer check.
skillopt.py buffer-check --buffer rejected.jsonl --patches patches.json. Exit 0 prints KNOWN-BAD: discard and reflect
again. Exit 1 prints NOVEL: proceed.
- Apply to a candidate.
skillopt.py apply --skill target.md --eval eval.json --patches patches.json --budget L_t --out candidate.md. The
incumbent is never written; an --out that resolves to the incumbent exits
- Exit 2 is a usage or validation error, not a gate loss: an empty
sel
split, an incumbent --out, budget overflow, a bad anchor, a protected
fence, or marker text inside a patch.
- Re-roll every task on
candidate.md into candidate_results.json.
For a bundle, apply writes a new directory. Finish every edit to its
scripts first, then run the command tasks, then hash the final
CANDIDATE_DIR. That digest is the candidate's identity, so results from
an earlier state of the bundle would gate a different candidate. The sel
runner does the sel half. For command tasks, run
skillopt.py run --eval eval.json --bundle CANDIDATE_DIR --role candidate --out-dir results/, then add your grades for tasks with no command.
- Gate.
skillopt.py pair-gate --eval eval.json --incumbent-results incumbent_results.json --candidate-results candidate_results.json. If you
used run --out-dir results/, pass the files under results/. For a
nondeterministic scorer, add --null-results null_results.json --incumbent-sha256 <sha> --null-sha256 <sha>; without them the gate exits
- Exit 0,
ACCEPT_PARETO: at least one held-out gain, zero held-out
regressions, every held-out result a literal boolean.
- Verdict precedence:
REJECT_UNVERIFIED first (a missing or non-boolean
result), then REJECT_NULL_CONTROL (the null run moved and no genuine
gain remains), then REJECT_CEILING (the incumbent already passes every
sel task and the candidate regresses none), then ACCEPT_PARETO or
REJECT.
- Exit 1,
REJECT, REJECT_UNVERIFIED, or REJECT_NULL_CONTROL: record
it with skillopt.py buffer-add --buffer rejected.jsonl --patches patches.json --score <cand> --incumbent <best> --note "..." --step T.
- Exit 1,
REJECT_CEILING: the edit lost nothing and is not bad. Do not
buffer it. No candidate can pass a gate that has no headroom, so stop the
loop and read the hint: freeze a new sel set of harder tasks that
sample the failure mode, re-score the incumbent on it, then author a
fresh candidate. Never re-gate this candidate on tasks written after it.
For a first draft whose sel tasks came from its own requirements, hand
back to skillforge.
- Cost, latency, or token claim at the ceiling: when the incumbent passes
every
sel task and the claim is a gain declared in the frozen manifest
before authoring, run skillopt.py efficiency-gate --eval eval.json --incumbent-results incumbent_results.json --candidate-results candidate_results.json --measurements measurements.json. Read
references/eval-set-format.md (Efficiency measurements, Efficiency
gate) first. It never replaces pair-gate for a claim of better behavior.
- Repeat until the budget reaches its floor and several steps reject, or
the step budget runs out.
Use pair-gate as the acceptance decision. Scalar gate --candidate X --incumbent Y (exit 0 ACCEPT, exit 1 REJECT, ties reject) is only for a
single-objective experiment declared before the candidate existed. A resource
objective such as context reduction may be one sel task with
"kind": "resource", declared before candidate creation. It cannot offset a
semantic regression, because pair-gate rejects any regression.
Before promotion
ACCEPT_PARETO is necessary, not sufficient:
- Training review. Run
pair-gate --split train and read its
regressions. A lost training pass on a load-bearing verifier blocks
promotion. If fixing it changes the candidate, freeze a fresh sel set.
- Null control for nondeterministic scorers. It runs at baseline and
feeds the step 7 gate. Set
"scorer": "nondeterministic" in the manifest;
pair-gate on sel then exits 2 without --null-results. Rerun the
byte-identical incumbent (confirm with
skillopt.py hash target.md --expect <sha256>, exit 1 on MISMATCH) into
null_results.json, then pass --null-results null_results.json --incumbent-sha256 <sha> --null-sha256 <sha> to pair-gate, using the
digests hash printed for the incumbent and for the file the null run
used. Both are required, and a mismatch exits 2. A gain the no-op also
produced is noise; if no genuine gain remains the verdict is
REJECT_NULL_CONTROL.
- Evidence. Keep incumbent, candidate, patches, every results file, the
pair report, hashes,
rejected.jsonl, and one line of evidence per task.
skillopt.py report lays out the evidence, benchmark spread, and case
signals (references/eval-set-format.md, Run report). It explains, never gates.
Promotion
- Confirm the canonical source still hashes to the evaluated incumbent.
- Apply the accepted candidate to the canonical source, not to an eval copy
or a generated mirror.
- Hash the canonical source after applying it
(
skillopt.py hash canonical.md --expect <candidate sha256>). It must
equal the accepted candidate's hash before you check the live copy.
- Regenerate mirrors, run the owner's tests, and load the skill once.
- Commit and push through the owner's repository workflow.
- Reload the live skill and confirm its hash matches the canonical file.
If any step stops short, report ACCEPTED_NOT_DEPLOYED. Never call an
eval-workspace candidate installed.
Protected sections
Fence guidance that step edits must never touch:
<!-- SLOW_UPDATE_START -->
Durable guidance changed only by hand, at epoch boundaries.
<!-- SLOW_UPDATE_END -->
Contract: references/slow-section-convention.md.
Finding targets from sessions
mine_sessions.py ranks skills by usage and a weak correction signal from
local transcripts, and pulls the prompts that triggered a skill:
sources lists each supported source and whether it is available.
rank, tasks --skill X, collect --out PATH, merge.
--patterns patterns.json reads a session-patterns/1 handoff instead of
transcripts.
Supported sources: Claude Code transcripts (~/.claude/projects/**/*.jsonl)
and the GitHub Copilot CLI session store (~/.copilot/session-store.db),
both resolved from the user's home directory on Linux, macOS, and Windows.
Any other harness is unsupported; supply a session-patterns/1 file
instead. A missing source is skipped, never an error; sources shows which
ones were found, and a run with no source at all prints a note to stderr. Session
data stays local; keep extracted prompts out of git unless the user names
another destination.
A correction nominates a candidate. It is not a gold label. Confirm the
defect in the target and in two distinct sessions before funding rollouts.
Workflow: references/cross-machine-mining.md.
Description changes
Read references/description-optimization.md before you edit a description
field or claim a skill triggers better. Output-quality rollouts do not measure
activation, so score a description candidate on trigger rate instead.
When a candidate patch adds or rewrites skill prose, read
../skillforge/references/skill-content-rules.md first.
Limits
The gate needs tasks with checkable pass or fail outcomes. A style guide or
taste document with no measurable success signal has no sel split, so the
gate cannot run. Say that you are flagging, not gating, and do not invent a
score to make the gate pass.
A harness that cannot run a separate sel runner must not run skillopt at
all, not even to flag. See Precondition 4.
Anti-Patterns
| Avoid |
Why |
Instead |
| Editing the target by hand |
The gate never sees the edit |
Route every edit through apply |
| Promoting on a higher aggregate score |
It can hide a lost behavior |
Use pair-gate |
Reflecting on sel tasks |
That trains on the test set |
Reflect on train only |
| Treating a correction as a defect |
Corrections are weak signals |
Confirm in the target and two sessions |
| Calling an accepted candidate installed |
Acceptance is not deployment |
Report ACCEPTED_NOT_DEPLOYED until promoted |
Verification
Extension Points
- A new acceptance rule belongs in
pair-gate, with a test, not in prose.
- A new session source belongs in
mine_sessions.py sources and its table
in references/cross-machine-mining.md.
Close-out
After each run, check this skill and its references for a reusable lesson
from a user correction, failed verifier, or tool discovery. Patch the
class-level guidance; do not add an incident-named reference.
References
references/eval-set-format.md: eval manifest, patch, and results schemas.
references/description-optimization.md: trigger-rate eval for description edits.
references/rollout-validity.md: preflight, grading, and audit rules for rollouts.
references/target-selection.md: defect confirmation, split timing, anchors, cohorts.
references/slow-section-convention.md: protected-fence contract.
references/mechanism.md: the training analogy and paper citation.
references/cross-machine-mining.md: session sources and the handoff.
references/example-eval.json with example-incumbent-results.json and
example-candidate-results.json: a runnable pair-gate example.
1---2name: skillopt3description: Improve a skill, agent file, or prompt through bounded patches and a held-out, per-task acceptance gate. Use when you say `optimize this skill`, `make this skill better`, `compress these skills safely`, `tune this prompt against an eval`, or a skill keeps failing the same way. Refuses unscored improvement claims. Do NOT use to create or route skills (use skillforge) or to decide skill placement or compress markdown without an eval (use context-optimizer).4metadata:5 source: https://skillmd.com/skills/rjmurillo/skillopt6---78# SkillOpt910> A candidate lands only when it adds at least one held-out pass and loses11> none. No scored eval set means no gate, and no gate means no edit.1213SkillOpt treats a skill document as the trainable state of a frozen model.14You run rollouts and reflect on failures. The bundled scripts decide whether15an edit may exist: the edit budget, the patch applier, the protected-section16guard, the rejected-edit buffer, and the acceptance gate are deterministic.17Do not edit the target by hand. Route every edit through the scripts so the18gate is real. Method: Yang et al., arXiv:2605.23904. The paper's reported19gains are its claims; this skill does not reproduce them.2021## Triggers2223| Phrase | Action |24|--------|--------|25| `optimize this skill` | Run the Process below against a scored eval set |26| `make this skill better` | Same, after the Preconditions pass |27| `compress these skills safely` | Declare a resource task up front, then gate |28| `tune this prompt against an eval` | Same loop with a prompt as the target |29| A skill keeps failing the same way | Mine sessions, confirm the defect, then gate |3031## Boundaries3233| Need | Owner |34|------|-------|35| Create a skill, find one, or route a request to one | `skillforge` |36| Decide skill vs passive context, or compress markdown with no eval | `context-optimizer` |37| Measured improvement of an existing skill, agent file, or prompt | this skill |38| Held-out optimization of agents, rules, and hooks inside the `rjmurillo/ai-agents` checkout | `optimize-artifact.py` in that repository's eval tooling (not shipped) |3940## Preconditions (hard stop if unmet)41421. **Target artifact.** A `SKILL.md`, agent file, or prompt.432. **Scored eval set.** A nonempty held-out `sel` split with per-task pass or44 fail verifiers (`references/eval-set-format.md`). Without one, stop and say45 so. `score` and `pair-gate` exit 2 on a missing or empty split.463. **Canonical owner.** Name the versioned source, its generated mirrors, and47 the live install path before any candidate runs. A file in an eval48 workspace is not deployed behavior.494. **Two isolated runners.** The candidate author and the `sel` runner must50 be separate: a fresh subagent or another session. The `sel` runner writes51 and holds the full `eval.json`. The author gets a train-only copy that52 keeps the `sel` task ids but drops the `sel` task bodies; `apply` reads53 only those ids. In every step the author must not write, run, or see the54 `sel` prompts, verifiers, or per-task results. Only the gate verdict goes55 back. If the harness cannot provide both, do not run skillopt. Report that the gate cannot be56 independent. Replacement holdouts written by the same agent are still57 exposed.5859Freeze the `train` and `sel` task lists before authoring any candidate. Keep60`sel` prompts, verifiers, and outcomes away from the candidate author for the61whole run (Precondition 4). If they were exposed, list them under `exposed` in62the manifest, move them to `train`, and add untouched holdout tasks written63by a runner who has not seen the candidate (Precondition 4).64`pair-gate` exits 2 while an exposed or invalid task sits in the `sel` split.6566Deterministic tasks taken from a written spec before any candidate exists67are frozen once their list and verifiers stop changing. Freezing is not68isolation. If the agent that wrote them also writes the candidate, it has69seen every verifier, so list them under `exposed` and keep them in `train`.70They still catch regressions. They count as `sel` only when a different71author or session wrote them and kept them from the candidate author.7273Read `references/target-selection.md` before you fund rollouts for a target,74and when a patch fails to apply. It covers confirming the defect, splitting75after baseline, anchor rules, and library-wide passes.7677## Scripts7879Every subcommand takes explicit file paths, so it runs from any working80directory and needs no clone of the source repository. Pure standard library.8182```bash83python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/skillopt/scripts/skillopt.py" --help84python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/skillopt/scripts/mine_sessions.py" --help85SKILLOPT_DIR="${SKILLOPT_DIR:-${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/skillopt/scripts}"86```8788If neither `COPILOT_PLUGIN_ROOT` nor `CLAUDE_PLUGIN_ROOT` is set (for example89Codex, or a plain checkout under PowerShell), use the directory that holds90this SKILL.md. Set `SKILLOPT_DIR` to the `scripts` folder beside it:9192```bash93SKILLOPT_DIR="<folder that holds this SKILL.md>/scripts"94python3 "$SKILLOPT_DIR/skillopt.py" --help95```9697On Windows PowerShell, no Bash required:9899```powershell100if (-not $SkillOptDir) {101 $Root = if ($env:COPILOT_PLUGIN_ROOT) { $env:COPILOT_PLUGIN_ROOT } else { $env:CLAUDE_PLUGIN_ROOT }102 if (-not $Root) { throw 'Set COPILOT_PLUGIN_ROOT or CLAUDE_PLUGIN_ROOT, or set $SkillOptDir to the scripts folder beside this SKILL.md' }103 $SkillOptDir = Join-Path $Root 'skills\skillopt\scripts'104}105$SkillOpt = Join-Path $SkillOptDir 'skillopt.py'106py -3 $SkillOpt --help107```108109Below, `skillopt.py` means that resolved path. References use `$SKILLOPT_DIR`110(`$SkillOptDir` in PowerShell) for the same folder.111112| Command | Exit codes |113|---------|------------|114| `skillopt.py budget`, `apply`, `score`, `buffer-add` | 0 success, 2 usage or validation error |115| `skillopt.py gate`, `pair-gate` | 0 accept, 1 reject, 2 usage or validation error |116| `skillopt.py efficiency-gate` | 0 `ACCEPT_EFFICIENCY`, 1 reject, 2 usage or validation error |117| `skillopt.py report` | 0 report printed, 2 usage or validation error. Never gates |118| `skillopt.py run` | 0 results written, 1 results written with a task `UNVERIFIED`, 2 refused or invalid (no results written) |119| `skillopt.py hash PATH...` | 0 digests printed; a directory hashes as one whole bundle |120| `skillopt.py hash PATH --expect SHA` | 0 match, 1 `MISMATCH`, 2 usage or validation error |121| `skillopt.py buffer-check` | 0 `KNOWN-BAD` (discard), 1 `NOVEL` (proceed), 2 usage or validation error |122| `mine_sessions.py sources`, `rank`, `tasks`, `collect`, `merge` | 0 success, 2 usage or validation error |123124`buffer-check` is inverted on purpose: exit 0 means the edit was already125rejected. Branch on the printed word or the exact code, never on `&&`.126127Modules named `_skillopt_*.py` and `_mine_*.py` are internal helpers the128two entry scripts import. Do not call them directly.129130## Process131132One optimization step:1331341. **Baseline.** Read `references/rollout-validity.md` before the first135 rollout. Split the roles first. The candidate author sees only136 `train` tasks and their results. A separate runner, such as a fresh137 subagent or another session (Precondition 4), holds the `sel` prompts,138 verifiers, and results in every step, and reports only the verdict back.139 Each runner preflights its own tasks: prompt, verifier, inputs, fixtures,140 and permitted scope. Run the rollout under exactly the scope the task grants.141 A rollout with the wrong artifact or a scope change is invalid, not a pass142 or a fail; mark the task `"status": "invalid"` and replace it. Grade the143 complete persisted output against every clause of the verifier. A runner's144 self-check or summary is not proof. A clipped or missing answer is145 `"UNVERIFIED"` in the results file, which the gate refuses to count.146 Record `{"task_id": true|false}` to `incumbent_results.json` and hash the147 incumbent: `skillopt.py hash target.md`. For tasks with an argv `command`,148 `skillopt.py run --eval eval.json --bundle DIR --role incumbent --out-dir149 results/` runs them and writes `results/incumbent_results.json`. It150 replaces the whole file, so run it first and add your grades for tasks with151 no `command` after it finishes. Read the security note in152 `references/eval-set-format.md` first, because `eval.json` commands run any153 binary on `PATH`. When the target is a skill directory with scripts, pass154 the directory to `hash` and `apply`; it hashes as one whole bundle. When the155 manifest sets156 `"scorer": "nondeterministic"`, the `sel` runner also reruns the157 byte-identical incumbent into `null_results.json` now (see Before158 promotion, step 2).1592. **Budget.** `skillopt.py budget --step T --total N` prints L_t, the160 cosine-decayed cap on atomic patches this step. T is 0-based, from 0 to161 N-1; `--step N` exits 2 with `out of range`.1623. **Reflect.** Failures become corrective rules; successes become behaviors163 to preserve. Write at most L_t patches to `patches.json`. Use only `train`164 evidence.1654. **Buffer check.** `skillopt.py buffer-check --buffer rejected.jsonl166 --patches patches.json`. Exit 0 prints `KNOWN-BAD`: discard and reflect167 again. Exit 1 prints `NOVEL`: proceed.1685. **Apply to a candidate.** `skillopt.py apply --skill target.md --eval169 eval.json --patches patches.json --budget L_t --out candidate.md`. The170 incumbent is never written; an `--out` that resolves to the incumbent exits171 2. Exit 2 is a usage or validation error, not a gate loss: an empty `sel`172 split, an incumbent `--out`, budget overflow, a bad anchor, a protected173 fence, or marker text inside a patch.1746. **Re-roll** every task on `candidate.md` into `candidate_results.json`.175 For a bundle, `apply` writes a new directory. Finish every edit to its176 scripts first, then run the command tasks, then `hash` the final177 `CANDIDATE_DIR`. That digest is the candidate's identity, so results from178 an earlier state of the bundle would gate a different candidate. The `sel`179 runner does the `sel` half. For command tasks, run180 `skillopt.py run --eval eval.json --bundle CANDIDATE_DIR --role candidate181 --out-dir results/`, then add your grades for tasks with no `command`.1827. **Gate.** `skillopt.py pair-gate --eval eval.json --incumbent-results183 incumbent_results.json --candidate-results candidate_results.json`. If you184 used `run --out-dir results/`, pass the files under `results/`. For a185 nondeterministic scorer, add `--null-results null_results.json186 --incumbent-sha256 <sha> --null-sha256 <sha>`; without them the gate exits187 2.188 - Exit 0, `ACCEPT_PARETO`: at least one held-out gain, zero held-out189 regressions, every held-out result a literal boolean.190 - Verdict precedence: `REJECT_UNVERIFIED` first (a missing or non-boolean191 result), then `REJECT_NULL_CONTROL` (the null run moved and no genuine192 gain remains), then `REJECT_CEILING` (the incumbent already passes every193 `sel` task and the candidate regresses none), then `ACCEPT_PARETO` or194 `REJECT`.195 - Exit 1, `REJECT`, `REJECT_UNVERIFIED`, or `REJECT_NULL_CONTROL`: record196 it with `skillopt.py buffer-add --buffer rejected.jsonl --patches197 patches.json --score <cand> --incumbent <best> --note "..." --step T`.198 - Exit 1, `REJECT_CEILING`: the edit lost nothing and is not bad. Do not199 buffer it. No candidate can pass a gate that has no headroom, so stop the200 loop and read the `hint`: freeze a new `sel` set of harder tasks that201 sample the failure mode, re-score the incumbent on it, then author a202 fresh candidate. Never re-gate this candidate on tasks written after it.203 For a first draft whose `sel` tasks came from its own requirements, hand204 back to `skillforge`.205 - Cost, latency, or token claim at the ceiling: when the incumbent passes206 every `sel` task and the claim is a gain declared in the frozen manifest207 before authoring, run `skillopt.py efficiency-gate --eval eval.json208 --incumbent-results incumbent_results.json --candidate-results209 candidate_results.json --measurements measurements.json`. Read210 `references/eval-set-format.md` (Efficiency measurements, Efficiency211 gate) first. It never replaces `pair-gate` for a claim of better behavior.2128. **Repeat** until the budget reaches its floor and several steps reject, or213 the step budget runs out.214215Use `pair-gate` as the acceptance decision. Scalar `gate --candidate X216--incumbent Y` (exit 0 `ACCEPT`, exit 1 `REJECT`, ties reject) is only for a217single-objective experiment declared before the candidate existed. A resource218objective such as context reduction may be one `sel` task with219`"kind": "resource"`, declared before candidate creation. It cannot offset a220semantic regression, because `pair-gate` rejects any regression.221222## Before promotion223224`ACCEPT_PARETO` is necessary, not sufficient:2252261. **Training review.** Run `pair-gate --split train` and read its227 `regressions`. A lost training pass on a load-bearing verifier blocks228 promotion. If fixing it changes the candidate, freeze a fresh `sel` set.2292. **Null control** for nondeterministic scorers. It runs at baseline and230 feeds the step 7 gate. Set `"scorer": "nondeterministic"` in the manifest;231 `pair-gate` on `sel` then exits 2 without `--null-results`. Rerun the232 byte-identical incumbent (confirm with233 `skillopt.py hash target.md --expect <sha256>`, exit 1 on `MISMATCH`) into234 `null_results.json`, then pass `--null-results null_results.json235 --incumbent-sha256 <sha> --null-sha256 <sha>` to `pair-gate`, using the236 digests `hash` printed for the incumbent and for the file the null run237 used. Both are required, and a mismatch exits 2. A gain the no-op also238 produced is noise; if no genuine gain remains the verdict is239 `REJECT_NULL_CONTROL`.2403. **Evidence.** Keep incumbent, candidate, patches, every results file, the241 pair report, hashes, `rejected.jsonl`, and one line of evidence per task.242 `skillopt.py report` lays out the evidence, benchmark spread, and case243 signals (`references/eval-set-format.md`, Run report). It explains, never gates.244245## Promotion2462471. Confirm the canonical source still hashes to the evaluated incumbent.2482. Apply the accepted candidate to the canonical source, not to an eval copy249 or a generated mirror.2503. Hash the canonical source after applying it251 (`skillopt.py hash canonical.md --expect <candidate sha256>`). It must252 equal the accepted candidate's hash before you check the live copy.2534. Regenerate mirrors, run the owner's tests, and load the skill once.2545. Commit and push through the owner's repository workflow.2556. Reload the live skill and confirm its hash matches the canonical file.256257If any step stops short, report `ACCEPTED_NOT_DEPLOYED`. Never call an258eval-workspace candidate installed.259260## Protected sections261262Fence guidance that step edits must never touch:263264```text265<!-- SLOW_UPDATE_START -->266Durable guidance changed only by hand, at epoch boundaries.267<!-- SLOW_UPDATE_END -->268```269270Contract: `references/slow-section-convention.md`.271272## Finding targets from sessions273274`mine_sessions.py` ranks skills by usage and a weak correction signal from275local transcripts, and pulls the prompts that triggered a skill:276277- `sources` lists each supported source and whether it is available.278- `rank`, `tasks --skill X`, `collect --out PATH`, `merge`.279- `--patterns patterns.json` reads a `session-patterns/1` handoff instead of280 transcripts.281282Supported sources: Claude Code transcripts (`~/.claude/projects/**/*.jsonl`)283and the GitHub Copilot CLI session store (`~/.copilot/session-store.db`),284both resolved from the user's home directory on Linux, macOS, and Windows.285Any other harness is unsupported; supply a `session-patterns/1` file286instead. A missing source is skipped, never an error; `sources` shows which287ones were found, and a run with no source at all prints a note to stderr. Session288data stays local; keep extracted prompts out of git unless the user names289another destination.290291A correction nominates a candidate. It is not a gold label. Confirm the292defect in the target and in two distinct sessions before funding rollouts.293Workflow: `references/cross-machine-mining.md`.294295## Description changes296297Read `references/description-optimization.md` before you edit a `description`298field or claim a skill triggers better. Output-quality rollouts do not measure299activation, so score a description candidate on trigger rate instead.300301When a candidate patch adds or rewrites skill prose, read302`../skillforge/references/skill-content-rules.md` first.303304## Limits305306The gate needs tasks with checkable pass or fail outcomes. A style guide or307taste document with no measurable success signal has no `sel` split, so the308gate cannot run. Say that you are flagging, not gating, and do not invent a309score to make the gate pass.310311A harness that cannot run a separate `sel` runner must not run skillopt at312all, not even to flag. See Precondition 4.313314## Anti-Patterns315316| Avoid | Why | Instead |317|-------|-----|---------|318| Editing the target by hand | The gate never sees the edit | Route every edit through `apply` |319| Promoting on a higher aggregate score | It can hide a lost behavior | Use `pair-gate` |320| Reflecting on `sel` tasks | That trains on the test set | Reflect on `train` only |321| Treating a correction as a defect | Corrections are weak signals | Confirm in the target and two sessions |322| Calling an accepted candidate installed | Acceptance is not deployment | Report `ACCEPTED_NOT_DEPLOYED` until promoted |323324## Verification325326- [ ] `eval.json` has a nonempty `sel` split frozen before the candidate.327- [ ] A runner other than the candidate author wrote and ran the `sel` tasks,328 and the author never saw them.329- [ ] No `sel` task is exposed or invalid.330- [ ] The incumbent hash is recorded and unchanged after `apply`.331- [ ] `pair-gate` returned `ACCEPT_PARETO` with no unverified task.332- [ ] Training regressions were reviewed.333- [ ] A null control ran when the scorer is nondeterministic.334- [ ] The live copy hashes to the canonical source, or the report says335 `ACCEPTED_NOT_DEPLOYED`.336337## Extension Points338339- A new acceptance rule belongs in `pair-gate`, with a test, not in prose.340- A new session source belongs in `mine_sessions.py sources` and its table341 in `references/cross-machine-mining.md`.342343## Close-out344345After each run, check this skill and its references for a reusable lesson346from a user correction, failed verifier, or tool discovery. Patch the347class-level guidance; do not add an incident-named reference.348349## References350351- `references/eval-set-format.md`: eval manifest, patch, and results schemas.352- `references/description-optimization.md`: trigger-rate eval for `description` edits.353- `references/rollout-validity.md`: preflight, grading, and audit rules for rollouts.354- `references/target-selection.md`: defect confirmation, split timing, anchors, cohorts.355- `references/slow-section-convention.md`: protected-fence contract.356- `references/mechanism.md`: the training analogy and paper citation.357- `references/cross-machine-mining.md`: session sources and the handoff.358- `references/example-eval.json` with `example-incumbent-results.json` and359 `example-candidate-results.json`: a runnable `pair-gate` example.