Skill Opt

skill-opt

What This Does

One agent switches hats: Setup → Rollout → Score → Reflect → Edit → Gate → Memory, then repeats. The run directory .skill-opt/runs/<skill>-<n>/ is the message bus — every phase reads/writes files there, making runs resumable from any point. The one-man-play is the backbone, not a constraint: when parallelism > 1, rollouts and gating fan out to fresh subagents.

Up-Front Questionnaire

Ask the user these questions, then write config.yml from templates/config.yml:

See references/feedback-sources.md for how each feedback_source builds the task suite.

The Loop

SETUP: questionnaire→config.yml; build/ingest suite→tasks/{train,holdout}; snapshot skill/v0.md;
       ROLLOUT(v0) over holdout → baseline via `scripts/ledger.py record`.
LOOP iter=1..max (early-stop after `early_stop_patience` non-improving gates, or user stop):
  ROLLOUT  : for each train-minibatch task, dispatch a FRESH SUBAGENT given ONLY {current skill text, task};
             write rollouts/iter-NN/task-MM/trajectory.md.
  SCORE    : judge each trajectory (programmatic if available else LLM-judge subagent) → score.json;
             `ledger.py record --split train`.
  REFLECT  : split minibatch into SUCCESS and FAILURE; reflect on each SEPARATELY; read memory/rejected-edits.md.
  EDIT     : propose bounded add/del/replace ops within edit_budget → candidates/iter-NN/{candidate.md,edit.json}.
  GATE     : ROLLOUT(candidate) over tasks/holdout (fresh subagents); `ledger.py record --split holdout`;
             `ledger.py gate` decides. accept→skill/v(K+1).md & update current.md; reject→append memory/rejected-edits.md.
  MEMORY   : slow update — established rules need accumulated evidence to be overturned.
FINALIZE : emit per output_mode (overwrite w/ v0 backup, or save-as-new <skill>-opt/); write report.md.

Disciplines (Non-Negotiable)

• Frozen target via fresh subagent — each rollout subagent receives only {skill text, task}; no contamination, no self-grading.
• Gate only on held-out — never use train scores for acceptance decisions.
• Deterministic gate decision — scripts/ledger.py gate does arithmetic on ledger.csv; an LLM never decides accept/reject.
• Edits bounded by edit_budget — caps ops and net words per iteration (the "textual learning rate"); prevents catastrophic overwrites.
• Consult memory/rejected-edits.md before every edit proposal — do not re-propose known-bad edits without new evidence.

Run Directory Layout

.skill-opt/runs/<skill>-<n>/
  config.yml
  skill/          v0.md  v1.md ...  current.md
  tasks/          train/  holdout/  suite.json
  rollouts/       iter-NN/task-MM/  trajectory.md  score.json
  candidates/     iter-NN/          candidate.md   edit.json
  memory/         rejected-edits.md  accepted-log.md
  ledger.csv      (source of truth; columns: iter, kind, version, split, mean_score, n, decision)
  report.md

Each subagent writes to a unique leaf path — parallel writes never collide.

Resume

Re-invoke on an existing run directory. Read ledger.csv to find the last completed phase, then continue. No double-work: every phase appends to ledger.csv before proceeding.

Reference Docs

• references/loop.md — phase mechanics, defaults, edit-budget enforcement, gate margin, memory/slow-update policy, parallelism and edit_panel
• references/fidelity.md — SkillOpt correspondence map (five mechanisms, any deviations justified)
• references/feedback-sources.md — the 4 signal modes: how Setup builds the suite and how Judge scores per mode
• references/rubrics.md — drafting task suites and scoring rubrics; programmatic vs LLM-judge; judge calibration