# Skillopt

> Run controlled skill optimization cycles on any skill document. Uses kanban-based pipelines with validation gates — the methodology from Microsoft Research's SkillOpt (arXiv 2605.23904).

- Skill: `magnus919/skillopt` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add magnus919/skillopt`
- Raw SKILL.md: https://api.skillmd.com/api/skills/magnus919/skillopt/raw
- Safety review: pending (external: skill-scanner PASS, skillspector CAUTION)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- License: MIT
- Author: magnus919 (https://skillmd.com/u/magnus919)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/magnus919/skillopt

---


# SkillOpt — Controlled Skill Optimization for Hermes Agent

Optimize any Hermes skill document using a rigorous, methodology-driven pipeline inspired by Microsoft Research's SkillOpt paper (Yifan Yang et al., 2025). The core insight: **evaluate skill changes by measuring task execution, not by reading the skill text.**

## When to Use This Skill

- A skill exists but its performance is inconsistent — sometimes it works, sometimes it doesn't
- You want to improve a skill but aren't sure which changes will actually help
- You've proposed edits to a skill and want to validate them before deploying
- You maintain multiple skills and want a repeatable process for quality improvement
- You're curious whether a skill change actually improved anything (the surface-plausibility trap)
- **You've just created a new skill and want to pre-optimize it before publication** — catch discoverability and structural issues (buried principles, missing phase mappings, unanchored procedures) before any user encounters them. The edits are prominence and navigation fixes, not performance patches. This session's spec-driven-development run is the worked example: all baseline rollouts passed, but Epoch 1 found core principles buried in a reference file and a Quick Start that lacked reference-to-phase cross-links.

**SkillOpt is designed for skills with measurable task outcomes.** For creative skills (image generation, writing) where "correctness" is subjective, the methodology still works but the validation criteria need thoughtful definition. **Crucially, the acceptance criteria themselves may need user iteration** — the first set of criteria you propose may not match what the user considers "correct" for creative output. Propose criteria alongside your edit proposals and get user sign-off before running validation. This session's image-magnus919 run is the worked example: the initial validation criteria included a 30-word SUBJECT limit that the user correctly rejected as contradicting the rich, complex subjects that produce the best covers.

## How It Works

SkillOpt treats a skill document like a parameter vector in text-space. Instead of gradient descent, it uses a six-phase **kanban pipeline** that runs on your Hermes Agent's existing infrastructure:

```
Backlog → Rollout → Reflect → Propose → Validate → Merge → Done
                                               ↓              ↑
                                          Reject Buffer ──────┘
                                           (every 4 epochs)
```

Each phase produces structured artifacts. Downstream phases read these artifacts from disk — nothing depends on LLM context retention across phases.

### The Validation-Gate Principle

The single most important idea in this skill comes from the companion SkillLens paper (arXiv 2605.23899): **surface plausibility is not predictive of skill effectiveness.** LLM judges are 46.4% worse than chance at distinguishing effective from ineffective skills by reading them. Format has no significant effect (p > 0.34).

This means: **do not evaluate skill changes by reading the skill.** Evaluate by running it. The validation gate — a held-out set of tasks that the skill has never seen — is the only reliable quality signal.

Training and validation task sets MUST be distinct. This is not optional.

### Phase 0 — Description Audit (before first Rollout)

Before running any rollouts, audit the target skill's `description` field. Per Schmid (Google DeepMind, AI Engineer 2026), **50% of skill failures trace to incorrect triggering from weak descriptions.** A skill whose body is perfectly optimized but whose description fails to trigger is invisible.

**Checks:**
1. **Imperative phrasing** — Does the description start with a directive verb ("Use", "Build", "Deploy")? Passive descriptions ("This skill does...") trigger less reliably.
2. **Directives, not essays** — Is the description a directive ("Use the Interactions API if you're building a chat application") or an explanation ("The Interactions API is recommended for multi-chat because it handles session state")? Models follow directives; they infer implications from essays unreliably. Rewrite explanations as directives.
3. **Negative boundaries** — Does it say when NOT to use the skill? Without negative boundaries, the skill over-triggers on near-miss prompts.
4. **Trigger coverage** — Test 5 representative prompts that should trigger the skill and 5 near-miss prompts that should not. If trigger accuracy is below 80%, propose description edits alongside body edits.

**Entry:** Target skill's `description` field
**Exit:** Description audit result (pass/fail per check) + proposed description edits if needed

Description edits are proposed and validated through the same pipeline as body edits — they go through Propose, Validate, and Merge like any other change.

### The Six Phases

#### 1. Rollout — Execute the current skill against training tasks

The target LLM (any model you'd normally use with Hermes) executes a set of training tasks using the current skill document. Each trajectory produces a (task, execution trace, outcome) triple.

**Entry:** Current skill document + training task suite
**Exit:** N trajectory records (one per training task)

**Important — confirm test infrastructure constraints BEFORE designing training tasks.** The user's production infrastructure (blogs, sites, services, databases) is off-limits for rollouts unless explicitly offered. Ask: "Do you have a test environment I can use, or should I create a synthetic one?" before writing any training task that touches real infrastructure. This prevents rollouts from leaving traces in production content, triggering notifications, or corrupting live data. This session's hugo-theme run is the worked example: the user redirected from production blog testing to a synthetic Hugo site, saving the cleanup cost of undoing test content on a live blog.

#### 2. Reflect — Identify systematic failure patterns

Review the rollout trajectories in a batch. Identify what went wrong, under what conditions, and what kind of change would address those failures. Batching prevents overfitting to any single failure mode.

**Ablation baseline comparison.** When feasible, include 1-2 rollout tasks run WITHOUT the skill (bare model, no skill loaded). Compare the skill-on trajectories against the skill-off trajectories for the same tasks. Instructions that produce identical behavior with and without the skill are candidates for removal or compression — they are not earning their context cost. This is not a retirement signal (the skill may be essential on other models or harnesses); it is an improvement signal that identifies dead-weight instructions within the skill. Record the ablation comparison in the reflection document alongside the failure patterns.

**Entry:** Trajectory records from Phase 1 (+ optional ablation baseline)
**Exit:** Reflection document with identified failure patterns categorized by frequency and severity, plus ablation observations (which instructions changed behavior vs. which did not)

#### 3. Propose — Generate bounded edits

Based on the reflection, propose 1-4 specific, targeted edits to the skill document. Each edit is a concrete text operation: add a line, replace a clause, delete an instruction. The edit budget is the textual analogue of a learning rate — it prevents large destabilizing changes.

**Before writing any "add" or "expand" edits, verify the content doesn't already exist elsewhere in the skill.** A common failure mode (the "buried content" trap) is proposing to add new content — rules, warnings, examples — when that content already exists in the Pitfalls section, a reference file, or a different part of the procedure. The rollout may have revealed that agents aren't *following* the existing content, not that it's missing. If the content exists but isn't discoverable, the fix is prominence (cross-references, hard-gates, blockquote alerts), not duplication. Search the entire skill document and all linked reference files before writing an add-type edit.

**No-op scan before any add-type edit.** Per Schmid (credit to Matt Pocock), AI-generated skills accumulate no-op instructions — phrases that consume context tokens without changing agent behavior. Common no-ops: "write clear, high-quality code", "follow best practices", "handle errors appropriately", "ensure high quality", "make it easy to read", "write maintainable code". Before proposing to add content, scan the existing skill for no-ops in the vicinity of the proposed edit. If a no-op exists, propose a **delete** edit to remove it alongside (or instead of) the add. Removing a no-op is a free token-efficiency gain that never changes task pass rate. The most impactful edit is sometimes the one that removes text, not adds it.

**Portfolio-awareness check for framework/comparison skills.** When the skill being optimized is part of a portfolio of sibling skills that serve related purposes (e.g., framework skills like LlamaIndex, LangGraph, PydanticAI), add a pre-Propose check: does the skill help agents choose between sibling skills? If the skill covers a domain where the user has multiple alternative skills, one of the proposals should add or improve a routing/comparison table that maps scenarios to the correct sibling skill. This prevents the optimized skill from existing in isolation — agents need to know when to reach for THIS skill vs a sibling, not just how to use THIS skill. This session's llamaindex greenfield run is the worked example: the user corrected the initial flat "When NOT to Use" list into a fully cross-referenced Framework Routing Guide covering 5 competing frameworks.

**Over-prescription detection.** Per Schmid: "If the workflow is always the same, you should not use skills. Maybe you should write a script." When reviewing rollout trajectories, watch for skills that prescribe rigid step-by-step procedures for deterministic workflows. If a skill's instructions are a fixed sequence that never varies by task ("Step 1: read the config. Step 2: update the port. Step 3: deploy."), the skill is over-prescribed — it wastes model tokens on a procedure that should be a script. Propose replacing rigid procedures with goals and constraints: "If the config needs changing, here's the file, make the change" instead of "Read the config, update the port, and deploy again." The model knows how to do the steps; it needs to know the goal and the boundaries.

**Entry:** Reflection document
**Exit:** Proposed edits (1-4), each with: type (add/replace/delete), location, old_text/new_text, rationale

#### 4. Validate — Test each edit against held-out tasks

This is the heart of the methodology. Apply each proposed edit to a copy of the skill. Run the validation task suite with the edited skill. Compare against the baseline metrics. Accept edits that improve or maintain performance. Reject the rest.

**Token efficiency is a secondary validation criterion.** Per Schmid's "always-paid context cost" insight, every token in a skill competes for context window space on every invocation. When comparing baseline and candidate, record token counts alongside pass rates. Accept edits that maintain pass rate while reducing token cost. Reject edits that improve pass rate by a marginal amount (≤2%) but increase token cost by >20%. A skill that passes all tasks at 2000 tokens is better than one that passes the same tasks at 5000 tokens — especially in multi-skill environments where every token in one skill competes with every other skill.

**Entry:** Proposed edits + validation task suite + baseline metrics
**Exit:** Accepted edits (merged into deployment candidate) + rejected edits (stored in buffer with rationale and metrics) + validation dossiers at `03-dossiers/epoch-N-validation-edit-X.json` consumed by the root pyramid (`update_root_pyramid()` in `scripts/pyramid_utils.py`)

#### 5. Merge — Deploy accepted changes

Apply all accepted edits to the working skill document. Update the baseline snapshot. Increment the epoch counter. If this is epoch 4 or validation gains have plateaued, trigger the slow-meta phase.

**Version bump convention:** Incremental SkillOpt improvements (additive fixes, workflow scaffolding, documentation) use **patch version bumps** (e.g., v1.3.0 → v1.3.1) with a `fix:` commit prefix. Use minor version bumps (v1.3.0 → v1.4.0) with a `feat:` prefix only when the scope is a true feature release or breaking change. **Mismatching prefix and version level** (e.g., `feat:` on a patch bump, or minor bump for documentation-only additions) confuses the version narrative. The commit prefix and version number must agree: `fix:` → patch, `feat:` → minor. This keeps the version number meaningful — the patch level signals "safe incremental improvement from SkillOpt" while the minor level signals "intentional feature release."

**Entry:** Deployment candidate (accepted edits)
**Exit:** Updated skill document + new baseline snapshot

**PR timing gate (required):** Do not create a PR merely because an initial implementation validates. Before opening it, confirm that every commissioned research, rollout, review, and validation worker has returned; incorporate, reject with rationale, or explicitly defer every finding; then rerun final validation on the exact candidate. If SkillOpt starts while an existing PR is still open, record that PR's current head as epoch 0. Stack epoch commits on its branch only when the user explicitly requests one-PR optimization; do not post a completion verdict or merge while any epoch worker remains active. If the PR merges while a late result arrives, do not push the merged PR branch. Preserve the result and, only when the user directs a follow-up, branch afresh from current `main`.

**Post-publication optimization is a follow-up run, not PR catch-up.** If SkillOpt is invoked after the target skill already merged, record the merge commit as epoch 0, create a fresh optimization branch from current canonical `main`, and open a new PR only after the commissioned epochs finish. Never reuse or push the merged PR branch. If the user chooses one PR for several epochs, stack only the new epoch commits on that fresh branch.

**Hardware-backed runs serialize the physical boundary.** Delegated rollout and validation workers use synthetic/read-only fixtures and must not contend for a shared serial device. The orchestrator alone performs live identification, flash, reset, or monitor gates in sequence, records observed versus documented evidence, and removes device identifiers, ports, credentials, and private paths from durable dossiers.

**Important:** The above post-merge commit/PR workflow assumes the target skill is in a git repository. For local-only skills installed directly in `~/.hermes/skills/` (no `.git` directory), see [references/non-git-skills.md](references/non-git-skills.md) — the edits apply immediately with no branch or commit cycle, and version tracking happens via frontmatter version bumps and baseline snapshots instead.

#### 6. Slow/Meta — Learn from rejected edits (every 4 epochs)

The optimizer reflects on the accumulated rejected-edit buffer — all proposals that failed validation across recent epochs. Look for patterns: are you proposing the same kind of edit that keeps failing? Is there a structural issue the per-epoch optimizer isn't addressing? This produces a strategy refinement, not a direct skill edit.

**Entry:** Rejected-edit buffer (accumulated across epochs)
**Exit:** Meta-reflection document — optimizer strategy adjustments, structural observations

### Epoch Structure and Edit Budget

A full optimization run consists of 4 epochs by default. Each epoch follows the Rollout → Reflect → Propose → Validate → Merge cycle.

- **Default edit budget:** 4 edits per epoch (matching the paper's optimal Lt=4)
- **Budget decay:** Cosine decay to a floor of 2 edits per epoch
- **Epoch 4 or plateau:** Trigger the slow-meta phase

The edit budget is configurable in `board-metadata.json` under `edit_budget`.

## Quick Start

1. **One-time install** — clone the repo into your skills directory:
   ```bash
   git clone https://github.com/magnus919/hermes-SkillOpt \
     ~/.hermes/skills/skillopt
   ```

2. **In a conversation** with your agent, say something like:
   ```
   I want to optimize my vault-note skill.
   ```

3. The agent loads this skill via `skill_view(name='skillopt')`, guides you through defining training and validation tasks, seeds the kanban board, and orchestrates the six-phase pipeline — reporting results at each stage.

## Scripts — Power Users Only

The primary interface for SkillOpt is conversational — your agent drives the pipeline. These shell scripts exist for power users who want to run phases from the command line instead. The agent ignores them and uses `hermes oneshot` + `hermes kanban` directly.

**Known limitation:** The shell scripts (`run-phase.sh`, `seed-board.sh`) are unreliable in this environment and should be avoided for phase execution. Use the conversational approach instead:

- **Rollouts:** `delegate_task` for each training task (handles multi-line instructions correctly, runs in parallel, captures full output). The shell script's `--exec` mode frequently returns 0 records written due to oneshot failures that are silently swallowed by `2>/dev/null`.
- **Reflect:** Write the reflection JSON directly from rollout records. The shell script's oneshot prompt often returns `{"error": "execution failed"}` without useful diagnostics.
- **Seed flags:** `seed-board.sh` uses `--train-tasks-file` and `--val-tasks-file` (not `--test-suite`). Task files must be plain JSON arrays, not objects with a `tasks` key. Kanban's `--board` is a global option and must precede the subcommand: `hermes kanban --board skillopt-<name> create ...`, not `hermes kanban create ... --board ...`. Do not trust a board switch alone: verify with `hermes kanban --board skillopt-<name> list` and `boards list` counts before creating tasks. `boards show` reports the process-wide active board and is not evidence that a `--board`-scoped operation targeted the intended board.
- **Pyramid utils:** When calling `update_root_pyramid()` outside the PYEOF blocks, set `PYTHONPATH` to the skillopt `scripts/` directory using its absolute path: `PYTHONPATH="/Users/magnus/.hermes/skills/skillopt/scripts:$PYTHONPATH"`. The `dirname "$0"` approach may not resolve correctly from python3 heredocs.

The `seed-board.sh` triple-quote JSON bug was fixed in v1.1.0 (omnibus PR #12).

| Script | What it does |
|--------|-------------|
| `scripts/pyramid_utils.py` | Shared `update_root_pyramid()` — regenerates root pyramid from `03-dossiers/` contents. Import by setting `PYTHONPATH` to the scripts dir absolute path before calling python3. It does not export a snapshot helper; create `snapshots/epoch-N/` by copying the complete target skill directory and verify the copied `SKILL.md`. |
| `scripts/seed-board.sh` | Create kanban board, state directory, baseline snapshot |
| `scripts/run-phase.sh` | Execute a single pipeline phase |
| `scripts/archive-run.sh` | Finalize a run, store metrics, clean up the board |

## References

| Reference | What it covers |
|-----------|----------------|
| `references/artifact-pyramid-phase-outputs.md` | Unified pyramid structure, dossier naming conventions, `update_root_pyramid()`, backward compat with old-format directories |
| `references/multi-skill-patterns.md` | Cross-cutting observations from a 3-skill SkillOpt session — the buried content trap, progressive disclosure, creative skill metrics, subject derivation priority |
| `references/test-suite-design.md` | How to pick training and validation tasks |
| `references/validation-trajectory-integrity.md` | Direct-execution output contract, infrastructure-invalid classifications, parent scoring, cleanup, and variance control |
| `references/evidence-governed-execution.md` | Pre-response claim ledger, command operational-validity checks, unknown-artifact discipline, and raw-response scoring |
| `references/artifact-formats.md` | JSON schemas for intermediate artifacts |
| `references/command-syntax-verification.md` | CLI command examples in proposed edits |
| `references/clean-room-evaluation.md` | Docker environment patterns for rollouts |
| `references/meta-optimization.md` | SkillOpt applied to evaluation tooling — worked example: Groktobench HARP Epoch 1 |
| `references/multi-epoch-progression.md` | What each epoch should focus on — the structural-to-decision-to-pattern arc, with a worked example from mermaid-magnus919 v1.0.0→v1.0.3 |
| `references/artifact-pyramid-phase-outputs.md` | Conversion status and canonical structure for all phase outputs — baseline, validation, rollout, proposal, slow-meta, run summary |
| `references/delegation-to-authority-pattern.md` | Worked example: replacing duplicated conventions with skill_view() delegation — news-scan → vault-note case study |
| `references/post-merge-commit-catchup.md` | What to do when the Merge phase leaves improvements unstaged on main — detection, recovery branch, commit, PR |
| `references/non-git-skills.md` | How to handle Merge/versioning when the target skill has no git repo — local-only skill installations |
| `references/local-installation-after-merge.md` | Install a merged repository-backed skill without disturbing unrelated checkout work; verify durable source, symlink, self-test, and Hermes discovery |
| `references/publish-to-github.md` | Publishing a SkillOpt-honed skill to a new GitHub repo — sanitize, rename, create repo, copy, AGENTS.md, push |
| `references/creative-skill-optimization.md` | Adapting SkillOpt for skills with subjective outputs — measurable wrappers, validation strategy, worked example from image-magnus919 |
| `references/cli-skill-preflight-audit.md` | Systematic audit for CLI/tool reference skills — cross-referencing against `--help` output and official docs before Rollout. Worked example: obsidian-cli coverage gap analysis (June 2026) |
| `references/version-granularity-in-reference-tables.md` | Adding a version column to reference tables to surface per-feature minimum versions — prevents agents from trying unsupported features |
| `references/greenfield-skill-optimization.md` | Optimizing a skill that was just created and has never been used — the baseline tells you nothing, so Epoch 1 tests prominence/discoverability and Epoch 2 pre-loads decision intelligence. Worked example from spec-driven-development v1.0→v1.1.0 |

## Design Principles

1. **Training and validation sets must be distinct.** Non-negotiable.
2. **Evaluate execution, not text.** Never ask an LLM to judge a skill change by reading it.
3. **Edits are bounded per epoch.** The edit budget prevents large destabilizing changes.
4. **Rejected edits are preserved.** They become the input to the slow-meta phase.
5. **The optimizer and target can be the same model.** Same-model optimization still produces strong gains.
6. **Prefer delegation over duplication.** Don't duplicate cross-cutting patterns into the target skill — reference the authoritative skill instead.
7. **Hermes-opinionated design.** SkillOpt is designed for Hermes Agent and should leverage Hermes-specific features (profiles, `/goal` system, artifact-pyramids, kanban) rather than trying to be portable to other harnesses. Optimize for Hermes, not for generality.

## Pitfalls

- **State-schema preflight is mandatory before the first dossier.** Create the unified pyramid paths exactly as specified: `00-index.md` is a file at the state root; `01-summary/` and `02-analysis/` are directories; `03-dossiers/` is flat. Validate `board-metadata.json` against the fields consumed by `scripts/pyramid_utils.py` before regenerating the pyramid, especially `pass_rate_history` entries with `pass_rate`, `accepted`, and `rejected`. A plausible state tree with mismatched names or keys can produce an incomplete summary without raising an error.
- **A rollout worker's PASS label is not a validation result.** Before recording any baseline or candidate score, inspect the raw output against the rubric yourself. Literal output contracts need literal checks: a worker may claim it used the required headings while emitting bold labels instead of Markdown headings, or claim a permitted verdict while using an alias. Score the artifact, not the worker's self-assessment.
- **Verify claimed skill absence against the candidate artifact before scoring it.** A worker may say a routed procedure is missing even when the candidate reference contains it. Search/read the candidate yourself. If the content exists and the task should have routed there, classify the trajectory as infrastructure-invalid and rerun only that task; do not reject the edit based on the worker's false absence claim.
- **Acceptance criteria are conjunctive hard gates, not aggregate suggestions.** An edit that improves one mapped task still fails when any user-approved acceptance criterion remains unmet. Preserve the partial gain in the rejection buffer, reformulate it in a later epoch, and do not let a higher aggregate score override a missing rollback, security, causality, or correctness requirement.
- **An expected failure can come from the wrong enforcement layer.** Matching an error code or rejection is not proof that the behavior under test caused it. Establish prerequisites and a positive control before scoring the negative case: for PostgreSQL RLS, grant the test role schema/table privileges and prove an authorized same-tenant operation succeeds before accepting `42501` as RLS evidence; otherwise ordinary relation-permission denial can create a false PASS. Apply the same causal check to authentication, network, parser, and policy tests: verify the request reached the intended gate, then verify that gate rejected it. If the trajectory cannot distinguish the causal layer, mark it failed rather than crediting a coincidental error.
- **A validation trajectory must contain the full direct task response.** A textual review of whether the skill covers the rubric, or a summary containing only pass counts and self-scores, is infrastructure-invalid even when it sounds correct. Prefer having the worker return the direct user-facing answer while the orchestrator owns the dossier envelope and records task ID, delegation metadata, raw output, failure modes, and parent scoring. Use the five-key worker JSON contract only when an automated consumer genuinely requires it and the model has demonstrated reliable adherence. If one complete response is mechanically separable, preserve it unchanged and record the normalization boundary instead of rerunning solely for envelope or fence contamination; never reconstruct missing or truncated content. See [references/validation-trajectory-integrity.md](references/validation-trajectory-integrity.md).
- **Verify root-pyramid completeness after every regeneration.** `update_root_pyramid()` can return successfully while its phase discovery omits dossiers whose filename has no hyphen after the phase name (for example, `epoch-N-reflection.json`). Compare every current `03-dossiers/epoch-N-*.json` file against the generated epoch overview; if anything is missing, retain the dossier as the source of truth and record the indexing gap before relying on the L1/L2 navigation.
- **Not separating training and validation sets.** The most common and most damaging mistake.
- **Validating a candidate with a task that does not trigger its edit.** Before dispatching baseline or candidate workers, map every proposed edit to at least one held-out criterion. A task that lacks the condition the edit governs is infrastructure-invalid for that edit: replace it and rerun its baseline before candidate validation. Record the discarded trajectory and why; do not count a convenient pass as evidence.
- **Skipping the validation gate.** The gate is the only thing preventing overfitting to the training batch.
- **Unbounded edits.** An optimizer that rewrites the entire skill causes skill drift.
- **Running epochs without plateau detection.** After 4 epochs without improvement, run slow-meta.
- **Combined code + skill fixes:** Check for a tool bug BEFORE proposing skill edits. Fix the tool, re-run validation, then observe whether skill edits change outcomes.
- **Leaf help must expose inherited safety flags.** A CLI may accept global options after a subcommand while `leaf --help` hides them, causing workers to falsely claim that dry-run, JSON output, confirmation, or bounds are unsupported. Probe the exact leaf help used in the task. If behavior exists but help omits it, fix the shared help contract, run the full tests, rebuild both baseline and candidate copies, and rerun paired validation before scoring the skill edit.
- **Stale test assertions exposed by tool fixes:** After applying tool-bug fixes, run the full test suite before validation.
- **Omnibus PR scope creep — verify each claimed issue is actually closed.** When a PR body claims to close multiple issues (e.g., `Closes #25, #26, #27, #28`), verify that the code actually addresses each one. A PR that refactors the infrastructure may declare the issues closed but leave the bash-level phase runners writing to old directories. After merge, check each issue individually — the PR body's claim is a plan, not a guarantee. If issues remain open, file a follow-up or extend the PR before merge.
- **Rollout subagents can go off-skill.** Frame training tasks as "using ONLY the skill content" to prevent source code drift.
- **Late delegation results can invalidate a provisional fallback.** A local deterministic fallback may finish before `delegate_task` workers return. Keep the epoch provisional until every dispatched worker has returned or been explicitly marked failed; if late trajectories arrive, record them and revise the reflection/validation artifacts before final reporting.
- **Edit ordering matters when edits overlap in location.** Check whether a replace-type edit's old_string includes territory an add-type edit modifies.
- **Revalidate proposal applicability against the post-merge baseline before constructing candidates.** A proposal can become stale when an earlier epoch already applied the same wording or when the current baseline contains a stronger equivalent. Never apply a substring replacement blindly: first assert that the exact `old_text` occurs once and that the intended new state is not already present. If the edit is already present, record it as redundant/rejected and validate the cumulative candidate without duplicating the text. This prevents false regressions caused by malformed validation copies.
- **A validated candidate can become stale before Merge.** Late corrections may strengthen the live target after the validation copy was built. Before merging, diff the final candidate against the current target and classify every difference as accepted edit, later correction, rejected edit, or path-normalization artifact. Apply only accepted deltas onto the current target; never replace stronger live text with an older candidate wholesale. Rebuild the post-merge snapshot from the exact installed target and run one mapped regression task against that installed artifact before closing Merge.
- **When baseline pass rate is already 100%, validation measures regression only.** Accept edits based on non-regression plus manual verification.
- **Hermes kanban --priority prefers numeric values.** Use integers for guaranteed compatibility.
- **Command syntax edits require end-to-end testing.** Run every command in a proposed edit's examples before merging. Treat a plausible flag as untrusted until the exact installed binary's `--help` output confirms it; a server/API capability does not imply a CLI capability.
- **Safety-classifier fixes must prove they introduce no false negatives.** Replacing a conservative regex or guard with a parser-like scanner can fix harmless false positives while opening a confirmation bypass. Run paired benign/dangerous cases, adversarial delimiter escape parity, one bounded live read, one blocked-before-side-effect positive control, and an exact-head re-review after any finding is fixed. Use [references/safety-classifier-validation.md](references/safety-classifier-validation.md) for the compact gate.
- **Pipeline grammar is part of the command contract.** A rollout can understand the safety principle yet compose a shell pipeline incorrectly (for example, placing `| gzip` outside a `--stdin-from-command` child, which pipes the parent tool's output instead). Add a held-out task at the exact shell/process boundary, and score producer-failure propagation independently: a wrapper pipeline without `pipefail` is not a safe PASS.
- **Production-backed rollouts need two independent gates.** User-authorized read-only access may establish a real baseline, but validation workers must not start a second process, mutate service/config state, or read secret values. For plan-only tasks, score mutation leakage as a failure even when the proposed procedure is technically correct. Make the target's actual deployment shape, including legacy hybrid layouts such as systemd wrapping a home-directory checkout, explicit in the task context before evaluating discoverability.
- **A greenfield template is an executable contract, not just documentation.** A rollout can pass while a canonical template contradicts the skill's own decision rules or contains a field that the current implementation rejects. Before accepting template edits, run the rendered template through the target tool's config/schema validation and one bounded, non-destructive smoke path. Treat official documentation and the running implementation as separate evidence; when they disagree, record the compatibility boundary and prefer the tested implementation for portable defaults.- **Atom/molecule scope creep is a rollout signal.** Capture structural mismatches as failure modes.
- **The first failure pattern in Reflect is usually the most visible, not the most impactful.** Survey the full landscape before converging.
- **seed-board.sh JSON quoting fails on mixed-quote task content.** Use Python seed for tasks with special characters.
- **Shell-script phase runners are unreliable — use delegate_task for all rollouts.** The `run-phase.sh --exec` mode frequently returns 0 records written because `hermes oneshot` calls fail silently (output swallowed by `2>/dev/null`). The reflect phase similarly returns `{"error": "execution failed"}` via oneshot. For rollouts, use `delegate_task` (handles multi-line instructions correctly, runs tasks in parallel, captures full output). For reflection, write the JSON directly from rollout records rather than relying on the shell script's oneshot prompt.
- **Multi-epoch progression: Epoch 1 adds the what, Epoch 2 adds the how.** Design training tasks for multi-epoch awareness.
- **Parallel rollouts via delegate_task speed the pipeline.** Use subagents for rollout execution.
- **The "buried content" trap in Propose.** Before writing any "add" edit, check whether the target content already exists elsewhere in the skill or its reference files. A pitfall, a paragraph in a different section, or a reference document may already contain the exact rule you're about to duplicate. If the content exists but agents aren't following it, the real fix is prominence (cross-references, hard-gates, blockquote callouts), not duplication. This session produced a clear example: all four initially-proposed "add content" edits were already present in the target skill — just in locations agents weren't reaching during the Procedure. The edits that survived validation were the prominence fixes, not the content additions. Always search the full skill before proposing an add-type edit.
- **The "frame accuracy" trap — buried content is about TEXT, not CONCEPT.** A procedurally perfect SkillOpt cycle can still be directionally wrong if the skill's core conceptual framing doesn't match its canonical source or prematurely chooses for the user. The artifact-pyramids Epoch 1 (2026-05-31) is the worked example: all edits passed validation and merged cleanly, but the skill's layer naming directly contradicted the published canonical article. A second form appears in infrastructure skills that cover several valid operating models: giving one path most of the procedural prominence, or placing a development-only warning where it can be read as applying to production self-hosting, silently turns documentation structure into a recommendation. **Prevent this before Epoch 1:** verify canonical terminology and relationships; enumerate each supported operating model independently; scope warnings to the exact model they govern; and add a neutral discovery task where user intent is unknown plus a held-out task where the user explicitly prefers the less-prominent option. Score whether the agent discovers intent and constraints before recommending, not merely whether every option is mentioned somewhere. Frame misalignment is not fixed by polishing the favored path; preserve it in the baseline, measure it, then make the smallest structural correction that restores accurate, user-intent-led routing.
- **Version humility — don't over-version session-young skills.** A skill created in the same session is at best v0.0.x, not v2.0. The version number should reflect the skill's maturity and iteration count, not the significance perceived in the moment. A skill that's existed for 3 hours and gone through 3 revisions is v0.0.3. Reserve minor bumps (0.x.0) for features proven across multiple sessions, and major bumps (x.0.0) for stable, cross-session releases. The artifact-pyramids Epoch 1 (2026-05-31) is the worked example: the PR was titled v2.0 for a skill created the same day, when v0.0.3 was the honest version. This inflates expectations and confuses the version narrative for anyone reading the repo history. **Mirror-image pitfall: updating the PR/commit title but not the body.** When you correct a version (or name, or framing) in a PR title, audit every other occurrence — the body, the commit message, the branch description, any linked issues. The artifact-pyramids Epoch 1 correction (v2.0 → v0.0.3) is the worked example: the `gh pr edit --title` fix changed the headline but left the body describing the PR in "v2.0" language, which the user caught as incomplete. Fix: after any title-level correction, re-read the full PR body and commit message for stale references before declaring done. A partial fix that leaves contradictions in the artifact is worse than no fix — it creates a state where two different versions claim authority in different places of the same record.
- **Style-catalog optimization must validate routing and complete rendered coverage.** For a creative skill that recommends reusable named visual styles, begin with text-only tasks that request one named treatment, a use-case rationale, and a style-only prompt clause. Span distinct registers across training and held-out tasks, and score stable naming, fit, and reproducible visual grammar. Keep scene content separate from style. After the user approves the catalog and criteria, generate and pixel-review at least one accepted example for **every named treatment** before claiming the catalog is validated or complete. A representative subset validates only those sampled treatments; use subset sampling only when the user explicitly scopes the deliverable that way. Recommendation tasks validate discoverability and routing; rendered examples validate treatment fidelity and full-catalog coverage.
- **Creative-skill acceptance criteria need user validation too.** When running SkillOpt on skills with subjective outputs, the acceptance criteria you write for validation tasks may be wrong — not too strict for the agent, but too strict for what the user considers "good." The image-magnus919 Epoch 2 run is

…(truncated)