Skill Creator

Skill Creator

Take skills from "drafted" to "proven" through rigorous multi-layer evaluation.

Installation

bash $HOME/.golems/skills/golem-powers/skill-creator/scripts/install.sh

The installer is idempotent — safe to re-run. It:

1. Symlinks the skill into ~/.claude/skills/skill-creator (golem-powers convention)
2. Ensures $SKILL_CREATOR_REPO/.claude/agents/, $SKILL_CREATOR_REPO/.codex/agents/, and $SKILL_CREATOR_REPO/scripts/ exist as the project-scope home
3. Symlinks the session-miner agent definition into the project-scope agents dir (gates the sub-agent so only skillCreator-family sessions can spawn it)
4. Symlinks the parser into the project-scope scripts dir

Run bash scripts/install.sh --dry-run to preview without changes. bash scripts/install.sh --help for usage.

Core Loop: DRAFT → EVAL → RED → GREEN → SMOKE → SHIP

Every skill change goes through this pipeline. No exceptions.

1. DRAFT — Understand the Skill

• Read the SKILL.md or MCP tool definition completely
• brain_search for: prior evals, user complaints, known issues, related skills
• Identify: what does this skill claim to do? What's the baseline without it?
• Document intent, inputs, outputs, edge cases
• Classify: capability uplift (may become obsolete) vs encoded preference (durable)

2. EVAL DESIGN — Create Structured Test Cases

Design eval cases as structured JSON in evals/evals.json:

{
  "id": 1,
  "name": "descriptive-kebab-name",
  "category": "compliance|structure|quality",
  "description": "What this eval tests",
  "prompt": "The input scenario given to the agent",
  "assertions": [
    {
      "type": "tool_usage|content|negative",
      "name": "assertion-name",
      "description": "What correct behavior looks like"
    }
  ]
}

Cover: happy path, edge cases, failure modes, interaction with other skills. See references/scoring-rubric.md for scoring methodology.

3. RED — Baseline Measurement (without skill)

• Run each eval WITHOUT the skill loaded
• Score baseline performance across all eval cases
• Document baseline scores
• Gate: If baseline >70%, the skill may not be adding value — flag it

4. GREEN — With-Skill Measurement

• Run each eval WITH the skill loaded
• Score delta between with_skill and without_skill
• Gate: Delta <10% = marginal, consider retiring. Delta >30% = clearly valuable.
• Maximum 3 iterations before flagging for human review

5. SMOKE — Live Agent Testing

Two tiers based on skill importance:

Tier A: Static Smoke (all skills)

• Review evals.json assertions manually
• Verify no contradictions between skill instructions and assertions
• Check that strongest discriminators are tested

Tier B: Live Agent Smoke (flagship skills only)

• Run live-eval-runner.sh for top 3 discriminator evals
• Agent routing: Claude skills → Sonnet, code skills → Codex, audit skills → Cursor
• Compare captured output against assertions
• Gate: Live delta must be within 15% of static delta
• Results stored in evals/results/live-{date}.json and brain_store'd

See workflows/live-eval.md for the full live eval workflow.

6. SHIP — Package and Document

• Ensure every skill ships with its evals (no eval = no ship)
• brain_store eval results, delta scores, issues found
• Update skill metadata (version, last-eval-date, compliance-score)
• Follow workflows/create-skill.md and workflows/audit-skill.md for SKILL.md structure compliance before shipping structural changes.
• REGISTER a NEW skill — committed ≠ installed. A new golem-powers skill is invisible to every agent until it's symlinked into ~/.claude/skills/<name> (never ~/.claude/commands/; run golem-install, which auto-discovers + links every golem-powers/*/ dir; or symlink the one new skill). Then VERIFY it appears in the available-skills list — committing/merging does NOT register it. (2026-05-30: /weave was committed + merged but unusable by any agent until the symlinks existed. See workflows/create-skill.md "Final Step: REGISTER".)

Declaring Required Environment Variables

Add optional requires: frontmatter only when a skill cannot do its job at all without an environment variable:

requires:
  - FILE_HOST_TOKEN

List variable names (^[A-Z][A-Z0-9_]*$), never secret values. The declaration and runtime behavior are two halves of one contract: lint proves the metadata is well-formed, but the skill body must fail loud when a requirement is missing. Carry this one-line instruction in the body:

If FILE_HOST_TOKEN is unset, tell the user and stop. Do not guess, improvise a fallback, or report partial success.

Eval Methodology

with_skill vs without_skill comparison is MANDATORY. No exceptions.

Eval Result Provenance — Validity Gate

No provenance = no eval. Every static or live eval result MUST record one provenance entry per agent or eval arm with:

model_effective must be the concrete runtime ID, not a floating alias or friendly display label, and uses an ID token with no whitespace. Observation sources begin with the exact case-sensitive labels shown above; append details after :, -, —, or ( when useful.

JSON records use eval-result-provenance.schema.json. Markdown records use an ## Eval Provenance table with these fields plus an agent_or_arm column. Examples inside fenced code blocks are not evidence.

When runtime observation was impossible, the corresponding value and source MUST say NOT DETERMINED; omission, inference, and silent fallback are invalid. This is valid honesty but the checker marks the result NON_COMPARABLE; it cannot support a cross-arm/model delta claim. A NON_COMPARABLE record that contains any score, delta, or positive comparability claim is mechanically INVALID, not merely discouraged by prose. Pre-contract historical results may use provenance: alias-only and are also non-comparable; an alias-only record that carries a score, delta, or positive comparability claim is likewise INVALID. New results may not use that historical escape.

Before publishing or citing a score/delta, run the checker with --require-comparable. Exit 0 means comparable evidence, exit 3 means honest retained history that cannot carry the claim, and exit 1 means invalid input. Run the mechanical provenance check in workflows/audit-skill.md before scoring. An invalid result is not evidence and receives no score, delta, or verdict.

The requested alias is insufficient: cmuxlayer spawn_agent.model has a confirmed defect that silently drops non-alias models while reporting them honored. A requested model records intent, not runtime outcome—the same false surface class as delivered:true or exit 0 without live verification.

Behavioral Compliance Scoring

Scoring Thresholds

Agent Routing for Live Evals

Workflows

References

Integration with Other Skills

• /cmux-agents — spawning live eval agents in cmux panes
• /never-fabricate — Read() every file before reporting results
• /pr-loop — shipping skill changes through the full PR lifecycle

Packaged Sub-Agents (skill-creator family only)

These sub-agents ship in BOTH Claude and Codex formats. They are invokable only from sessions with cwd inside $SKILL_CREATOR_REPO/ (i.e., skillcreatorClaude, skillcreatorCodex, or a repoGolem session rooted there). Sessions in other repos cannot spawn them directly — they must dispatch a skillCreator first.

Canonical source-of-truth lives in this skill's agents/ dir. scripts/install.sh symlinks them into the project repo's project-scope dirs (.claude/agents/ for Claude, .codex/agents/ for Codex). The dual-format packaging means the same sub-agent is available regardless of whether the parent is Claude Code or Codex CLI.

Both formats invoke the same deterministic parser at scripts/session-miner.py. The Claude format uses subagent_type="session-miner" via the Agent tool; the Codex format is referenced by name in natural-language prompts (session_miner, mine X to Y).

For when to choose sub-agent vs skill: see references/subagent-vs-skill.md.

Critical Rules

1. Never claim "tests pass" without running them. Read actual output.
2. Every skill ships with evals. No eval = no ship.
3. brain_search BEFORE starting work — someone may have already evaluated it.
4. brain_store AFTER completing work — results, delta scores, decisions.
5. Max 3 iterations per skill before flagging for human review.
6. Read() every file you cite. No citing from compacted summaries.
7. Sequential skill compounding — build skills one at a time, each learning from the last.