Spec Craft

Spec Craft

LLM-judgment critique of spec quality (proposals + ADRs) against a curated rubric catalog from the spec-quality canon. Per-section critique with rubric-to-section mapping. Second member of the craft-pipeline initiative; highest-leverage craft skill because spec quality compounds across the entire planning → implementation → review lifecycle below it. Emits 3-axis findings (tier × impact × confidence per ADR 0019).

When to Use

• During PR review on a new or substantially-rewritten spec
• After authoring a proposal, before circulating it for review
• When onboarding a new contributor (audit specs they introduced)
• Periodically (per-sprint or per-release) to catch spec drift
• As the cross-cutting spec critic for harness-brainstorming (when newly-authored specs land)
• NOT for spec-structure enforcement (use harness-soundness-review — that's the rule-based floor)
• NOT for README / general-doc critique (use docs-craft)
• NOT for autofix / spec rewriting (this is judgment-only; v1.x may add align-spec sibling)
• NOT for source-code comment critique (use code-craft or docs-craft)
• NOT for RFCs in v1 (v1.x)

Capability Roles

• Defines (Service Definition): the shared craft critique contract (packages/cli/src/shared/craft/) — LlmProvider + finding/axes schema + run store — shared across all *-craft skills. This skill implements, and does not own, that contract.
• Provides (Provider): this skill — a spec/proposal and ADR quality critique implemented over the shared contract (packages/cli/src/spec-craft/).
• Consumes (Consumer): craft-fleet (the craft-pipeline elevation sweep) and the harness natural-language router, which invoke every *-craft provider uniformly through the shared critique/finding shape

Process

Phase 1: DISCOVER — Find spec files

1. Read project configuration. Check harness.config.json for:

   • craft.spec.enabled — gate (default true)
   • craft.spec.maxFiles — doc count cap (default 50)
   • craft.spec.maxSectionsPerFile — per-doc section cap (default 10)

2. Discover spec files:

   • proposals: docs/changes/*/proposal.md and docs/changes/*/<sub>/proposal.md (one level of nesting, for initiatives with sub-projects)
   • ADRs: docs/knowledge/decisions/*.md (excluding README)
   • Restrict via --kinds proposal or --kinds adr for single-kind runs.

Phase 2: PARSE — Split into named sections

For each spec file:

1. Strip YAML frontmatter (--- ... --- block at top, if present).
2. Split by H2 (## ...) into named sections. Each section captures:
   • heading — original H2 text (e.g., "Decisions", "Out-of-scope (v1)")
   • canonical — normalized form for rubric matching ("decisions", "out-of-scope-v1")
   • body — content between this H2 and the next
   • line / endLine — line range
3. Subsections (H3) stay part of the parent H2's body. v1.x may add per-H3 critique for sections like Decisions that have one row per H3.

Phase 3: CRITIQUE — Per (file, section, rubric) loop

7 seed rubrics:

For each eligible (section, rubric) pair:

1. Build prompt with rubric description + spec file path + section heading + section body (truncated to 2000 chars if longer for cost control).
2. LLM returns fenced JSON: null (rubric doesn't apply / section is fine) OR { tier, impact, confidence, message }.
3. On non-null: emit a SpecFinding with cite.rubricId populated for ADR 0020 traceability.

Phase 4: REPORT — Aggregate + cost telemetry

Emit SpecCraftOutput:

{
  findings: SpecFinding[];
  summary: {
    phaseRun: ['critique'];
    durationMs: number;
    llmCalls: { provider, model, count, costUsd };
    catalog: { rubricsApplied: string[] };
    docsScanned: number;
    sectionsScanned: number;
    runId: string;
  }
}

Harness Integration

• harness spec-craft — CLI entry. --files <glob> / --kinds proposal,adr / --sections decisions,scope / --max-files <n> / --max-sections-per-file <n> / --json / --verbose.
• mcp__harness__spec_craft — MCP tool. Same input/output. Consumed by agents.
• Cross-cutting API: critiqueSpecFile(file, opts) exported from packages/cli/src/spec-craft/index.ts. Future craft skills (or harness-brainstorming) can call this when they have a doc in hand without re-walking the project.
• Shared craft infrastructure (extracted on this PR): LlmProvider, MockLlmProvider, derivePriority, 3-axis types all live in packages/cli/src/shared/craft/. design-craft + naming-craft + spec-craft import from there; design-craft + naming-craft keep their old import paths via re-export shims.

Success Criteria

See docs/changes/craft-pipeline/spec-craft/proposal.md for the full 34 success criteria. Highlights:

• 7 seed rubrics ship at catalog/rubrics/<id>.ts (file-per-rubric, matches naming-craft)
• 3-axis output preserved (tier × impact × confidence, never collapsed)
• cite.rubricId populated on every finding (ADR 0020)
• Section parser strips frontmatter; splits by H2 only
• Rubric-to-section mapping skips silently when rubric doesn't apply
• critiqueSpecFile cross-cutting API works on a single file without project walk
• All existing design-craft + naming-craft tests still pass after shared/craft extraction (zero behavior change)

Rationalizations to Reject

These are common rationalizations that sound reasonable but lead to incorrect results. When you catch yourself thinking any of these, stop and follow the documented process instead.

Examples

Example: Vague Decisions section

Input: A proposal's ## Decisions section reads:

| Decision | Why |
|----------|-----|
| Use modern stack | scalable and clean |
| Defer auth | not in scope |

Output (mock LLM):

SPEC-R001 [polish/medium/medium] ## Decisions:34
  "Modern stack" and "scalable and clean" are vague — no concrete framework
  named, no metric for "scalable", no operational definition of "clean".
  Sharpen: name the framework, state the scale target (req/sec, team size),
  define what "clean" means in observable terms.
SPEC-R004 [polish/medium/medium] ## Decisions:34
  The Decisions section pads load-bearing choices with vague qualifiers
  rather than naming the trade-off chosen and the rejected alternative.

Example: Strawmanned Rationalizations

Input: A ## Rationalizations to reject section reads:

| "Use a different framework" | Other frameworks are worse |

Output:

SPEC-R005 [foundational/large/high] ## Rationalizations to reject:88
  "Other frameworks are worse" is a strawman — not stated charitably, not
  paired with a specific reason. Steelman the rejected position: name the
  competing framework, name its strongest feature, then explain the specific
  trade-off that made it unsuitable here.

Example: Empty project — no specs

Input: Project has no docs/changes/ or docs/knowledge/decisions/ directory.

Output:

No spec findings.

Summary: 0 findings across 0 docs (0 sections, 7 rubrics, 0 LLM calls, $0.0000, 3ms)

Gates

• No autofix. Sibling align-spec deferred until signal warrants safe-to-apply rewrites.
• No README / general doc critique. docs-craft territory.
• No source-code comment critique. code-craft / docs-craft territory.
• No B' bootstrap. Same posture as naming-craft v1.
• No graph persistence. Phase 1 MVP.
• No vision/deep mode. Specs are text.
• No structural floor enforcement. harness-soundness-review checks the floor; spec-craft assumes the floor is satisfied and critiques the ceiling.

Escalation

• When LLM cost is too high: drop maxSectionsPerFile to 5 or maxFiles to 25. Per-doc cost = sections × rubrics × per-call. Rubric-to-section mapping already prunes most calls; further: use --sections decisions to target the highest-value section.
• When a rubric produces high false-positive rate: v1 has no per-rubric disable; v1.x adds craft.spec.disabledRubrics: ['SPEC-R007']. Until then: filter findings by cite.rubricId in your consumer.
• When a spec has intentionally aspirational vagueness (e.g., a manifesto-style Overview): SPEC-R001 will flag it; low-confidence findings are de-emphasized per ADR 0019. v1.x adds per-section opt-out via <!-- spec-craft:skip --> HTML comment.
• When you want a doc-level summary instead of per-section findings: v1 is per-section only; v1.x adds a --mode doc opt-in for whole-doc critique.
• When you want to critique RFCs: v1.x. For now, point --files <rfc.md> at a single file — the section parser works on any markdown.

Status

v1 — in implementation. See:

• Spec: docs/changes/craft-pipeline/spec-craft/proposal.md
• Roadmap entry: part of the craft-pipeline initiative (the highest-leverage craft skill)
• Sibling craft skills: harness-design-craft (design-pipeline), naming-craft (craft-pipeline)
• Shared infrastructure: packages/cli/src/shared/craft/ (extracted on this PR)
• Future: align-spec (FIX side), docs-craft, test-craft, code-craft — each can call critiqueSpecFile if they want spec-level critique for a doc they're already processing.