Spec Format Experiment

Spec Format Experiment: GWT vs EARS+Properties

Note: This skill is a manual experiment protocol, not automated tooling. The operator runs each step manually by dispatching subagents with the reference prompts and recording results. Future iteration: could be automated with an orchestrator skill.

Purpose

Compare two behavioral spec formats for AI coding agents:

• GWT (Given-When-Then): Scenario-based, verbose, explicit test data
• EARS+Properties (Easy Approach to Requirements Syntax + universal invariants): Requirement-based, compact, invariant-focused

Both formats sit on top of the same contract-as-code (compiled C# DTOs). Only the behavioral spec format varies.

Hypothesis

• H0: GWT and EARS+properties produce equivalent results
• H1: EARS+properties is more token-efficient with equal or better completeness and implementer success

Commands

Protocol

Phase 1: Generate Specs

For each feature (1-5), for each format (GWT, EARS):

1. Load the spec writer prompt from references/gwt-spec-writer-prompt.md or references/ears-spec-writer-prompt.md
2. Load the feature description from references/feature-descriptions.md
3. Dispatch a spec writer subagent (Opus) with the prompt + feature + contracts
4. Save output to _docs/spec-format-experiment/specs/feature-{N}-{format}-run{R}.md
5. Repeat 3x per feature per format for consistency measurement
6. Use run 1 as the primary spec for implementation

Phase 2: Score Specs

For each of the 10 primary specs:

1. Load the scoring rubric from references/scoring-rubric.md
2. Load the completeness checklist for the feature from references/feature-descriptions.md
3. Score: completeness (binary per item), precision (binary per assertion), token count
4. Save scores to _docs/spec-format-experiment/results/raw-scores.md

Phase 3: Implement From Specs

For each of the 10 primary specs:

1. Create a fresh project scaffold with the feature's Contracts/
2. Dispatch saurun:backend-implementer with:
   • The spec (GWT or EARS format)
   • The compiled contract types
   • Standard conventions from CLAUDE.md
3. Run 2 trials per spec (20 total dispatches)
4. After each trial, measure:
   • dotnet build pass? (yes/no)
   • Contract compliance (curl each endpoint, check status + response shape)
   • Auth boundary correctness
5. Save results to _docs/spec-format-experiment/results/impl-results.md

Phase 4: Verdict

Load all scores. Apply weights:

• Completeness: 30%
• Precision: 20%
• Token efficiency: 15%
• Implementer success: 25%
• Consistency: 5%
• Readability: 5%

Minimum Viable Experiment

If full 5-feature run is too expensive, use 3-feature subset:

• Feature 1 (baseline CRUD)
• Feature 3 (auth boundaries)
• Feature 4 (state machine)

This covers key differentiators in ~60% of cost.

Token Budget

Bias Mitigation

• Both prompts frozen before experiment starts — no revision during
• Completeness scoring is binary (covers or doesn't) — no "close enough"
• 2 implementation trials per spec for non-determinism control
• 3 spec generation runs for consistency measurement
• Features alternate dispatch order (Feature 1 GWT first, Feature 2 EARS first)

Reference Files

• references/gwt-spec-writer-prompt.md — GWT format spec writer prompt
• references/ears-spec-writer-prompt.md — EARS+properties format spec writer prompt
• references/feature-descriptions.md — All 5 test features with completeness checklists
• references/scoring-rubric.md — Full scoring rubric with weights
• references/experiment-protocol.md — Detailed step-by-step execution guide