Acceptance Eval

Acceptance Eval

Pre-execution LLM-judgment: does this spec carry measurable, testable, complete acceptance criteria before work begins? Resolves the spec's acceptance section, critiques observability / testability / completeness (advisory criteriaFindings), flags user-visible behaviors with no covering test (advisory coverageFindings), and emits a confidence-rated AcceptanceVerdict (MEASURABLE | NOT_MEASURABLE | INCONCLUSIVE) with a rationale. Merge authority is derived in TypeScript, never trusted from the LLM: a high-confidence NOT_MEASURABLE blocks merge; every other verdict is advisory. The upstream twin of the outcome-eval ship gate — it keeps the downstream gate fed with judgable specs.

When to Use

• On every spec entering the repo under docs/changes/** — triggered on_pr (and manual) — before execution begins.
• When you need a durable, structured answer to "can this spec's success be objectively judged later?"
• NOT for judging whether an implementation satisfied its spec post-execution (use outcome-eval, the downstream twin).
• NOT for authoring acceptance criteria — acceptance-eval reviews; humans own the thinking layer. It never writes the criteria it judges.
• NOT for rule-based floors (lint/architecture/entropy) or other craft ceilings — those run elsewhere.
• NOT when no judgable spec section exists — the verdict degrades to INCONCLUSIVE/advisory and never blocks.

Process

Phase 1: RESOLVE — Find the judgment section

The evaluator resolves the section internally via the fallback chain ## Success Criteria -> ## User-Visible Behavior -> ## Overview (reusing outcome-eval's resolver, not a fork), recording the match in judgedAgainst. No manual action — pass specPath and let AcceptanceEvaluator resolve. If no section is judgable, the verdict is INCONCLUSIVE/advisory.

Phase 2: GATHER — Locate test evidence for coverage (optional)

For responsibility (b), supply test evidence so the judge can flag user-visible behaviors with no covering test: pass testGlobs (globs the tool reads) or testContent (snippets you already read). This evidence is optional: omitting it degrades coverageFindings to advisory-empty but NEVER affects the (c) measurability gate. Gather it when the spec describes user-visible behavior.

Phase 3: JUDGE — Invoke the evaluator

1. Invoke the MCP tool mcp__harness__acceptance_eval with { specPath } plus optional { testGlobs | testContent, model }. The tool constructs AcceptanceEvaluator cli-side and returns the verdict; the supported v1 provider is the anthropic analysis provider (ANTHROPIC_API_KEY).
2. The LLM returns ONLY measurability / confidence / criteriaFindings / coverageFindings / rationale. authority is computed in TypeScript from (measurability, confidence) via deriveAcceptanceAuthority and is never read from the LLM — do not attempt to override it. The tool returns the verdict exactly as the evaluator derives it.
3. The call is degrade-safe: provider failure (incl. no ANTHROPIC_API_KEY), empty test evidence, or a missing judgable section yields INCONCLUSIVE/low/advisory. It never throws and never blocks.

Phase 4: GATE — Render and (conditionally) halt

1. Render the verdict: measurability, confidence, judgedAgainst, rationale, criteriaFindings (a, advisory), and coverageFindings (b, advisory).
2. Authority rule (must match deriveAcceptanceAuthority): authority is blocking iff measurability === 'NOT_MEASURABLE' && confidence === 'high'; every other combination — including all INCONCLUSIVE and MEASURABLE cases, and all medium/low NOT_MEASURABLE — is advisory.
3. On a blocking verdict: HALT before merge. Report the missing measurable criteria and stop; the spec must not merge. Resolution requires a human adding measurable success criteria and re-running acceptance-eval.
4. On an advisory verdict: report the criteriaFindings and coverageFindings for human attention and proceed. Advisory findings do not stop the workflow.

Harness Integration

• mcp__harness__acceptance_eval — MCP tool (the invocation surface). Inputs: specPath (required), testGlobs / testContent (optional (b) evidence), model (optional). The handler builds the cli AnalysisProvider, constructs AcceptanceEvaluator, and returns the AcceptanceVerdict with authority exactly as derived in TypeScript. (No graph persistence in v1 — the seam is documented in the tool header.)
• Evaluator surface: AcceptanceEvaluator, deriveAcceptanceAuthority, acceptanceVerdictSchema, AcceptanceVerdict are exported from @harness-engineering/intelligence. The section resolver is imported from outcome-eval, not duplicated.
• Provider path (v1 supported): the anthropic analysis provider (ANTHROPIC_API_KEY). When no provider is configured the call degrades to INCONCLUSIVE/advisory.
• Relationship to outcome-eval: acceptance-eval is the upstream twin. It guards spec measurability before execution; outcome-eval judges implementation satisfaction after. The same "authority is never read from the LLM" discipline spans both ends of the lifecycle.

Known Limitations

• Coverage findings (b) are heuristic and advisory. Without testGlobs/testContent they are advisory-empty; even with evidence they are LLM-judgment over the behavior section plus located tests, not a graph-backed coverage map (deferred — see spec Decision D4). Heuristic misses are low-harm because (b) never blocks.
• The (c) gate blocks only on high-confidence NOT_MEASURABLE. A spec with weak-but-present criteria yields an advisory verdict, not a block — intentional, to avoid taste-blocks-merge false positives.
• openai-compatible strict mode is not the v1 path; the supported provider is anthropic/claude-cli.

Success Criteria

See docs/changes/harness-pm-persona/proposal.md for the full criteria. This skill satisfies SC1 (skill exists, tier: 2, type: rigid, four-client artifacts), SC6 (blocking halt on high-confidence NOT_MEASURABLE; advisory-and-proceed otherwise), and SC7 (emits advisory criteriaFindings / coverageFindings).

Rationalizations to Reject

These are common rationalizations that sound reasonable but lead to incorrect results. When you catch yourself thinking any of these, stop and follow the documented process instead.

Examples

Example: NOT_MEASURABLE with high confidence (blocks)

Input: a spec whose only "success criteria" are "the feature works well" and "users are happy" — no observable, testable assertions.

Verdict:

measurability:   NOT_MEASURABLE
confidence:      high
judgedAgainst:   success-criteria
authority:       blocking
criteriaFindings:
  - { target: "'works well' / 'users are happy'", message: "No observable assertion — cannot be tested." }
rationale:       "The success section contains only subjective statements; nothing can be judged at outcome time."

Action: HALT before merge. Report the missing measurable criteria; the spec must not merge until a human adds them.

Example: measurable criteria (advisory, proceeds)

Input: a spec whose Success Criteria list concrete, testable assertions; one user-visible behavior lacks an obvious covering test.

Verdict: measurability: MEASURABLE confidence: high authority: advisory, with one coverageFindings {target, message} object (the one user-visible behavior lacking a covering test) surfaced for review. The workflow proceeds.

Gates

• Authority is never read from the LLM. The verdict's authority is always deriveAcceptanceAuthority(measurability, confidence) computed in TypeScript. If you find yourself letting the model assert blocking/advisory, STOP — that defeats the entire purpose of this gate.
• Block only on high-confidence NOT_MEASURABLE. authority === 'blocking' iff measurability === 'NOT_MEASURABLE' && confidence === 'high'. Every other combination is advisory. Do not halt on an advisory verdict.
• Never author criteria. acceptance-eval reviews specs; it does not write the acceptance criteria it judges. Humans own the thinking layer.
• Never block on infrastructure noise. A provider failure, an unparseable response, or a missing spec section must resolve to INCONCLUSIVE/advisory, never a thrown error or a block. The evaluator enforces this; do not reintroduce a hard failure in the wrapper.

Escalation

• Blocking verdict the author disputes: the resolution is for a human to add measurable success criteria (or fix the section the resolver judged) and re-run acceptance-eval — not to override the gate.
• Repeated INCONCLUSIVE on a real spec: usually means no judgable section exists. Confirm the spec has a Success Criteria / User-Visible Behavior / Overview section.
• No ANTHROPIC_API_KEY configured: every verdict degrades to INCONCLUSIVE/advisory and nothing blocks. Surface this to the human — the gate is effectively disabled until a provider is configured.
• Verdict seems wrong (false positive/negative): capture the spec section and verdict and route to the maintainers; do not loosen the conservative-confidence prompt ad hoc.