Evaluate

Evaluate Skill

Verify that the implemented group meets all sprint contract criteria by running live checks against the application: API calls, Playwright browser interaction, and schema validation.

Ultracode tip: Leave ultracode off here (/effort high). Evaluation is deterministic three-layer verification against a fixed contract — there is nothing to fan out.

Usage

/evaluate C

Evaluates group C's sprint contract. The group ID matches a node in specs/stories/dependency-graph.md and a file at sprint-contracts/{group}.json.

Prerequisites

Before running /evaluate, verify:

• sprint-contracts/{group}.json exists and is valid JSON.
• project-manifest.json exists with evaluation.api_base_url, evaluation.ui_base_url, and evaluation.health_check fields.
• Docker stack is expected to be running. If it is not, the health check in Step 4 will catch this and produce a FAIL.
• The Playwright MCP browser tools (mcp__plugin_playwright_playwright__browser_*) are available. If the contract has playwright_checks or design_checks and the tools are missing, do NOT silently skip those layers or improvise with curl: write VERDICT: FAIL with failure_layer: infrastructure and the fix Enable "playwright@claude-plugins-official": true in .claude/settings.json enabledPlugins, restart Claude Code, then re-run /evaluate.

Execution Steps

Step 1 — Load Evaluation Patterns

Read the stack-matched verification reference under .claude/skills/evaluate/references/ (see the References section below) for the rigor that applies to this project's stack, plus any custom assertion helpers.

Step 2 — Load Sprint Contract

Read sprint-contracts/{group}.json. The contract contains:

• api_checks: list of HTTP endpoint checks.
• playwright_checks: list of browser interaction sequences.
• design_checks: list of visual and component checks (evaluated in Full mode only).
• architecture_checks.files_must_exist: list of file paths that must be present on disk.
• features: list of feature IDs this group satisfies.

Every runtime check should include matrix_ids from specs/test_artefacts/verification-matrix.json. Missing required matrix coverage is a hard verification failure. When a matrix row declares implementation_paths, regenerate evidence after those production files change; stale evidence older than an implementation_paths file must not clear the executed gate.

Step 3 — Load Project Manifest

Read project-manifest.json. Extract:

• evaluation.api_base_url — base URL for all API checks (e.g., http://localhost:8000).
• evaluation.ui_base_url — base URL for Playwright navigation (e.g., http://localhost:3000).
• evaluation.health_check — path to the health endpoint (e.g., /health).

Step 4 — Verify Docker Stack

Run a health check to confirm the application is live:

curl --retry 5 --retry-delay 3 -sf {evaluation.api_base_url}{evaluation.health_check}

If the health check fails after 5 retries, immediately record a FAIL with failure_layer: "docker" and stop. Do not proceed to API or Playwright checks. A broken stack is not a partial pass.

Layer 1 — API Checks

For each entry in api_checks:

1. Execute the request via Bash:

   curl -s -w '\n%{http_code}' -X {method} {evaluation.api_base_url}{path}
   

   Include -H headers and -d body as specified in the check entry.

2. Parse the response: the last line is the HTTP status code; everything before it is the response body.

3. Verify status code matches expect.status. A mismatch is a FAIL for this check.

4. Verify the response body contains every string listed in expect.body_contains. A missing string is a FAIL for this check.

5. If the check entry contains a schema_ref field, validate the response body against the schema:

   python3 -c "
   import json, jsonschema, sys
   body = json.loads(sys.stdin.read())
   schema = json.load(open('specs/design/api-contracts.schema.json'))
   ref = schema['{schema_ref}']
   jsonschema.validate(body, ref)
   print('schema valid')
   " <<< '{response_body}'
   

   A schema validation error is a FAIL for this check.

Record each check as PASS or FAIL with the actual vs. expected values and the matrix_ids executed.

Debugging API Failures

Before reporting an API check as FAILED, read the server logs:

docker compose logs backend --tail=50 2>&1

Include the relevant error from the logs in the failure report. This gives the generator the actual stack trace, not just "got 500 instead of 200."

Performance Checks

Performance is a regression ratchet, not an absolute-budget gate: a build FAILs only when a read endpoint gets measurably slower than its recorded baseline. Absolute-budget overruns and first-time measurements are WARN — they inform, they don't block. This catches "the change doubled latency" without flaking greenfield builds that have no baseline yet.

Step P1 — Resolve the endpoints and budgets

• Endpoints to measure: every endpoint in the contract's api_checks plus any performance_checks entry. Split them by method:
  • Read (GET): safe to sample repeatedly → eligible for the regression ratchet.
  • Write (non-GET): sampling a POST/PUT/DELETE 20× would create 20 records → single-shot budget WARN only, never sampled or ratcheted.
• Budget per endpoint (advisory): the contract's performance_checks[].max_response_time_ms if present, else project-manifest.json → execution.latency_budget_ms.read for GET / .write for non-GET (defaults 300 / 800 ms). If neither exists, skip the budget WARN for that endpoint (the ratchet still applies to reads).
• Regression threshold: execution.latency_budget_ms.regression_pct if set, else the perf-baseline.js default of 50(%).

Step P2 — Read endpoints: regression ratchet

Run the existing baseline tool against the read endpoints, keeping the loop's baseline separate from the brownfield "measure-first" baseline:

READS="/items,/items/1,/health"   # comma-joined GET paths from the contract
BASELINE=specs/reviews/perf-baseline.json

if [ -f "$BASELINE" ]; then
  # compare current latency to the recorded baseline; exit 1 = p95 regression
  node .claude/scripts/perf-baseline.js --compare --endpoints "$READS" --out "$BASELINE"
  PERF_STATUS=$?
else
  # first evaluation: establish the baseline, do not fail on it
  node .claude/scripts/perf-baseline.js --endpoints "$READS" --out "$BASELINE"
  PERF_STATUS=2   # treat "baseline just created" as WARN, not FAIL
fi

Interpret PERF_STATUS:

• 1 (REGRESSION) → performance FAIL. Record failure_layer: "performance" with the endpoint and the p95 before -> after (+N%) line from the tool's stdout. This flips the overall verdict to FAIL.
• 2 (no baseline / just captured) → WARN only. Note "performance baseline established — ratchet active next evaluation." Do not fail.
• 0 (OK) → pass. Still emit a WARN for any read endpoint whose measured p95 exceeds its resolved budget (slow, but not a regression).

Record each performance entry with the matrix_ids for the endpoint(s) measured when the contract maps them to matrix rows.

After a PASS verdict for the whole group, refresh the baseline so the ratchet tracks the accepted state:

node .claude/scripts/perf-baseline.js --endpoints "$READS" --out "$BASELINE"   # only on overall PASS

Never refresh the baseline on a FAIL — that would launder a regression into the new normal.

Step P3 — Write endpoints: single-shot budget WARN

For each non-GET endpoint with a resolved budget, take one measurement with --measure (no baseline write, no ratchet) and --samples 1 — the tool skips warmup for non-GET, so this fires exactly one request and won't create extra records:

node .claude/scripts/perf-baseline.js --measure \
  --endpoints "/items" --method POST --body '{...}' --samples 1
# stdout: MEASURE: POST /items p50=910ms p95=910ms p99=910ms (1 samples)

Parse the p95 from the MEASURE: line. If it exceeds the endpoint's resolved budget, record a WARN — never a FAIL. Writes are not ratcheted because a single sample has no meaningful p95 distribution and repeated sampling would mutate state. (A plain curl -s -o /dev/null -w "%{time_total}" is an acceptable fallback if invoking the script per write endpoint is awkward.)

Step P4 — SLO check (when observability.enabled)

After the app is booted (no new boot), run the runtime-SLO sensor:

node .claude/scripts/slo-check.js

It scrapes {api_base_url}{observability.metrics_path} and compares against observability.slo, writing specs/reviews/slo-verdict.json. Fold the verdict:

• verdict: "fail" (5xx error-rate over slo.error_rate_pct, counting only 5xx) → FAIL with failure_layer: "slo" in Full mode; WARN in Lean.
• verdict: "warn" (p95 over slo.p95_ms, or no traffic, or /metrics unreachable) → WARN, never FAIL. p95 regression is owned by the perf ratchet (Step P2), not this check.
• verdict: "disabled" (observability off) → skip silently.

Layer 2 — Playwright Checks

Selector and assertion rules are single-sourced in .claude/skills/evaluate/references/playwright-patterns.md (the canonical Playwright reference, shared with /test). The rules below are a summary — defer to that file if they differ.

For each entry in playwright_checks:

1. Use Playwright MCP tools to execute the interaction sequence:

   • browser_navigate — navigate to a URL.
   • browser_click — click an element (use getByRole, getByText, or getByLabel; never CSS selectors).
   • browser_fill_form — fill form fields.
   • browser_snapshot — capture the DOM snapshot for assertion.

2. Execute each step in the order specified. Do not reorder or skip steps.

3. Verify each assertion listed in the check entry:

   • Element visible: confirm the element appears in the snapshot.
   • Text matches: confirm the exact or partial text is present.
   • URL: confirm browser_navigate landed on the expected path.

4. Use expect().toBeVisible() for visibility assertions. Never use waitForTimeout() — if an element is not immediately visible, the check fails.

5. Record each check as PASS or FAIL with a description of what was asserted, what was found, and the matrix_ids executed.

Accessibility (axe-core) — when the contract has accessibility_checks

Semantic selectors prove an element is reachable; they do not prove the page is accessible. When the contract carries an accessibility_checks block, run an axe-core audit on each page (default: evaluation.ui_base_url; or the block's urls):

1. browser_navigate to the page.
2. Inject and run axe with browser_evaluate — load axe-core (e.g. from the axe-core package or CDN) and return await axe.run().
3. Collect violations; group them by impact (minor / moderate / serious / critical).
4. Verdict: any violation whose impact is in block_impacts (default ["serious", "critical"]) is a failure. In Full mode this FAILs the evaluation (failure_layer: "accessibility"); in Lean mode record it as a WARN. Always list the rule id, impact, offending selector(s), and matrix_ids in the report.

A missing axe run when accessibility_checks.required is true is a FAIL, not a pass — mirror the security-gate rule (a missing scan is never a pass).

Layer 3 — Design Checks (Full Mode Only)

Skip this layer entirely in Lean mode (the design-critic runs once at group end instead).

In Full mode, delegate to the design-critic agent:

• Pass the list of design_checks entries from the sprint contract.
• Pass evaluation.ui_base_url.
• The design-critic returns PASS/FAIL per check with visual evidence (screenshots or snapshots).

The design-critic writes specs/reviews/eval-scores.json with keys design_quality, originality, craft, and functionality; these map 1:1 onto the contract's design_checks keys, and each criterion's score must meet or exceed its min_score in the contract for the check to pass.

Record the design-critic's verdicts as-is. Do not override them.

Architecture Checks

For each path listed in architecture_checks.files_must_exist:

• Verify the file exists on disk at the given path.
• If the file does not exist, record a FAIL with the missing path.

This check does not require Docker to be running.

Layer 4 — Security Gate

The validator is not security-complete without this layer. Run it in Full and Lean modes — every mode runs the security gate.

1. Spawn the security-reviewer agent against the group's changed files (run it concurrently with Layers 1–2 when possible — it does not need the app running).
2. The agent writes specs/reviews/security-verdict.json ({ pass, block_severities, summary, findings[] }). Read it.
3. Determine the blocking threshold: use the sprint contract's contract.security_checks.block_severities if present, else the default ["critical", "high"].
4. The security gate FAILs if any finding's severity is in the blocking set (equivalently, security-verdict.json#pass === false). Medium/low findings are WARN/INFO — record them, do not fail on them.
5. If security-verdict.json is missing, treat that as a FAIL with failure_layer: "security" and reason "security-reviewer did not produce a verdict" — a missing scan is not a pass.

Record security report entries with the matrix_ids for the checks, stories, or endpoints whose boundary was reviewed.

This layer does not require Docker. It is independent of the app being reachable.

Update features.json

After all checks complete, update features.json for every feature ID listed in the sprint contract's features array:

• passes: true if all checks for that feature passed, false otherwise.
• last_evaluated: current timestamp in ISO 8601 format.
• failure_reason: null if passing; otherwise a human-readable description of the first failure (e.g., "GET /users/1 returned 404, expected 200").
• failure_layer: null if passing; otherwise one of "api", "playwright", "design", "accessibility", "unit_test", "docker", "security", "performance".

Do not remove existing fields from features.json. Merge the updates into the existing structure. Only update evaluation state fields: passes, last_evaluated, failure_reason, and failure_layer. Preserve immutable feature identity/specification fields such as id, category, story, group, description, and steps.

Write Evaluator Report

Write the full evaluation report to specs/reviews/evaluator-report.md:

# Evaluator Report — Group {group}

Date: {ISO 8601 timestamp}
VERDICT: PASS | FAIL

## API Checks

- [PASS] POST /users → 201 ✓ (`matrix_ids`: [`VM-001`])
- [FAIL] GET /users/1 → expected 200, got 404 (`matrix_ids`: [`VM-002`])
- [PASS] DELETE /users/1 → 204 ✓ (`matrix_ids`: [`VM-003`])

## Playwright Checks

- [PASS] Upload page renders ✓ (`matrix_ids`: [`VM-004`])
- [FAIL] Submit button not clickable (`matrix_ids`: [`VM-005`])
- [PASS] Success message visible after form submit ✓ (`matrix_ids`: [`VM-006`])

## Design Checks

- [PASS] Button uses primary color token ✓ (`matrix_ids`: [`VM-007`])
- [SKIP] Design checks skipped (Lean mode)

## Architecture Checks

- [PASS] All expected files exist ✓ (`matrix_ids`: [`VM-008`])
- [FAIL] Missing: src/repository/user-repository.ts (`matrix_ids`: [`VM-009`])

## Security Gate

- [FAIL] VULN-001 (high): SQL injection in src/api/users.ts:47 (`matrix_ids`: [`VM-002`])
- block: 1, warn: 2, info: 0 → gate FAIL

## Performance

- [PASS] GET /items p95 41ms -> 44ms (+7%) — within ratchet (`matrix_ids`: [`VM-010`])
- [FAIL] GET /items/1 p95 38ms -> 210ms (+452%) — regression (`matrix_ids`: [`VM-011`])
- [WARN] POST /items 910ms > 800ms write budget (single-shot, not ratcheted) (`matrix_ids`: [`VM-012`])
- [WARN] No baseline existed — established; ratchet active next evaluation

## Features Updated

- F001: PASS
- F002: FAIL (api: GET /users/1 expected 200, got 404)
- F003: PASS

The overall VERDICT is PASS only if every check across all layers passes, the security gate passes (security-verdict.json#pass === true), the performance ratchet reports no read-endpoint regression, and required matrix coverage is present. A single FAIL in any layer — an open BLOCK (critical/high) security finding, a p95 latency regression beyond threshold, or missing required matrix coverage — produces a FAIL verdict. Performance WARNs (over budget, or first-build baseline established) do not affect the verdict.

Mode Behavior

(Accessibility runs only when the contract carries an accessibility_checks block; in Lean mode its violations are WARN, in Full mode blocking impacts FAIL.)

Determine the current execution mode from project-manifest.json field execution.default_mode (full/lean), or the --mode override when invoked under /auto or /build. Default to Full if absent. Note: this is distinct from verification.mode (docker/local/stub), which controls how the app is reached — do not confuse the two. When a regression run uses --replay (G34/G36), boot the app with HARNESS_TEST_REPLAY=1 so its DB/HTTP/LLM resolve to the recorded boundary-doubles and a missing fixture is a hard fail.

Gotchas

• Never skip a check: Every entry in the sprint contract must be evaluated. Skipping a check to make the verdict green is not acceptable.
• Never rationalize failures: If the API returns 404 and the contract expects 200, that is a FAIL — not a "known issue" or "works on my machine." Record it as a FAIL.
• Use getByRole, not CSS: Playwright checks must use semantic locators (getByRole, getByText, getByLabel). CSS selectors break with minor UI changes and are not permitted.
• Use expect().toBeVisible(), not waitForTimeout(): Arbitrary timeouts hide real failures. If an element does not appear immediately, the check fails.
• Docker won't start — that's a FAIL: If the stack is unhealthy, record failure_layer: "docker" and stop. Do not attempt workarounds or partial evaluations.
• Do not modify sprint contracts: The contract is a read-only input. If the contract appears wrong, report it; do not edit it to make checks pass.

References

The evaluation reference pack (read by the evaluator and design-critic agents) lives under .claude/skills/evaluate/references/:

Evaluator Behavioral Rules

These rules are non-negotiable. Deviation invalidates the evaluation.

1. Execute every check. Do not skip a check because a related check passed.
2. Never rationalize a failure. If the check specifies status: 200 and you get 201, that is a FAIL.
3. Evidence over opinion. Every verdict must cite specific output: response body, screenshot path, line number.
4. No partial credit on binary checks. API and Playwright checks are pass/fail.
5. Design scores are evidence-based. Cite what you observed, not what you assumed.
6. Do not infer intent. If the contract says check X and X is absent, the check fails.
7. Run checks in order. Layer 1 before Layer 2 before Layer 3, then the Layer 4 security gate, then the performance ratchet (it needs the app warm and the functional checks already exercised).
8. Document every check result, even passing ones.
9. Security is a gate, not advice. The overall verdict is FAIL if specs/reviews/security-verdict.json#pass is false (any critical/high finding). A functional pass with an open BLOCK vulnerability is still a FAIL. The security-guidance plugin is advisory and does not satisfy this gate — the security-reviewer agent does.