Spec Evaluate

spec-evaluate — Acceptance Test Runner

Execute the acceptance test plan (test.md) against the built feature, capture evidence, and report requirement-level pass/fail as findings that feed back into the implementation loop.

Language Rules

1. Auto-detect input language → output in the same language
2. Japanese input → Japanese output, use the *.ja.md reference files
3. English input → English output, use the *.md reference files
4. Explicit override takes priority

The evaluator instruction sheet is references/evaluator-prompt.md (.ja.md for Japanese). The result file format is references/result-format.md (.ja.md for Japanese).

Options

Core Principle — Evidence Over Self-Report

An evaluator claiming "I tested it and it passed" is not trusted. Every passing case must point to real evidence on disk (a screenshot, a command log, or a verified artifact). Before a result is accepted, the evidence pointers in it are checked for existence: a case marked PASS whose referenced evidence file does not exist is forced to FAIL, regardless of what the evaluator reported. See §Evidence Rules and references/result-format.md.

Execution Flow

The runner (SKILL.md) is a thin driver: it prepares inputs, dispatches the single evaluator instruction sheet to the chosen backend, then machine-verifies the returned result. It does not itself decide pass/fail from memory.

Step 1: Load Inputs

1. Read {spec}/test.md. If missing → stop with an error (nothing to run).
2. Parse each case: ID (T-Axx), requirement ID, Steps, Expected, Verify method (playwright / command / file-check), Command.
3. Read the launch recipe and roles from pipeline.yml (app: and roles.e2e_runner). If pipeline.yml is absent, there is no app recipe and no role assignment — proceed with backend = self and no app launch.

Step 2: Resolve AI Role, Then Execution Backend

Resolution order: --backend flag > roles.e2e_runner from pipeline.yml > self.

The option name --backend is retained for compatibility, but claude and codex select the evaluator AI role. Resolve the vehicle only after that role is selected:

Pipeline callers must pass their recorded host_runtime. A standalone caller that explicitly selects claude or codex must also supply it; if omitted, ask the user to choose the current host and never guess. self bypasses this requirement.

The evaluator instruction sheet is the same one file for every backend; only the execution vehicle changes. See references/execution-backend.md.

Step 3: Determine Round and Prepare Evidence Directory

1. Round number: from --round, else max(existing evaluate-{n}.md) + 1, else 1.
2. Create {spec}/evidence/{round}/. This is where all screenshots and logs go.

Step 4: Run the Evaluator

Compose the runtime context (test.md path, parsed cases, app recipe, evidence directory, round, output path) and hand it to the backend together with references/evaluator-prompt.md. The evaluator:

1. Launches the app from the recipe if any playwright case is present.
2. Executes every case top-to-bottom by its verification method.
3. Saves evidence per case under {spec}/evidence/{round}/.
4. Stops the app.
5. Writes the result file in the references/result-format.md format.

For a UI (playwright) case with no app: recipe available, the evaluator marks the case blocked (skipped + warning), which is distinct from a test failure. See §App Recipe Missing.

Step 5: Machine-Verify Evidence (do not skip)

After the backend returns, the runner independently checks the result file:

1. For every case reported PASS, read its evidence pointer(s).
2. Resolve each pointer relative to {spec}/ and test for existence.
3. Any PASS case with a missing or empty evidence file → rewrite it to FAIL and add a Critical finding noting the missing evidence.
4. Recompute each Evidence Manifest row's sha256/size and compare to the manifest; on a mismatch, rewrite that case to FAIL with a Critical finding.
5. Recompute the Gate line after any such downgrade.

This step is what makes the evidence principle enforceable rather than advisory.

Step 6: Emit Result

Write the finalized result file to --output (default {spec}/evaluate-{round}.md). Report a one-line summary: round, pass/total, gate, and the evidence directory path.

Cross-AI Backend

When the selected evaluator AI role differs from host_runtime, run the evaluator through agent-delegate. Acceptance testing launches the app and drives a browser, so it needs write access — use --mode delegate --sandbox workspace-write, never review mode (review is read-only and cannot launch or operate the app).

1. Write a prompt file that is references/evaluator-prompt.md followed by the runtime context block (Step 4). Before launch, record the exact result path, a caller-generated correlation value, its freshness baseline, and the Step 5 evidence validator; require the evaluator result to carry that correlation.

2. Invoke the delegate script with an explicit target and detached execution. Capture the expected run id and report path, then arm a durable watcher that polls every 15 seconds and never less often than every 30 seconds:

   launch="$(agent-delegate.sh --mode delegate --target "$evaluator_role" \
     --sandbox workspace-write --prompt-file <prompt> \
     --out-dir {spec}/evidence/{round} --detach)"
   expected_run_id="$(printf '%s\n' "$launch" | sed -n 's/^run_id: //p')"
   report="$(printf '%s\n' "$launch" | tail -1)"
   

   Apply the public contract state machine: validate the expected-run report first, then owner, pid, heartbeat, and process state. Re-evaluate at 30-minute intervals and apply the contract's controlled stop at 2 hours; report absence while the run is live is not failure.

3. After a valid terminal report, read status from report.json. If the expected run is blocked with blocker_category: env_error, apply the public contract's fail-closed artifact recovery. Continue only when the predeclared result is fresh, correlated, and passes the normal Step 5 machine checks; retain the blocked report as a runtime diagnostic. Treat every other blocked result, or failed artifact recovery, as an evaluation failure with the blocker recorded, never a silent pass.

Depend only on the agent-delegate public contract (arguments and report.json schema), never on the script's internals. Details: references/execution-backend.md.

Evidence Rules

• Evidence lives in {spec}/evidence/{round}/.
• Naming: T-{id}-*.png for screenshots, T-{id}-*.log for command output and browser console/network logs (e.g. T-A01-login.png, T-A02-latency.log).
• The result file references evidence by relative pointer, never by embedding the content.
• A PASS with a dangling evidence pointer fails machine verification (§Step 5).

App Recipe Missing

A playwright case needs the app: recipe (start command, URL). When it is absent or incomplete:

• Mark the affected cases blocked with a warning, distinct from FAIL.
• Standalone / self: report the blocked cases and the missing recipe fields so the caller can supply them; do not fabricate a pass.
• Inside the pipeline the orchestrator decides escalation (manual asks the human; auto routes blocked to arbitration). spec-evaluate only reports the blocked state faithfully; it does not upgrade blocked to pass.

Error Handling

Usage Examples

# Standalone acceptance test (runner is the evaluator)
/spec-evaluate --spec .specs/user-auth/

# Specific round with an explicit backend override
/spec-evaluate --spec .specs/user-auth/ --round 2 --backend self

# Pipeline use: role follows e2e_runner; host is explicit
/spec-evaluate --spec .specs/user-auth/ --pipeline .specs/pipeline.yml --host-runtime codex