# Evaluate

> Run the application and verify sprint contract criteria via API tests, Playwright interaction, and schema validation.

- Skill: `cwijayasundara/evaluate-3` (Agent Skill, multi-file: 7 files)
- Install (CLI): `npx skillmds@latest add cwijayasundara/evaluate-3`
- Raw SKILL.md: https://api.skillmd.com/api/skills/cwijayasundara/evaluate-3/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: cwijayasundara (https://skillmd.com/u/cwijayasundara)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/cwijayasundara/evaluate-3

---


# Evaluate Skill

Verify that the implemented group meets all sprint contract criteria by running live checks against the application: API calls, Playwright browser interaction, and schema validation.

> **Ultracode tip:** Leave ultracode **off** here (`/effort high`). Evaluation is deterministic three-layer verification against a fixed contract — there is nothing to fan out.

---

## Usage

```
/evaluate C
```

Evaluates group C's sprint contract. The group ID matches a node in `specs/stories/dependency-graph.md` and a file at `sprint-contracts/{group}.json`.

---

## Prerequisites

Before running `/evaluate`, verify:

- `sprint-contracts/{group}.json` exists and is valid JSON.
- `project-manifest.json` exists with `api_base_url`, `ui_base_url`, and `health_check` fields.
- Docker stack is expected to be running. If it is not, the health check in Step 4 will catch this and produce a FAIL.

---

## Execution Steps

### Step 1 — Load Evaluation Patterns

Read the stack-matched verification reference under `.claude/skills/evaluate/references/` (see the References section below) for the rigor that applies to this project's stack, plus any custom assertion helpers.

### Step 2 — Load Sprint Contract

Read `sprint-contracts/{group}.json`. The contract contains:
- `api_checks`: list of HTTP endpoint checks.
- `playwright_checks`: list of browser interaction sequences.
- `design_checks`: list of visual and component checks (evaluated in Full mode only).
- `architecture_checks.files_must_exist`: list of file paths that must be present on disk.
- `features`: list of feature IDs this group satisfies.

### Step 3 — Load Project Manifest

Read `project-manifest.json`. Extract:
- `api_base_url` — base URL for all API checks (e.g., `http://localhost:8000`).
- `ui_base_url` — base URL for Playwright navigation (e.g., `http://localhost:3000`).
- `health_check` — path to the health endpoint (e.g., `/health`).

### Step 4 — Verify Docker Stack

Run a health check to confirm the application is live:

```
curl --retry 5 --retry-delay 3 -sf {api_base_url}{health_check}
```

If the health check fails after 5 retries, immediately record a FAIL with `failure_layer: "docker"` and stop. Do not proceed to API or Playwright checks. A broken stack is not a partial pass.

---

## Layer 1 — API Checks

For each entry in `api_checks`:

1. Execute the request via Bash:
   ```
   curl -s -w '\n%{http_code}' -X {method} {api_base_url}{path}
   ```
   Include `-H` headers and `-d` body as specified in the check entry.

2. Parse the response: the last line is the HTTP status code; everything before it is the response body.

3. Verify status code matches `expect.status`. A mismatch is a FAIL for this check.

4. Verify the response body contains every string listed in `expect.body_contains`. A missing string is a FAIL for this check.

5. If the check entry contains a `schema_ref` field, validate the response body against the schema:
   ```
   python3 -c "
   import json, jsonschema, sys
   body = json.loads(sys.stdin.read())
   schema = json.load(open('specs/design/api-contracts.schema.json'))
   ref = schema['{schema_ref}']
   jsonschema.validate(body, ref)
   print('schema valid')
   " <<< '{response_body}'
   ```
   A schema validation error is a FAIL for this check.

Record each check as PASS or FAIL with the actual vs. expected values.

### Debugging API Failures

Before reporting an API check as FAILED, read the server logs:
```bash
docker compose logs backend --tail=50 2>&1
```
Include the relevant error from the logs in the failure report. This gives the generator the actual stack trace, not just "got 500 instead of 200."

---

## Performance Checks

For each `performance_checks` entry in the contract:
```bash
# Measure response time
time_ms=$(curl -s -o /dev/null -w "%{time_total}" -X {method} {api_base_url}{endpoint} | awk '{printf "%.0f", $1 * 1000}')
```
If `time_ms > max_response_time_ms`, report as WARN (not BLOCK — performance is advisory unless critical).

---

## Layer 2 — Playwright Checks

Selector and assertion rules are single-sourced in `.claude/skills/evaluate/references/playwright-patterns.md` (the canonical Playwright reference, shared with `/test`). The rules below are a summary — defer to that file if they differ.

For each entry in `playwright_checks`:

1. Use Playwright MCP tools to execute the interaction sequence:
   - `browser_navigate` — navigate to a URL.
   - `browser_click` — click an element (use `getByRole`, `getByText`, or `getByLabel`; never CSS selectors).
   - `browser_fill_form` — fill form fields.
   - `browser_snapshot` — capture the DOM snapshot for assertion.

2. Execute each step in the order specified. Do not reorder or skip steps.

3. Verify each assertion listed in the check entry:
   - Element visible: confirm the element appears in the snapshot.
   - Text matches: confirm the exact or partial text is present.
   - URL: confirm `browser_navigate` landed on the expected path.

4. Use `expect().toBeVisible()` for visibility assertions. Never use `waitForTimeout()` — if an element is not immediately visible, the check fails.

5. Record each check as PASS or FAIL with a description of what was asserted and what was found.

---

## Layer 3 — Design Checks (Full Mode Only)

Skip this layer entirely in **Lean** mode (the design-critic runs once at group end instead).

In Full mode, delegate to the `design-critic` agent:
- Pass the list of `design_checks` entries from the sprint contract.
- Pass the `ui_base_url`.
- The design-critic returns PASS/FAIL per check with visual evidence (screenshots or snapshots).

Record the design-critic's verdicts as-is. Do not override them.

---

## Architecture Checks

For each path listed in `architecture_checks.files_must_exist`:

- Verify the file exists on disk at the given path.
- If the file does not exist, record a FAIL with the missing path.

This check does not require Docker to be running.

---

## Layer 4 — Security Gate

The validator is not security-complete without this layer. Run it in Full and Lean modes — every mode runs the security gate.

1. Spawn the `security-reviewer` agent against the group's changed files (run it concurrently with Layers 1–2 when possible — it does not need the app running).
2. The agent writes `specs/reviews/security-verdict.json` (`{ pass, block_severities, summary, findings[] }`). Read it.
3. Determine the blocking threshold: use the sprint contract's `contract.security_checks.block_severities` if present, else the default `["critical", "high"]`.
4. The security gate **FAILs** if any finding's `severity` is in the blocking set (equivalently, `security-verdict.json#pass === false`). Medium/low findings are WARN/INFO — record them, do not fail on them.
5. If `security-verdict.json` is missing, treat that as a FAIL with `failure_layer: "security"` and reason `"security-reviewer did not produce a verdict"` — a missing scan is not a pass.

This layer does not require Docker. It is independent of the app being reachable.

---

## Update features.json

After all checks complete, update `features.json` for every feature ID listed in the sprint contract's `features` array:

- `passes`: `true` if all checks for that feature passed, `false` otherwise.
- `last_evaluated`: current timestamp in ISO 8601 format.
- `failure_reason`: `null` if passing; otherwise a human-readable description of the first failure (e.g., `"GET /users/1 returned 404, expected 200"`).
- `failure_layer`: `null` if passing; otherwise one of `"api"`, `"playwright"`, `"design"`, `"unit_test"`, `"docker"`, `"security"`.

Do not remove existing fields from `features.json`. Merge the updates into the existing structure.
Only update evaluation state fields: `passes`, `last_evaluated`, `failure_reason`, and `failure_layer`. Preserve immutable feature identity/specification fields such as `id`, `category`, `story`, `group`, `description`, and `steps`.

---

## Write Evaluator Report

Write the full evaluation report to `specs/reviews/evaluator-report.md`:

```markdown
# Evaluator Report — Group {group}

Date: {ISO 8601 timestamp}
VERDICT: PASS | FAIL

## API Checks

- [PASS] POST /users → 201 ✓
- [FAIL] GET /users/1 → expected 200, got 404
- [PASS] DELETE /users/1 → 204 ✓

## Playwright Checks

- [PASS] Upload page renders ✓
- [FAIL] Submit button not clickable
- [PASS] Success message visible after form submit ✓

## Design Checks

- [PASS] Button uses primary color token ✓
- [SKIP] Design checks skipped (Lean mode)

## Architecture Checks

- [PASS] All expected files exist ✓
- [FAIL] Missing: src/repository/user-repository.ts

## Security Gate

- [FAIL] VULN-001 (high): SQL injection in src/api/users.ts:47
- block: 1, warn: 2, info: 0 → gate FAIL

## Features Updated

- F001: PASS
- F002: FAIL (api: GET /users/1 expected 200, got 404)
- F003: PASS
```

The overall VERDICT is PASS only if every check across all layers passes **and** the security gate passes (`security-verdict.json#pass === true`). A single FAIL in any layer — including an open BLOCK (critical/high) security finding — produces a FAIL verdict.

---

## Mode Behavior

| Mode  | Layer 1 (API) | Layer 2 (Playwright) | Layer 3 (Design) | Layer 4 (Security) |
|-------|--------------|---------------------|-----------------|--------------------|
| Full  | Run          | Run                 | Run             | Run                |
| Lean  | Run          | Run                 | Skip            | Run                |

Determine the current execution mode from `project-manifest.json` field `execution.default_mode` (`full`/`lean`), or the `--mode` override when invoked under `/auto` or `/build`. Default to Full if absent. Note: this is distinct from `verification.mode` (`docker`/`local`/`stub`), which controls how the app is reached — do not confuse the two.

---

## Gotchas

- **Never skip a check:** Every entry in the sprint contract must be evaluated. Skipping a check to make the verdict green is not acceptable.
- **Never rationalize failures:** If the API returns 404 and the contract expects 200, that is a FAIL — not a "known issue" or "works on my machine." Record it as a FAIL.
- **Use getByRole, not CSS:** Playwright checks must use semantic locators (`getByRole`, `getByText`, `getByLabel`). CSS selectors break with minor UI changes and are not permitted.
- **Use expect().toBeVisible(), not waitForTimeout():** Arbitrary timeouts hide real failures. If an element does not appear immediately, the check fails.
- **Docker won't start — that's a FAIL:** If the stack is unhealthy, record `failure_layer: "docker"` and stop. Do not attempt workarounds or partial evaluations.
- **Do not modify sprint contracts:** The contract is a read-only input. If the contract appears wrong, report it; do not edit it to make checks pass.

---

## References

The evaluation reference pack (read by the evaluator and design-critic agents) lives under `.claude/skills/evaluate/references/`:

| File | Contents |
|------|----------|
| `references/contract-schema.json` | Sprint contract JSON schema |
| `references/scoring-rubric.md` | Design scoring rubric (4 criteria, weights, exemplars) |
| `references/scoring-examples.md` | Calibration anchors (score 5, 7, 9) — read before scoring |
| `references/playwright-patterns.md` | Selector patterns and assertion patterns for Layer 2 (canonical Playwright reference, shared with `/test`) |
| `references/verify-python.md` | Deep Python verification rigor (pytest/mypy/ruff, traceback parsing, FastAPI/async) — load when the backend is Python |
| `references/verify-react.md` | Deep React/TS verification rigor (build/tsc/vitest, browser/console/hydration signals) — load when the frontend is React/TS |

## Evaluator Behavioral Rules

These rules are non-negotiable. Deviation invalidates the evaluation.

1. **Execute every check.** Do not skip a check because a related check passed.
2. **Never rationalize a failure.** If the check specifies `status: 200` and you get `201`, that is a FAIL.
3. **Evidence over opinion.** Every verdict must cite specific output: response body, screenshot path, line number.
4. **No partial credit on binary checks.** API and Playwright checks are pass/fail.
5. **Design scores are evidence-based.** Cite what you observed, not what you assumed.
6. **Do not infer intent.** If the contract says check X and X is absent, the check fails.
7. **Run checks in order.** Layer 1 before Layer 2 before Layer 3, then the Layer 4 security gate.
8. **Document every check result,** even passing ones.
9. **Security is a gate, not advice.** The overall verdict is FAIL if `specs/reviews/security-verdict.json#pass` is false (any critical/high finding). A functional pass with an open BLOCK vulnerability is still a FAIL. The `security-guidance` plugin is advisory and does not satisfy this gate — the `security-reviewer` agent does.