Test Site

Plugin check: Run node "${PLUGIN_ROOT}/scripts/check-version.js" — if it outputs a message, show it to the user before proceeding.

Test Power Pages Site

Test a deployed, activated Power Pages site at runtime. Navigate the site in a browser, crawl all discoverable links, verify pages load correctly, capture network traffic to test API requests, and generate a comprehensive test report.

Prerequisite: This skill expects a deployed and activated Power Pages site. Run /deploy-site and /activate-site first if the site is not yet live.

Core Principles

• Non-destructive: This skill is read-only — it does not create, modify, or delete any files or data. It only observes the site via the browser.
• API-first testing: The primary goal beyond page loads is verifying that all /_api/ (Web API / OData) requests return successful responses.
• Response-shape discovery: For /_api/serverlogics/ endpoints the test run must also capture and report the actual response body shape so frontend integrations can be written against the real response, not a guessed one. If frontend parsing or field access does not match the observed shape, report the mismatch and describe the parsing or field-access changes needed — this skill does not modify any code.
• User-controlled authentication: Never attempt to log in automatically. Always ask the user to log in via the browser window when authentication is required.
• Bounded crawling: Cap page crawling at 25 pages to prevent infinite loops on sites with dynamic or paginated URLs.

Validation Test Categories

Every run produces a categorized test report (docs/alm/last-test-site.json — see Phase 6.7a). Stable category IDs and the source phase that produces each:

plan-alm's Validation tab consumes this shape directly — each category becomes a collapsible group in the per-stage sub-tab, and the rolled-up runOutcome (passed / passed-with-warnings / failed) drives the green / yellow / red Outcome badge in both the Validation tab and the Execution checklist substep.

Initial request: $ARGUMENTS

Phase 1: Resolve Site URL

Goal: Determine the live URL of the Power Pages site to test.

Actions

1.1 Create Task List

Create the full task list with all 6 phases before starting any work (see Progress Tracking table).

1.2 Check User Input

If the user provided a URL in $ARGUMENTS:

1. Validate it starts with https://.
2. Store it as SITE_URL and skip to Phase 2.

1.3 Auto-Detect from Activation Status

If no URL was provided, attempt auto-detection:

1. Locate the project root by searching for powerpages.config.json:

   **/powerpages.config.json
   

2. Run the activation status check script:

   node "${PLUGIN_ROOT}/scripts/check-activation-status.js" --projectRoot "<PROJECT_ROOT>"
   

3. Evaluate the JSON result:

   • If activated is true and websiteUrl is present: Use websiteUrl as SITE_URL. Inform the user: "Detected your site URL: "
   • If activated is false: Inform the user: "Your site is not yet activated. Please run /activate-site first, then re-run this skill." Stop the skill.
   • If error is present: Fall through to step 1.4.

1.4 Ask the User

If auto-detection failed or was inconclusive, use AskUserQuestion:

Store the user-provided URL as SITE_URL.

Output

• SITE_URL resolved and ready for testing

Phase 2: Launch Browser & Initial Load

Goal: Open the site in a browser, verify the homepage loads, and capture baseline errors.

Actions

2.1 Resize Browser

Set the browser to a standard desktop viewport:

• Use browser_resize with width: 1280, height: 720.

2.2 Navigate to Site

• Use browser_navigate to open SITE_URL.

2.3 Wait for Page Load

• Use browser_wait_for with time: 5 seconds to allow the page to fully render (SPAs may need time for client-side routing and API calls).

2.4 Verify Homepage

• Use browser_snapshot to take an accessibility snapshot.
• Check the snapshot for signs of a working page:
  • Page has meaningful content (not blank, not a generic error page).
  • Look for common error indicators: "404", "Page not found", "500", "Internal Server Error", "This site can't be reached".
• If the page shows an error, report it to the user and ask whether to continue or stop.

2.5 Capture Console Errors

• Use browser_console_messages with level: "error" to check for JavaScript errors on initial load.
• Record any errors found — these will be included in the final report.

2.6 Capture Initial Network Requests

• Use browser_network_requests with includeStatic: false to capture the initial page load API calls.
• Record any /_api/ or OData requests and their status codes for Phase 5 analysis.

Output

• Browser launched at correct viewport size
• Homepage loaded and verified via snapshot
• Initial console errors and network requests recorded
• If the homepage shows a login screen, noted for Phase 3

Phase 3: Authentication Check

Goal: Detect if the site requires authentication and handle login if needed. Power Pages sites can have two layers of authentication:

1. Private site gate — The entire site is private. Navigating to the site redirects to an identity provider (Azure AD B2C, etc.) before any site content is visible. The browser URL will typically change to a different domain (e.g., login.microsoftonline.com, *.b2clogin.com).
2. Site-level authentication — The site is publicly accessible (homepage loads), but certain pages or features require a logged-in user with a specific web role. Indicated by "Sign in" / "Log in" links in the navigation, or pages that show restricted-access messages.

Actions

3.1 Analyze Homepage Snapshot for Private Site Gate

Review the browser snapshot from Phase 2.4 and the current browser URL for signs of a private site redirect:

• The page content shows an identity provider login form (Azure AD B2C, Azure AD, etc.)
• The browser URL has changed to a different domain than SITE_URL (e.g., login.microsoftonline.com, *.b2clogin.com, or a custom identity provider domain)
• A 401/403 response was returned before any site content loaded
• The page is blank or shows "Access denied" / "You do not have access" with no site navigation visible

3.2 Handle Private Site Gate

🚦 Gate (pause · test-site:3.2.private-gate-login): External wait — site redirected to identity provider; skill pauses until user completes login or cancels.

If a private site gate is detected, use AskUserQuestion:

If "I have logged in":

1. Use browser_snapshot to verify the user is now on the actual site (site content visible, navigation present, URL is back on the SITE_URL domain).

2. If still on the identity provider login page:

   🚦 Gate (pause · test-site:3.2.login-retry): Login not yet complete — re-prompt or cancel.

   • Use AskUserQuestion again: "It looks like the login hasn't completed yet. The browser should still be open — please complete the login and try again."
   • Repeat until login is confirmed or user cancels.

3. Once confirmed, re-run Phase 2.5 and 2.6 (capture console errors and network requests on the now-loaded homepage).

4. Continue to step 3.3 to check for site-level authentication.

If "Cancel testing":

• Stop the skill and inform the user they can re-run it after resolving access.

3.3 Analyze for Site-Level Authentication

After the homepage is loaded (either directly for public sites, or after passing the private site gate), review the snapshot for signs of site-level authentication:

• "Sign in" / "Log in" / "Register" links or buttons in the site navigation
• Pages that show "You must be signed in to view this page" or similar messages
• Content that indicates some areas are restricted to authenticated users

3.4 Handle Public Site (No Authentication Needed)

If neither a private site gate nor site-level authentication indicators are found:

• Inform the user: "Site is publicly accessible. Proceeding with page and API testing."
• Skip to Phase 4.

3.5 Handle Site-Level Authentication

🚦 Gate (plan · test-site:3.5.public-vs-auth): Site has Sign-in UI — test as authenticated user, skip auth-gated pages, or cancel.

If site-level authentication indicators are detected (login links in navigation, etc.), use AskUserQuestion:

If "I have logged in":

1. Use browser_snapshot to verify the user is now logged in (login link replaced with user name/profile, or authenticated content is visible).

2. If the login form is still showing:

   🚦 Gate (pause · test-site:3.5.login-retry): Site-level login not yet complete — re-prompt or cancel.

   • Use AskUserQuestion again: "It looks like the login hasn't completed yet. The browser should still be open — please complete the login and try again."
   • Repeat until login is confirmed or user cancels.

3. Create an additional task for testing authenticated scenarios using TaskCreate:

   Task subject	activeForm	Description
   Test authenticated pages and APIs	Testing authenticated scenarios	Re-crawl site as logged-in user, verify auth-gated pages load and authenticated API calls succeed

If "Skip authenticated pages":

• Note that only public pages will be tested. Some API calls may return 401/403 — these will be flagged but not treated as failures.
• Do not create the authenticated testing task.
• Continue to Phase 4.

If "Cancel testing":

• Stop the skill and inform the user they can re-run it after resolving authentication.

Output

• Authentication status resolved for both layers:
  • Private site gate: passed, not needed, or cancelled
  • Site-level auth: logged in, skipped, or not needed
• If authenticated: additional task created for authenticated testing in Phase 5.6

Phase 4: Crawl & Test Pages

Goal: Discover all navigable links on the site and verify each page loads correctly.

Actions

4.1 Discover Links from Current Page

Use browser_evaluate to extract all internal links:

() => {
  const links = Array.from(document.querySelectorAll('a[href]'));
  return links
    .map(a => a.href)
    .filter(href => href.startsWith(window.location.origin))
    .filter(href => !href.includes('#') || href.split('#')[0] !== window.location.href.split('#')[0])
    .map(href => href.split('#')[0])
    .filter((href, i, arr) => arr.indexOf(href) === i);
}

Present the discovered links to the user:

"Found X internal links on the homepage. Testing each page..."

4.2 Test Each Page

For each discovered URL, in sequence:

1. Navigate: Use browser_navigate to go to the URL.
2. Wait: Use browser_wait_for with time: 3 seconds for the page to render.
3. Snapshot: Use browser_snapshot to verify the page rendered content.
4. Check for errors: Look for error indicators in the snapshot (404, 500, blank page, error messages).
5. Console errors: Use browser_console_messages with level: "error" to check for JavaScript errors.
6. Discover new links: Use browser_evaluate (same script as 4.1) to find any new internal links not already in the queue.
7. Record result: URL, status (Pass/Fail), error count, notes.

4.3 Crawl Newly Discovered Links

• Add any newly discovered links from step 4.2.6 to the test queue.
• Continue testing until all links are visited or the 25-page cap is reached.
• If the cap is hit, inform the user: "Reached the 25-page testing limit. Y additional links were discovered but not tested."

4.4 Record Page Test Results

Build a results list tracking:

• URL tested
• Load status (Pass / Fail)
• Number of console errors
• Notes (error messages, blank page, redirect, etc.)

Output

• All discoverable pages crawled (up to 25)
• Pass/fail status recorded for each page
• New links discovered during crawl added to results

Phase 5: Test API Requests

Goal: Capture and analyze all API requests made by the site to verify they are working.

Actions

5.1 Revisit Data-Driven Pages

Navigate back to pages that are likely to make API calls — pages with dynamic content such as data tables, lists, forms, or dashboards. Prioritize pages where /_api/ requests were observed in Phase 2.6 or Phase 4.

For each data-driven page:

1. Use browser_navigate to go to the page.
2. Use browser_wait_for with time: 5 seconds to allow API calls to complete.

5.2 Capture Network Requests

• Use browser_network_requests with includeStatic: false to get all network requests.
• Filter for API requests matching these patterns:
  • /_api/ — Power Pages Web API / OData endpoints
  • /api/ — Custom API endpoints
  • URLs containing odata or $filter, $select, $expand query parameters

5.3 Analyze API Responses

For each captured API request, evaluate:

5.3b Inspect Server Logic Response Shapes

Server logic endpoints (/_api/serverlogics/<name>) commonly return a standard envelope whose data field is typically a string on success, but it is not guaranteed to be one in every response — in failure or edge cases data may be null, absent, a non-JSON string, or another non-string value. When the server logic returns serialized JSON (common), data must be parsed to reveal the actual payload — and that payload itself may be nested further. Frontend code often parses to the wrong level, reads the wrong keys, or treats data as the final object, producing silent UI failures. Test-site must surface the exact observed shape so the frontend can be corrected rather than assuming data is always directly parseable.

For each /_api/serverlogics/ request observed on any tested page:

1. Record the endpoint name from the URL (/_api/serverlogics/<endpointName>), the HTTP method, query string parameters, and HTTP status code.

2. When the request was a GET (no CSRF token required), re-execute it from the browser with browser_evaluate so the full response body is captured. Progressively parse the payload — keep parsing string-typed fields until further parsing fails — and record the shape at each level. Use a script of this form, replacing the URL with the observed one:

   async () => {
     const res = await fetch('<observed-url>', { credentials: 'include', headers: { 'Content-Type': 'application/json' } });
     const status = res.status;
     const text = await res.text();
     const levels = [];
     let current;
     try { current = JSON.parse(text); } catch { return { status, rawSample: text.slice(0, 2000) }; }
     levels.push({ type: Array.isArray(current) ? 'array' : typeof current, keys: current && typeof current === 'object' && !Array.isArray(current) ? Object.keys(current) : null, firstItemKeys: Array.isArray(current) && current[0] && typeof current[0] === 'object' ? Object.keys(current[0]) : null });
     // Progressively parse common nested-string fields (data, Body, body, payload, result)
     const nestedKeys = ['data', 'Body', 'body', 'payload', 'result'];
     while (current && typeof current === 'object') {
       const next = nestedKeys.find(k => typeof current[k] === 'string');
       if (!next) break;
       let parsed;
       try { parsed = JSON.parse(current[next]); } catch { break; }
       levels.push({ parsedFrom: next, type: Array.isArray(parsed) ? 'array' : typeof parsed, keys: parsed && typeof parsed === 'object' && !Array.isArray(parsed) ? Object.keys(parsed) : null, firstItemKeys: Array.isArray(parsed) && parsed[0] && typeof parsed[0] === 'object' ? Object.keys(parsed[0]) : null });
       current = parsed;
     }
     return { status, levels, rawSample: text.slice(0, 2000) };
   }
   

3. For non-GET server logic requests, do not re-execute them (they may mutate data). Rely on the already-captured browser_network_requests entry and report whatever response metadata is available. Note in the report that the body was not re-captured.

4. Build a "Server Logic Response Shapes" section for the Phase 6 report. For each endpoint record: endpoint name, HTTP method, status, the chain of parse levels (what key was parsed at each step, resulting type, keys at that level, and first-item keys when the level is an array), and a raw sample (first ~2000 chars, matching the slice in the script above). If envelope.success === false and envelope.error is present, report the error verbatim.

5. Compare the observed shape to how the frontend actually consumes it (service files, hooks, components found in the repo). If there is a mismatch — e.g. the UI reads envelope.data.value as an object but the actual payload is a string that needs parsing, or expects a field name that doesn't appear in the observed keys — identify the mismatch and recommend the minimal frontend change needed to match the real shape. Be specific about the parsing or field access change required, but do not modify frontend code or create a commit in this skill; leave implementation to a separate editing/remediation skill or phase.

Record the findings and any recommended frontend fixes for the Phase 6 report.

5.4 Provide Actionable Guidance for Failures

For each failed API request, provide specific remediation:

• 401 Unauthorized: "This endpoint requires authentication. If you skipped login in Phase 3, try re-running with authentication. Otherwise, check that the auth token is being passed correctly."
• 403 Forbidden on /_api/ calls: "Check the following:\n 1. Table permissions — Ensure a table permission exists for this table with the correct scope and privileges (Read, Write, etc.) assigned to the appropriate web role.\n 2. Site settings — Verify Webapi/<tablename>/enabled is set to true. The fields value uses LogicalNames for ordinary columns, _<LogicalName>_value for lookup reads, and exact Navigation Properties for relationships written through @odata.bind. For aggregate OData, include every grouping key, aggregate input, filter column, and ordered column.\n 3. Web role assignment — Confirm the authenticated user has the correct web role assigned."
• 404 Not Found: "Verify the entity set name (should be the plural form of the table logical name). Check that the table exists in Dataverse and is published."
• 500 Internal Server Error: "Enable the Webapi/error/innererror site setting (set to true) to get detailed error messages. Redeploy and retest to see the inner error details."

5.5 Test Form Submissions (Optional)

🚦 Gate (consent · test-site:5.5.form-submit): About to submit a form on the live site — may create or modify Dataverse records. Destructive against shared state (the live Dataverse env); requires explicit opt-in.

Trigger: Forms detected via browser_snapshot in Phase 5. Why we ask: Auto-submitting test data into production records pollutes real customer data. Cancel leaves: Nothing — read-only API checks continue from earlier 5.x phases.

If forms are detected on any page (via browser_snapshot showing form elements), ask the user via AskUserQuestion before interacting:

If "Yes":

1. Use browser_click to interact with form submit buttons.
2. Use browser_wait_for to wait for the form response.
3. Use browser_network_requests to capture the resulting POST/PATCH requests.
4. Analyze responses using the same criteria as 5.3.

If "Skip": Continue to Phase 5.6 (or Phase 6 if no authenticated testing task was created).

Output

• All API endpoints discovered and tested
• Pass/fail status with HTTP status codes recorded
• Actionable remediation guidance provided for each failure

5.6 Test Authenticated Scenarios (Only If User Logged In)

Skip this step entirely if the user chose "Skip authenticated pages" in Phase 3.5, or if no site-level authentication was detected in Phase 3.3.

Goal: Re-crawl the site as an authenticated user to discover and test pages and API calls that are only available after login.

Mark the "Test authenticated pages and APIs" task as in_progress.

5.6.1 Discover Authenticated Pages

After login, the site navigation may show additional links that were hidden or restricted for anonymous users (e.g., profile pages, dashboards, admin panels, account management).

1. Navigate back to SITE_URL (homepage).
2. Use browser_snapshot to capture the authenticated navigation.
3. Use browser_evaluate (same link extraction script as Phase 4.1) to discover internal links.
4. Compare against the links already tested in Phase 4. Identify any new links that were not visible before authentication.

If new links are found, inform the user:

"Found X additional pages visible after login that were not accessible anonymously. Testing each page..."

5.6.2 Test Authenticated Pages

For each newly discovered link, follow the same test procedure as Phase 4.2:

1. Navigate, wait, snapshot, check for errors, capture console errors.
2. Record results separately as authenticated page tests.
3. Respect the same 25-page cap (counting pages already tested in Phase 4).

5.6.3 Test Authenticated API Calls

For each authenticated page that makes /_api/ requests:

1. Use browser_network_requests with includeStatic: false to capture API calls.
2. Compare against API calls captured in Phase 5 — identify any new endpoints or endpoints that previously returned 401/403 and now succeed.
3. Analyze responses using the same criteria as Phase 5.3.

5.6.4 Record Results

Record authenticated test results separately so Phase 6 can report them in a distinct section:

• Authenticated pages discovered and tested (count, pass/fail)
• Authenticated API calls (count, pass/fail, any endpoints that changed from fail to pass after login)

Mark the "Test authenticated pages and APIs" task as completed.

Output

• Authenticated pages crawled and tested
• Authenticated API endpoints captured and analyzed
• Results recorded separately for the test report

Phase 6: Generate Test Report

Goal: Present a comprehensive summary of all test results and suggest next steps.

Actions

6.1 Record Skill Usage

Reference: ${PLUGIN_ROOT}/references/skill-tracking-reference.md

Follow the skill tracking instructions in the reference to record this skill's usage. Use --skillName "TestSite".

6.2 Present Page Test Results

Present results in a clear table:

## Page Test Results

| # | URL | Status | Console Errors | Notes |
|---|-----|--------|----------------|-------|
| 1 | /                | Pass | 0 | Homepage loaded successfully |
| 2 | /about           | Pass | 0 | |
| 3 | /products        | Pass | 2 | Minor JS warnings |
| 4 | /admin           | Fail | 1 | 403 Forbidden |

Pages tested: 4/4 | Passed: 3 | Failed: 1

6.3 Present API Test Results

## API Test Results

| # | Endpoint | Method | Status | Notes |
|---|----------|--------|--------|-------|
| 1 | /_api/cr4fc_products       | GET | 200 OK      | 12 records returned |
| 2 | /_api/cr4fc_categories     | GET | 200 OK      | 3 records returned  |
| 3 | /_api/cr4fc_orders         | GET | 403 Forbidden | Missing table permissions |

API endpoints tested: 3 | Passed: 2 | Failed: 1

If no API requests were captured, note: "No API requests (/_api/ or OData) were detected during testing. This site may not use the Web API, or API calls may require specific user interactions to trigger."

6.3b Present Server Logic Response Shapes

If any /_api/serverlogics/ requests were captured in Phase 5.3b, add a dedicated subsection so the frontend integration can be written against the real response. Show the full chain of parse levels — one block per distinct endpoint:

## Server Logic Response Shapes

### /_api/serverlogics/<endpoint-name>  (GET, 200)

- Level 0 (raw body): object, keys: [requestId, success, serverLogicName, data, error]
- Level 1 (parsed from `data`): object, keys: [status, items, count]
- Level 1 `items` property: array; first item keys: [id, name, ...]
- Raw sample: `{"requestId":"...","success":true,"data":"{\"status\":\"success\",\"items\":[{\"id\":\"...\",\"name\":\"...\"}],\"count\":1}", ...}`

Frontend parsing required:
    const level1 = JSON.parse(envelope.data);
    const items  = level1.items;

If a frontend mismatch was detected, list the files and lines that would need to change and the specific parsing/field access correction required — do not apply or commit the change in this skill. This section is the primary deliverable when a developer asks "I don't know what shape my server logic returns — what does the frontend need to do?"

6.4 Present Authenticated Test Results (If Applicable)

If Phase 5.6 was executed, present results in separate tables:

## Authenticated Page Test Results

| # | URL | Status | Console Errors | Notes |
|---|-----|--------|----------------|-------|
| 1 | /profile         | Pass | 0 | User profile loaded |
| 2 | /dashboard       | Pass | 1 | Minor JS warning |
| 3 | /admin/settings  | Fail | 0 | 403 Forbidden — insufficient web role |

Authenticated pages tested: 3 | Passed: 2 | Failed: 1

## Authenticated API Test Results

| # | Endpoint | Method | Status | Notes |
|---|----------|--------|--------|-------|
| 1 | /_api/cr4fc_orders         | GET | 200 OK      | Previously 403 — now accessible after login |
| 2 | /_api/cr4fc_userprofiles   | GET | 200 OK      | Only visible after auth |

Authenticated API endpoints tested: 2 | Passed: 2 | Failed: 0

If no additional pages or APIs were discovered after login, note: "No additional pages or API endpoints were found after authentication. The authenticated user sees the same content as an anonymous visitor."

6.5 Present Overall Summary

## Overall Test Summary

| Category                 | Tested | Passed | Failed | Warnings |
|--------------------------|--------|--------|--------|----------|
| Pages (public)           | 4      | 3      | 1      | 0        |
| Pages (authenticated)    | 3      | 2      | 1      | 0        |
| API Endpoints (public)   | 3      | 2      | 1      | 0        |
| API Endpoints (auth)     | 2      | 2      | 0      | 0        |
| Console Errors           | —      | —      | —      | 2        |

Overall: X/Y checks passed

If authenticated testing was skipped, omit the authenticated rows from the table.

6.6 Present Recommendations

For each failure, reiterate the specific remediation guidance from Phase 5.4. Group recommendations by category:

• Table permissions issues → /create-webroles or manually configure table permissions
• Site settings issues → Check Webapi/<table>/enabled and Webapi/<table>/fields settings
• Authentication issues → /setup-auth
• Missing endpoints → Verify table exists in Dataverse via /setup-datamodel
• Server errors → Enable Webapi/error/innererror site setting for diagnostics

6.7 Close Browser

• Use browser_close to clean up the browser session.

6.7a Write Machine-Readable Report (docs/alm/last-test-site.json)

Always write a structured JSON report so other skills (notably plan-alm) can ingest the run without re-parsing the markdown summary. The file is overwritten on every run. Ensure the docs/alm/ directory exists before writing — node -e "require('fs').mkdirSync('docs/alm',{recursive:true})".

Shape:

{
  "url": "https://contoso.powerappsportals.com",
  "stageName": "Staging",
  "runAt": "2026-04-29T08:50:00.000Z",
  "durationSec": 95,
  "runOutcome": "passed | passed-with-warnings | failed",
  "summary": {
    "critical": 0,
    "high": 1,
    "medium": 0,
    "low": 2,
    "total": 3,
    "automated": 2,
    "manual": 1,
    "passed": 2,
    "failed": 1,
    "skipped": 0
  },
  "categories": [
    {
      "id": "site-load",
      "name": "Site Load",
      "icon": "📦",
      "tests": [
        {
          "id": "t01",
          "name": "Homepage returns 200 OK",
          "severity": "critical | high | medium | low",
          "type": "automated | manual",
          "status": "passed | failed | skipped",
          "description": "Short why-this-matters sentence.",
          "steps": ["GET /", "Expect 200"],
          "expected": "200 OK",
          "actual": "200 OK",
          "validates": "Site activation"
        }
      ]
    }
  ]
}

Category mapping — emit one category per test-site phase that produced findings. Use these stable id values so consumers can recognize them:

Do NOT include a top-level notes string that embeds component counts, version numbers, or other run-specific data from prior deploys. Real-world failure: a last-test-site.json notes field hardcoded "4,037 components" from a deploy two runs ago, then surfaced via plan-alm even though the latest deploy shipped 4,051 components. If the marker needs a freeform narrative, base it strictly on the CURRENT run's data — never carry numbers across runs. The structured summary + categories[] is the audit trail; prose summaries belong in the rendered HTML, not in the JSON marker.

Severity rules (apply per test card):

• HTTP 5xx response → critical
• HTTP 4xx response on a public page or a documented public API → high
• HTTP 4xx response on an authenticated-only resource accessed anonymously → low (expected gate)
• Console errors on an otherwise-passing page → medium
• All other findings (info-only) → low

Computing the summary object — read carefully. The summary buckets MUST be aggregated by counting each test's severity field across all categories[].tests[], not derived from the category itself or defaulted to low. plan-alm's Validation tab and the rendered HTML's severity grid both rely on the summary directly — if every test lands in low regardless of its actual severity, every stage looks healthy even when critical failures exist. Real-world reproduction (Citizens portal, 2026-05-21): four tests with severities critical / high / high / low were dumped into summary: {critical:0, high:0, medium:0, low:4} because the agent skipped per-test counting.

Compute it as:

const summary = { critical: 0, high: 0, medium: 0, low: 0, total: 0, automated: 0, manual: 0, passed: 0, failed: 0, skipped: 0 };
for (const cat of (report.categories || [])) {
  for (const test of (cat.tests || [])) {
    summary.total += 1;
    const sev = String(test.severity || '').toLowerCase();
    if (sev === 'critical') summary.critical += 1;
    else if (sev === 'high') summary.high += 1;
    else if (sev === 'medium') summary.medium += 1;
    else summary.low += 1;            // default for unknown / missing severity
    if (test.type === 'manual') summary.manual += 1; else summary.automated += 1;
    if (test.status === 'failed') summary.failed += 1;
    else if (test.status === 'skipped') summary.skipped += 1;
    else summary.passed += 1;
  }
}

The plan-alm renderer (render-alm-plan.js buildValidationStagePane) draws four severity cards — Critical / High / Medium / Low — independently. Each summary bucket must reflect the actual count for that severity, not a combined "Medium / Low" total.

Status rules:

• passed — assertion held (200, no console errors, expected redirect, etc.)
• failed — assertion did not hold (5xx, 4xx where 200 was expected, login flow broke, etc.)
• skipped — phase or test was deliberately bypassed (e.g. user picked "Skip authenticated pages" in 3.5)

summary is computed from categories:

• critical/high/medium/low — count of tests at each severity, regardless of status (so reviewers see the test surface even when everything passed).
• total — total test cards.
• automated/manual — split by type.
• passed/failed/skipped — split by status.

runOutcome is the rolled-up verdict:

• failed if any test has status: "failed" AND severity: "critical" OR "high".
• passed-with-warnings if no critical/high failures but summary.failed > 0 OR there are console errors logged.
• passed otherwise.

Write the file (Node.js, run from the project root):

node -e "require('fs').mkdirSync('docs/alm',{recursive:true})"
node -e "require('fs').writeFileSync('docs/alm/last-test-site.json', process.argv[1])" "$(cat <<'EOF'
{...the JSON above...}
EOF
)"

or — when invoked from plan-alm, the orchestrator may supply the JSON inline. Either way, the marker file location is fixed: docs/alm/last-test-site.json (sibling to docs/alm/last-deploy.json and docs/alm/last-pipeline.json).

Always include stageName in the marker when known. The agent learns the stage label from the upstream context — docs/alm/last-deploy.json's stageName, the plan's stages[], or an explicit user mention. If the stage cannot be inferred (e.g. test-site invoked standalone against an arbitrary URL), set stageName to null; the refresh helper has fallback resolution paths but the explicit field is the most reliable signal.

6.7b Refresh the ALM plan (if one exists)

node "${PLUGIN_ROOT}/scripts/lib/refresh-alm-plan-data.js" \
  --projectRoot "." \
  --phase test-site \
  --stageName "{stageName}" \
  --render

{stageName} is the stage label this run tested (e.g. Staging, Production). Pass an empty string when unknown — refreshTestSite falls back to (1) the marker's stageName field (set in 6.7a above), then (2) the single target stage in planData.stages if there's only one. Multi-stage plans with no explicit stageName + no marker stageName won't be captured (the refresh re-renders without a per-stage validationRun update); always pass it explicitly when you can.

The helper reads docs/alm/last-test-site.json, populates planData.validationRuns[{resolvedStage}] with the categorized test outcome, and re-renders docs/alm-plan.html so the Validation tab updates immediately. When docs/.alm-plan-data.json is absent (standalone invocation, no plan in the project), the helper returns ok:false as a soft no-op — safe to run unconditionally.

Point the user at the next step (user-driven sequencing). The helper's stdout JSON includes nextStep: { name, skill: string | null } | null. When non-null, branch on skill: when skill is non-null, tell the user "Plan updated. Next in your plan: {nextStep.name} → run {nextStep.skill} when you're ready."; when skill is null (an internal step such as Finalize, no user command), name the step only — *"Plan updated. Next in your plan: {nextStep.name}.

…(truncated)