Test Red Team

test-red-team — Adversarial Full-App Defect Sweep

Degree of freedom: MIXED — matrix and severity [HIGH freedom]; ROE, harness, and benign-payload confirmation [LOW freedom — run exactly]. Web driver is playwright-cli. Do not write exploit PoCs into the repo.

A hostile, skeptical pass across UI/UX, data pipeline, security, and performance. Default output is a severity-ranked defect list, not inline fixes (test-playwright is the fix-as-you-go PDCA). Offer to fix after the report; ask which defects to prioritize.

Read protocol-browser-anti-stall before any browser action (Rule 0: manual & headed) and references/playwright-session-coordination.md.

Attack through the visible UI, by hand. One real action at a time. eval / CDP / WebView attach may set a condition (throttle, kill a request, emulate) or inspect — never perform the click/type/submit under test. The defect must be reachable the way a real attacker reaches it.

This skill vs neighbors

How to reason (every cell / finding)

1. Observe — screenshot + console + network + DB (for mutations)
2. Interpret — reachable defect vs expected deny vs untested (N-A)
3. Classify — PASS / DEFECT / N-A; chain if two cells combine
4. Severity — Critical/High/Medium/Low/Info by impact × likelihood

Worked example

Observe: User B GET /api/docs/A-id → 200 + A's body. SET ROLE authenticated + user_id != auth.uid() also returns the row. Interpret: IDOR; handler has no ownership check. Classify: DEFECT (Security / A01). Chain with weak session = takeover. Severity: Critical — cross-user data leak. Evidence: screenshot + network + SQL. Payload was a read, not a dump/DROP.

Coverage model

Do NOT do a blind DOM crawl (test-qa) or an unscripted guest-vs-logged-in wander (test-exploratory). Traverse a coverage matrix:

feature/capability
  → surfaces (routes, sub-pages, API endpoints, RPCs, tables)
    → components + states (forms, tables, modals, empty/error/loading, roles)
      → 4 attack dimensions: UI/UX · data pipeline · security · performance

Each cell (feature × surface × component-state × dimension) is PASS / DEFECT / N-A. The matrix is the plan and the audit trail.

Graceful fallback: no discernible features (pure marketing) → route → component structural traversal.

Phase 0: Scope & rules of engagement [LOW freedom — ROE exact]

0a. Detect the stack

package.json → framework, auth, ORM, RN/Expo, port. capacitor.config.ts/.json → Capacitor hybrid. app.json / app.config.* → Expo / RN. android/ → native Android.

Record: target type (web / RN/Expo / Capacitor / mixed); dev URL/port; auth pattern; test credentials (.env.local, .env.test, README); backend MCPs (Supabase, Sentry, Firecrawl).

0b. Pick the automation driver

0c. Capacitor WebView attach (when target includes Capacitor)

Exception to Rule 0: a native WebView can't be driven by snapshot-ref tools, so attaching is the only option. Once attached, still interact like a user — don't script the flow end-to-end.

const { _android } = require('playwright');
const [device] = await _android.devices();
// ADB online, Chrome ≥ 87, WebView debuggable
const webview = await device.webView({ pkg: 'com.your.app.id' });
const page = await webview.page();
// standard Playwright Page API — not playwright-cli verbs

Native chrome outside the WebView: adb shell input tap with coordinates from adb shell uiautomator dump.

0d. Rules of engagement

• Test data prefixed RT-TEST- for cleanup
• Do NOT mutate production data; ask before any non-reversible operation
• Confirm with benign evidence; no destructive PoCs against real rows; do not write exploit PoCs into the repo
• Secrets by name only — never printed

0e. Browser session (before Phases 2–5)

1. Own session — never share or claim another agent's:

PW="npx --yes @playwright/cli@latest"
$PW -s=rt-<app> open --headed --persistent \
    --profile "$HOME/.playwright-cli-profiles/<app>" "<app-url>"

2. Reuse the signed-in session (--profile, or state-save / state-load .playwright-mcp/auth/<host>.json). Google needs the one-time real-Chrome login in the coordination reference.
3. Never close-all / kill-all; close only -s=rt-<app> when done.
4. Do not log out between attack phases unless testing logout/fixation.

Phase 1: Recon & build the coverage matrix [HIGH freedom]

Build the matrix from source before opening the browser.

1a. Enumerate features / capabilities

1b. For each feature, decompose

Feature: [name]
  Surfaces: routes/pages, deep links, API/RPCs, DB tables
  Components + states: default/empty/loading/error/role-A/role-B; forms;
    modals; file uploads
  Roles / tenants: anon, user, admin, org-member, …
  Input surfaces: fields, URL params, query strings, headers, file content

1c. Output the matrix

Print before Phases 2–5. Fill ✅ PASS / ❌ DEFECT(#N) / — N-A as you go.

COVERAGE MATRIX
| Feature   | Surface/Component+State        | UI/UX | Pipeline | Security | Perf |
|-----------|--------------------------------|-------|----------|----------|------|
| Auth      | Login form / empty             |       |          |          |      |
| Auth      | Login form / error             |       |          |          |      |
| Documents | List / empty · 100+ · create · share/other-user |       |          |          |      |

Phase 2: UI/UX red team (per matrix cell) [HIGH freedom]

Driver: playwright-cli — goto, snapshot, screenshot, click, type, fill, console, requests, resize.

screenshot every DEFECT. Note the component path if identifiable.

Phase 3: Data pipeline red team (per matrix cell) [LOW freedom — 3-layer verify]

Every mutation surface (create / update / delete / upload). Verify at (1) UI (2) network (3) DB. Look up Supabase schemas first, then:

supabase:execute_sql
{
  "project_id": "<PROJECT_ID>",
  "query": "SELECT * FROM <table> WHERE <col> LIKE 'RT-TEST-%' ORDER BY created_at DESC LIMIT 10"
}

RLS (what the client can actually read/write):

supabase:execute_sql
{
  "project_id": "<PROJECT_ID>",
  "query": "SET ROLE authenticated; SELECT * FROM <table> WHERE user_id != auth.uid() LIMIT 5"
}

Also list_tables, get_logs(service: 'api'|'postgres'), get_advisors. Unexpected 5xx, unhandled exceptions, RLS denies, and new ERROR advisors are pipeline defects.

Phase 4: Security red team (per matrix cell, OWASP-mapped) [HIGH freedom; payloads = LOW]

Full payload tables: references/owasp-attack-checklist.md. Confirm with benign payloads (document.title='XSS', SELECT 1) — never DROP TABLE, never real exfil, never commit an exploit PoC.

Priority (highest real-world impact first):

Correlate with Sentry (401/403/CSP, last 30 days) and current OWASP via Firecrawl.

Phase 5: Performance red team (per matrix cell) [HIGH freedom; throttle setup = LOW]

Condition-setup only — then drive load/scroll/submit by hand.

const client = await page.context().newCDPSession(page);
await client.send('Network.emulateNetworkConditions', {
  offline: false, downloadThroughput: 50 * 1024 / 8,
  uploadThroughput: 20 * 1024 / 8, latency: 300
});

Capacitor/Android: adb shell tc qdisc add dev wlan0 root netem delay 300ms.

Same get_advisors as Phase 3: missing indexes, seq scans, bloated RLS = Medium–High perf defects.

Phase 6: Finding-chaining & triage [HIGH freedom]

Scan DEFECT cells for chains before the report:

• Two Mediums often combine into High/Critical (reflected input + missing CSP = stored XSS).
• Auth weakness + IDOR = account takeover.
• Stale cache + no optimistic revert = silent corruption.

Phase 7: Defect report (the deliverable) [LOW freedom — this shape]

## Red-Team Defect Report — [App / Repo] — [Date]

### Scope
- Target type: [web | RN | Capacitor | mixed]
- Dev URL: [...]  Auth: [...]
- Features red-teamed: [list]
- Coverage matrix: [link or inline]

### Critical Defects (fix before any release)
| # | Feature | Surface / Component+State | Finding | Evidence | File | Remediation |
|---|---------|--------------------------|---------|----------|------|-------------|
| 1 | Auth | Login / error | IDOR: B reads A via GET /api/docs/:id | Screenshot+network | src/app/api/docs/[id]/route.ts | `WHERE user_id = auth.uid()` |

### High / Medium / Low / Info
Same columns. Group by severity.

### Attack chains identified
1. [Finding #N] + [Finding #M] → [combined impact and path]

### Coverage matrix (final)
| Feature | Surface/Component+State | UI/UX | Pipeline | Security | Perf |
|---------|------------------------|-------|----------|----------|------|

### Backend truth-check
- Sentry: [new/related — count + highest severity]
- Supabase: [advisors, logs, RLS results]

### Launch-readiness verdict
**Ready / Ready after Critical fixes / Not ready** — [2-sentence justification]

### Test data cleanup
- RT-TEST-* rows deleted: [Y/N / N-A]

Self-critique before reporting [LOW freedom — do not skip]

1. No exploit PoC — benign confirm only; nothing committed to the repo
2. Every DEFECT evidenced — screenshot + console + network (+ DB for mutations)
3. Untested ≠ PASS — untested cells are N-A with why
4. Right owner — guest wander → test-exploratory; LLM inject → audit-llm-security; load → test-load
5. Report first — fix only after the user picks Critical/High

Guardrails

1. Manual & headed, never scripted — attack through the visible UI; code execution is condition-setup/inspection only; no *.spec.ts, no npx playwright test.
2. Anti-stall always — never block >3 s; max 4 attempts; [TIMEOUT] and move on.
3. Evidence for every DEFECT — screenshot + console + network + DB. "It looked broken" is not a defect.
4. No destructive PoCs — benign payloads only; never DROP TABLE or real exfil; do not write exploit PoCs into the repo.
5. Ask before mutating production rows — requested DDL may ship; DELETE/UPDATE/TRUNCATE on real prod data asks first.
6. Secrets by name only.
7. Honest verdict — untested cells are N-A, not PASS.
8. Offer to fix after the report — then test-playwright PDCA.
9. MCP schemas first — check mcps/<server>/tools/ before any MCP call.
10. Pure-native iOS/Android out of scope — say so in the report.

Related

• audit-llm-security — product LLM / prompt-injection depth
• test-load — concurrency breaking point (not a headed UI attack)
• test-playwright / test-qa / audit-security / iterate-post-launch