Capture

Traffic Capture (Phase 1)

Assess the site, then capture comprehensive HTTP traffic. This skill combines site assessment with full traffic recording in a single browser session.

Copy this checklist and check off items as you complete them:

Phase 1 Progress:
- [ ] Prerequisites: playwright-cli available, checkpoint restored if one exists
- [ ] Step 1: Browser open on target URL
- [ ] Step 2: Fingerprint run + auth handled + profile classified + assessment.md written
- [ ] Step 3: Full capture — exploration targets met for the profile
- [ ] Step 4: Trace parsed, validate-capture.py PASSED (the gate)
- [ ] Step 5: Browser closed, checkpoint + phase-state marked complete

CRITICAL EXECUTION RULES

NEVER use run_in_background: true for ANY playwright-cli command. All playwright-cli commands must run in the foreground with appropriate timeouts. Background execution causes task ID tracking failures — the command completes before you can read the output. See references/playwright-cli-commands.md for the timeout table.

NEVER use eval for complex expressions. eval fails silently on ternaries, comma operators, and multi-branch logic with "not well-serializable" errors. Use run-code instead. See references/framework-detection.md for details.

ESM context — no require(). run-code uses ESM. Use await import('fs') instead of require('fs'). See references/playwright-cli-commands.md.

Prerequisites (Hard Gate)

Do NOT start unless:

• playwright-cli is available (npx @playwright/cli@latest --version)
• Target URL is known

Default capture method: playwright-cli tracing (standard workflow below).

Optional --mitmproxy mode

When the --mitmproxy flag was passed to /cli-anything-web (or the capture needs untruncated bodies / enhanced metadata), follow this workflow with the step substitutions in references/mitmproxy-mode.md — Steps 1, 3, and 4 differ; everything else is identical. When playwright-cli itself is unavailable (e.g., no Node), fall back to chrome-devtools-mcp via launch-chrome-debug.sh — see HARNESS.md Tool Hierarchy.

Public API Shortcut

If the target site has a documented public REST/JSON API (e.g., Hacker News Firebase API, Dev.to API, Reddit API, Wikipedia API), browser capture is optional:

1. Probe the API endpoints directly with httpx or curl
2. Save responses as <app>/traffic-capture/raw-traffic.json
3. Skip to Phase 2 (methodology)

This applies when:

• API docs exist (OpenAPI/Swagger, developer docs page, /api/ prefix)
• The API is publicly accessible without browser-specific auth
• Endpoints return JSON (not HTML)

If unsure whether a public API exists, proceed with browser capture as normal.

Resume from Checkpoint

Before starting, check if a previous capture session exists:

python ${CLAUDE_PLUGIN_ROOT}/scripts/capture-checkpoint.py restore <app>

If a checkpoint exists, read the guidance field and resume from the last completed step instead of starting over. This prevents duplicate work when sessions are interrupted.

Step 1: Setup

# Create output directory
mkdir -p <app>/traffic-capture

# Clear any stale sessions
npx @playwright/cli@latest kill-all 2>/dev/null || true

npx @playwright/cli@latest -s=<app> open <url> --headed --persistent
# Note: heavy SPAs (Next.js, React) may show "TimeoutError: page._snapshotForAI" on open.
# This is non-fatal — verify with: npx @playwright/cli@latest list
#
# IMPORTANT — "Browser opened with pid..." in command output means the daemon
# RE-ATTACHED to the existing browser, NOT that a new session was created.
# Do NOT re-navigate or restart when you see this. The session is still open.

--mitmproxy mode: use the Step 1 substitution in references/mitmproxy-mode.md (start the proxy, then open through it).

Do NOT ask the user to log in yet — Step 2 will determine if auth is needed.

Step 2: Site Fingerprint (Single Command)

Run the all-in-one site fingerprint command instead of individual eval calls. This is faster, more reliable, and detects framework + protection + iframes + auth requirements in one shot.

Use the script file — multi-line JS with arrow functions and optional chaining fails in playwright-cli's single-line command parser. The script file approach has been tested and works reliably:

npx @playwright/cli@latest -s=<app> run-code "$(grep -v '^\s*//' ${CLAUDE_PLUGIN_ROOT}/scripts/site-fingerprint.js | tr '\n' ' ')"

IMPORTANT: The site-fingerprint.js script must be loaded via the command above. Do NOT copy-paste the JS inline — it will fail with SyntaxError. The grep -v strips comments and tr joins lines for single-line execution.

Interpret fingerprint results

The fingerprint returns four groups: framework, protection, auth, iframes. Map each true flag to the next action:

Claude-facing shortcuts:

• googleBatch: true → generate rpc/ subpackage (batchexecute protocol).
• cloudflareManagedChallenge: true → tier 4 (camoufox) is required; curl_cffi alone will fail.
• awsWaf: true → capture aws-waf-token cookie; use curl_cffi for GraphQL, cookie-only for SSR.
• akamai: true or datadome: true → 1–2 s delays between requests are mandatory.
• serviceWorker: true → note in assessment.md; generated CLI uses service_workers="block".
• iframeCount > 0 → re-run the fingerprint inside the iframe. Google Labs apps (Stitch / MusicFX / ImageFX) follow this pattern — parent has WIZ_global_data, iframe has the real app.

Note: snapshot and click <ref> auto-resolve iframes. Only drop down to run-code for iframe interaction when built-in commands fail.

Auth detection (BEFORE exploration)

Check the fingerprint auth fields:

If auth is needed:

1. Tell the user: "This site requires login. Please log in in the browser window."
2. Wait for user confirmation
3. Save auth state and tighten permissions (CLAUDE.md mandates chmod 600):

npx @playwright/cli@latest -s=<app> state-save <app>/traffic-capture/<app>-auth.json
chmod 600 <app>/traffic-capture/<app>-auth.json

If NO auth is needed: Skip directly to Step 2b.

2b. Classify Site Profile

Based on fingerprint results AND what you see in the UI, classify the site:

2c. Quick API Probe (Force SPA Navigation Trick)

Start a SHORT trace, click 3-4 internal links, stop. This reveals hidden API endpoints that SSR hides on initial page load.

npx @playwright/cli@latest -s=<app> tracing-start
npx @playwright/cli@latest -s=<app> click <internal-link-1>
npx @playwright/cli@latest -s=<app> click <internal-link-2>
npx @playwright/cli@latest -s=<app> click <internal-link-3>
npx @playwright/cli@latest -s=<app> tracing-stop

# Quick parse to see what endpoints appeared (saved alongside the full capture
# so it survives the session — don't output to /tmp).
python ${CLAUDE_PLUGIN_ROOT}/scripts/parse-trace.py .playwright-cli/traces/ --latest \
  --output <app>/traffic-capture/probe-traffic.json

This probe trace is separate from the full capture in Step 3 — Step 3 will start a fresh trace that overwrites the .network file in .playwright-cli/traces/. The parsed probe-traffic.json is kept in traffic-capture/ so it stays available for cross-referencing during Step 4.

Check the probe results — what API patterns did you find? See references/api-discovery.md for the priority chain and decision tree.

2d. Write Assessment Summary

Create <app>/traffic-capture/assessment.md to consolidate all findings:

# Site Assessment: <app>

- **URL**: <url>
- **Framework**: <detected framework or "none/custom">
- **Protocol**: <REST / GraphQL / batchexecute / HTML scraping / hybrid>
- **Protection**: <none / cloudflare / captcha / aws-waf / etc.>
- **Auth required**: <yes (type: Google SSO / cookie / JWT / API key) / no>
- **Iframes**: <yes (N frames, app in frame N at <url>) / no>
- **Site profile**: <Auth+CRUD / Auth+Generation / Auth+Read-only / No-auth+CRUD / No-auth+Read-only>
- **Capture strategy**: <API-first / SSR+API hybrid / batchexecute / HTML scraping / protected-manual>
- **Key observations**: <any quirks, localized UI, rate limits, special patterns>

Step 3: Full Traffic Capture

Now do the comprehensive capture based on what Step 2 revealed.

# Optional: Start HAR recording alongside trace for standard-format capture
# HAR files enable mitmproxy2swagger (auto OpenAPI spec) and third-party analysis tools
npx @playwright/cli@latest -s=<app> run-code "async page => {
  await page.context().routeFromHAR('<app>/traffic-capture/capture.har', {
    update: true,
    updateContent: 'embed',
    updateMode: 'full'
  });
  return 'HAR recording started';
}"

# Start fresh trace for full capture (note the trace ID from output!)
npx @playwright/cli@latest -s=<app> tracing-start
# Output: "trace-<ID>" — record this ID

--mitmproxy mode: skip tracing-start and HAR recording — the proxy is already capturing (see references/mitmproxy-mode.md).

HAR recording is optional but recommended. It produces a standard HAR file alongside the trace. This enables mitmproxy2swagger to auto-generate an OpenAPI spec: pip install mitmproxy2swagger && mitmproxy2swagger -i capture.har -o api-spec.yaml -p <base-url> The HAR file is saved when the browser context is closed (Step 5).

Exploration by site profile

Use the concrete targets in references/exploration-checklists.md for the profile identified in Step 2b. Each profile has an explicit entry count, distinct-path count, and WRITE-op target that validate-capture.py (Step 4) will enforce. Minimum bar across all profiles:

• ≥ 15 entries, ≥ 3 distinct URL paths, protocol ≠ unknown
• ≥ 1 WRITE op (unless the site is genuinely read-only — pass --read-only to the validator)
• < 50% error rate (dominant 4xx/5xx means auth or rate-limit failure)

Pacing for protected sites

If any of cloudflare, cloudflareManagedChallenge, akamai, datadome, awsWaf, or rateLimit fired in the fingerprint, leave 1–2 s between clicks / form submits. Faster exploration triggers per-IP challenges within ~30 requests and corrupts the trace.

General interaction rules

• Click by ref (from snapshot) is most reliable: snapshot → note ref → click <ref>
• Refs go stale — always take a fresh snapshot before clicking
• For localized UIs (Hebrew, Arabic, etc.) — use refs or data-testid, not text
• For iframe-embedded apps — snapshot + click <ref> auto-resolves iframes
• Wait after generation — if the app generates content async, wait for ≥ 15 s before the next action, otherwise the polling loop won't appear in the trace:

  npx @playwright/cli@latest -s=<app> run-code "async page => {
    await page.waitForTimeout(15000);
    return 'waited';
  }"
  

• Debounced inputs — after typing a search query, pause 1–2 s before the next action; submitting immediately misses the auto-complete endpoint.

Step 4: Stop, Save, Parse

npx @playwright/cli@latest -s=<app> tracing-stop

If tracing-stop fails: retry once with a 15s timeout; if it fails again the trace is lost — restart the trace at Step 3. Never retry more than twice. Full decision tree: skills/shared/RECOVERY.md §tracing-stop Failure (error signatures: references/playwright-cli-tracing.md).

python ${CLAUDE_PLUGIN_ROOT}/scripts/parse-trace.py \
  .playwright-cli/traces/ --latest \
  --output <app>/traffic-capture/raw-traffic.json

# parse-trace.py now auto-runs analyze-traffic.py and produces:
#   - <app>/traffic-capture/raw-traffic.json (raw request/response data)
#   - <app>/traffic-capture/traffic-analysis.json (auto-detected protocol, auth, endpoints)

# Gate — validate the capture before declaring Phase 1 complete.
# This check enforces: ≥15 entries, ≥3 distinct paths, protocol ≠ unknown,
# ≥1 WRITE op (add --read-only if the site is genuinely read-only), <50% error rate.
python ${CLAUDE_PLUGIN_ROOT}/scripts/validate-capture.py <app>
# OR for genuinely read-only sites:
# python ${CLAUDE_PLUGIN_ROOT}/scripts/validate-capture.py <app> --read-only

If parse-trace.py fails or produces an empty/static-only raw-traffic.json, follow skills/shared/RECOVERY.md §parse-trace Failure.

If validate-capture.py returns a non-zero exit code, do not proceed to Step 5. Map each failed gate to its targeted remediation in skills/shared/RECOVERY.md §validate-capture Non-Zero Exit (e.g., <15 entries → capture more pages; <3 distinct paths → exercise more features; no WRITE op → perform a create/update/delete in the UI). Re-open the browser (Step 1), fill the gaps, then re-run Step 4. Only mark the capture complete after the validator passes (or warns, with your explicit sign-off on each warning).

For deeper inspection:

python ${CLAUDE_PLUGIN_ROOT}/scripts/analyze-traffic.py \
  <app>/traffic-capture/raw-traffic.json --summary

--mitmproxy mode: replace the parse/analyze block with the Step 4 substitution in references/mitmproxy-mode.md (stop-proxy already includes analysis; the same validate-capture gate applies).

Step 5: Close

npx @playwright/cli@latest -s=<app> close

# Mark the capture checkpoint AND the pipeline phase complete
python ${CLAUDE_PLUGIN_ROOT}/scripts/capture-checkpoint.py update <app> --step complete
python ${CLAUDE_PLUGIN_ROOT}/scripts/phase-state.py complete <app> --phase capture \
  --output traffic-capture/raw-traffic.json

If an endpoint is missing — USE THE FEATURE

Don't grep JS bundles. Start a new trace → screenshot → click the button → fill → submit → stop → parse. The browser IS the API documentation.

Fallback

Fallback: If playwright-cli is not available, see HARNESS.md Tool Hierarchy for chrome-devtools-mcp fallback instructions.

Next Step

When capture is complete (raw-traffic.json has WRITE operations, or the site is read-only with only GET requests), invoke methodology to analyze the traffic and build the CLI.

References

Gate failures (tracing-stop, parse-trace, validate-capture, phase-state): skills/shared/RECOVERY.md. Implementation rules: skills/shared/CONVENTIONS.md.

See references/ for:

• playwright-cli-commands.md — command syntax, timeouts, ESM rules
• playwright-cli-tracing.md — trace file format, recovery protocol
• playwright-cli-sessions.md — named sessions, auth persistence
• playwright-cli-advanced.md — waits, iframes, localized UIs, downloads
• framework-detection.md — framework → protocol table
• protection-detection.md — anti-bot escalation ladder (curl_cffi → camoufox → hybrid)
• api-discovery.md — protocol priority chain, decision tree
• exploration-checklists.md — per-profile capture targets with concrete numbers
• mitmproxy-mode.md — step substitutions for --mitmproxy capture