Network Debug

Network Debug

Inspect HTTP traffic captured by PowHTTP proxy. Two primary use cases: fixing existing request flows and building new ones from scratch.

See tool-reference.md for tool documentation and token-optimization.md for token optimization.

Verify PowHTTP is Running

powhttp_session_active()

Workflow 1: Fix Existing Code

When a Go client request fails or returns unexpected results, compare it against browser traffic in PowHTTP to find the mismatch.

1. Capture both request flows

Run the same action in the browser and through the Go client while PowHTTP captures. Find entries from each source:

powhttp_search_entries(filters: {host: "*.example.com", process_name: "chrome"}, limit: 20)
powhttp_search_entries(filters: {host: "*.example.com", process_name: "server"}, limit: 20)

2. Compare the request sequence

Use powhttp_trace_flow on each flow's first request to see the full chain. Compare:

• Request order — both flows must make the same requests in the same sequence. A reordered request often causes auth failures or missing cookies downstream.
• Request count — if the Go client skips a request the browser makes (e.g., a token refresh, a session init), downstream requests may fail due to missing state.

3. Compare individual requests

For each corresponding pair of requests, use powhttp_diff_entries:

powhttp_diff_entries(baseline_entry_id: "browser_entry", candidate_entry_id: "go_entry")

Check these in order of impact:

1. URL and query parameters — path, parameter names, values, and order
2. Request headers — names, values, and order. Use powhttp_get_entry(entry_id: "...", include_headers: true) for full details.
3. Request body — field names, values, encoding (JSON vs form-encoded)
4. TLS fingerprint — use powhttp_fingerprint(entry_id: "...") to compare JA3/JA4 hashes. A mismatch here blocks most anti-bot systems.
5. HTTP/2 settings — pseudo-header order and SETTINGS frame order via powhttp_get_http2_stream

4. Fix and re-validate

After fixing, route the Go client through PowHTTP again and re-run powhttp_diff_entries to confirm the differences are resolved.

Workflow 2: Build a New Flow

When implementing a new API flow from scratch, use browser traffic captured by PowHTTP as the ground truth. Replicate each step exactly, then look for steps that can be removed or simplified.

1. Capture the browser flow

Navigate the browser through the target action while PowHTTP captures. Extract the request sequence:

powhttp_extract_endpoints(scope: {host: "target.com", time_window_ms: 120000}, filters: {category: "api"})
powhttp_trace_flow(seed_entry_id: "first_request")

2. Document each request

For each request in the flow:

powhttp_get_entry(entry_id: "...", include_headers: true, body_mode: "compact")

Record: method, URL, headers (names, values, order), query parameters, request body, and response shape. Use powhttp_infer_schema(cluster_id: "...") for response structure.

3. Identify unnecessary steps

Not every browser request needs to be replicated. Flag these for removal:

• Static asset loads — images, CSS, JS (category "asset" in endpoint extraction)
• Analytics/tracking requests — third-party domains, beacon endpoints
• CORS preflight requests — OPTIONS calls that the Go client won't trigger
• Duplicate requests — the browser may retry or make parallel requests that can be consolidated

Test each removal individually. If removing a step causes downstream failures (missing cookies, tokens, or required state), it must be kept.

4. Implement step by step

Implement each required request in Go, matching the browser's headers, parameters, and payload. After each step, validate through PowHTTP:

powhttp_validate_schema(entry_ids: ["go_entry"], schema: "type Resp struct { ... }", schema_format: "go_struct")
powhttp_diff_entries(baseline_entry_id: "browser_entry", candidate_entry_id: "go_entry")

Common Pitfalls

Date/filter cookies

Some sites encode filter state as base64 JSON in cookies rather than query parameters. If filtered results in the browser differ from API responses with the same query params, inspect cookie values — decode any base64 cookies to check for embedded date ranges, locale settings, or filter selections that affect the response.

Dynamic value chains

Some responses contain tokens, nonces, or CSRF values that must be extracted and injected into subsequent requests. Use powhttp_trace_flow to identify data dependencies between requests — look for edge_type_summary entries like auth_chain or same_auth. In Go, parse these values from the response body or headers and pass them forward rather than hardcoding.

Cookie chain tracking

Each request in a flow may depend on cookies set by earlier requests. Use powhttp_get_entry with include_headers: true to check Set-Cookie response headers. If the Go client skips a request that sets a required cookie, all downstream requests using that cookie will fail silently — the server returns a login page or empty response instead of an error.

Redirect handling

Browsers follow 302/303 redirects transparently, picking up cookies at each hop. With tls-client, redirects may need manual handling to capture intermediate Set-Cookie headers. Check powhttp_trace_flow for redirect edges — each redirect in the chain may set state that later requests depend on.

Content-Type mismatches

The browser may submit a form as application/x-www-form-urlencoded while you default to application/json. Some servers accept both but return different responses. Others reject the wrong type outright. Check the Content-Type request header in powhttp_get_entry and match it exactly.

Referer validation

Some sites check that the Referer header matches the expected navigation path. If you skip a page the browser visits, the referer chain breaks and the server may reject the request. Check whether the browser's requests include Referer headers and replicate the chain.

Challenge/WAF detection

A response with status 200 may still be a block page (Cloudflare, AWS WAF) rather than the expected content. Always check the response body, not just the status code. Common indicators: HTML with challenge scripts, response size much smaller or larger than expected, Set-Cookie headers containing challenge tokens. Use powhttp_query_body to check response content when a flow returns unexpected results.

Tool Quick Reference

Searching traffic

powhttp_search_entries(filters: {host: "*.example.com"}, limit: 10)

Results are thin by default. Set include_details: true for TLS/HTTP2/process info. See token-optimization.md for the full search parameter table.

Inspecting entries

powhttp_get_entry(entry_id: "...", include_headers: true, body_mode: "compact")

Body modes: compact (default, arrays trimmed), schema (structure only), full (complete body).

Understanding response shape

1. powhttp_describe_endpoint(cluster_id: "...") — overview with body shapes
2. powhttp_infer_schema(cluster_id: "...") — merged schema with field statistics
3. powhttp_query_body(expression: ".field", cluster_id: "...") — extract specific values

Comparing requests

powhttp_diff_entries(baseline_entry_id: "browser", candidate_entry_id: "go_client")
powhttp_fingerprint(entry_id: "...")
powhttp_get_tls(connection_id: "...")

Implementing in Go

When implementing a discovered API:

• Use bogdanfinn/tls-client with Chrome 144 profile for TLS fingerprinting
• Copy headers exactly as they appear in powhttp_get_entry, including order
• Use http.HeaderOrderKey and http.PHeaderOrderKey from fhttp for header ordering
• Do NOT manually set Cookie, Content-Length, Host, or Connection headers — they're handled automatically
• Add response types in the platform package's api.go file
• Use platform.DoJSON() for JSON APIs or raw client.Do() for HTML scraping
• Add status mapping in internal/platform/mapping.go if new statuses appear
• Follow the platform.BuildURL() pattern for query parameters
• Log with pipe-delimited format: s.log.Infof("platform | action | detail | key: value")

Validate implementation

After making a request through the proxy:

1. Validate schema: powhttp_validate_schema(entry_ids: ["scraper_entry"], schema: "type Resp struct { ... }", schema_format: "go_struct")
2. Validate stealth: powhttp_diff_entries(baseline_entry_id: "browser_entry", candidate_entry_id: "scraper_entry")
   • Header order mismatch → fix HeaderOrderKey
   • TLS fingerprint mismatch → check profiles.Chrome_144

Platform Access

• Viagogo public pages (www.viagogo.com) may trigger AWS WAF challenges
• Viagogo seller dashboard (my.viagogo.com) requires an authenticated browser session
• Vivid Seats API (vividseats.com/hermes/api/v1/...) doesn't require auth
• Lysted API (api.lysted.com) requires a Bearer token
• Section matching debug: logic in internal/sync/listings.go:sectionForMatching() and internal/platform/allocations.go:SectionMatches()