curl as ground truth
When it applies
Any time you need to know what a server actually returns — no proxy magic, no scanner
interpretation — or when automated tooling is being blocked at a CDN/WAF edge and you suspect the
request shape, not the content, is the problem. Typical trigger: bare curl or a datacenter UA gets
403 with a JSON access-policy body or a CF-RAY header, while the same endpoint works in a real
browser.
Why it works
Edge protection (Cloudflare & friends) classifies clients by TLS/HTTP fingerprints and headers. A
request carrying a browser User-Agent plus plausible Origin/Referer/Sec-Fetch-* headers often
sails through where a naked curl gets challenged — and unlike a scanner, curl shows you the exact
bytes that came back.
Method
- Baseline:
curl -sS -D - -o /dev/null --max-time 15 https://target/ — -D - dumps response
headers, -o /dev/null discards the body, --max-time keeps a hung endpoint from stalling you.
- On an edge 403, impersonate the browser before concluding anything:
curl -sS --max-time 15 \
-A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" \
-H "Accept: application/json, text/plain, */*" \
-H "Origin: https://app.example" \
-H "Referer: https://app.example/" \
-H "Sec-Fetch-Site: same-site" -H "Sec-Fetch-Mode: cors" -H "Sec-Fetch-Dest: empty" \
"https://api.example/v1/health"
Match Origin/Referer to the site's real front-end origin — that's usually what the policy
keys on.
- Bracket/query params (prototype-pollution shapes) need
--globoff: curl treats [] as URL
globs and dies with curl: (3) bad range in position N. Either:curl --globoff -sS "https://target/?__proto__%5Btest%5D=polluted"
curl --globoff -sS -G "https://target/path" --data-urlencode "__proto__[test]=polluted"
- Read the response like a signal, not a verdict:
403 + JSON access-policy / CF-RAY → retry with browser headers (step 2).
- Still
403 and HTML contains Cf-Mitigated: challenge → a real browser or an authenticated
session is required; do not brute-force the edge.
- SPA route that returns
200 with an identical body size for every path → front-end router,
not a real API hit — don't log it as an endpoint.
- Feed curl requests to other tools: once a request works, export/replay it exactly — save it
for
sqlmap -r (tools-sqlmap) or send it to Burp Repeater (tools-burp-suite) instead of
re-deriving it.
Gotchas
curl -I sends HEAD — some apps 405 or behave differently on HEAD; for a real GET with
headers shown, use curl -s -o /dev/null -D -.
- Windows
curl.exe vs the alias — in PowerShell curl may be an alias for
Invoke-WebRequest; call curl.exe explicitly, and remember --globoff for [].
- Windows
httpx collision — the httpx command may resolve to the Python httpx library CLI,
not ProjectDiscovery's httpx; check httpx -version before piping recon output into it.
- A working browser-header bypass is recon, not a bypass finding — it means the edge admits
browser-shaped traffic; record it and continue, don't report it.
- Timeouts are part of the command — always
--max-time; a hung curl in an automated loop
silently stalls the whole pipeline.
Verify success
You have a reproducible, copy-pasteable request that returns the expected status/body through the
edge, saved verbatim in the engagement notes so any later tool (sqlmap, Repeater, a finding's
repro steps) uses the exact same bytes.
References
curl manual (--globoff, --data-urlencode); Cloudflare edge behavior. Pipeline context:
automation-recon-pipeline.
Portions adapted from reverse-skill by zhaoxuya520, MIT License.
1---2name: tools-curl3description: >4 Use curl as the ground-truth HTTP client during recon and verification — browser-impersonation5 headers for CDN/WAF 403s, --globoff for bracket params, Windows curl.exe quirks, SPA false-2006 traps. Load when bare tooling gets 403/blocked at a CDN edge, an API needs precise hand-crafted7 requests, or scanner output needs a clean manual repro. Signals: curl, 403 access-policy, CF-RAY,8 Cf-Mitigated, "bad range in position", --globoff, data-center UA blocked.9domain: tools10type: technique11stability: learning12modes: [pentest, bugbounty]13severity: info14tools: [curl]15schema_version: 116metadata:17 source: https://skillmd.com/skills/noorqureshi/tools-curl18---1920# curl as ground truth2122## When it applies23Any time you need to know what a server *actually* returns — no proxy magic, no scanner24interpretation — or when automated tooling is being blocked at a CDN/WAF edge and you suspect the25request shape, not the content, is the problem. Typical trigger: bare curl or a datacenter UA gets26`403` with a JSON `access-policy` body or a `CF-RAY` header, while the same endpoint works in a real27browser.2829## Why it works30Edge protection (Cloudflare & friends) classifies clients by TLS/HTTP fingerprints and headers. A31request carrying a browser User-Agent plus plausible `Origin`/`Referer`/`Sec-Fetch-*` headers often32sails through where a naked `curl` gets challenged — and unlike a scanner, curl shows you the exact33bytes that came back.3435## Method361. **Baseline:** `curl -sS -D - -o /dev/null --max-time 15 https://target/` — `-D -` dumps response37 headers, `-o /dev/null` discards the body, `--max-time` keeps a hung endpoint from stalling you.382. **On an edge 403, impersonate the browser** before concluding anything:39 ```bash40 curl -sS --max-time 15 \41 -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" \42 -H "Accept: application/json, text/plain, */*" \43 -H "Origin: https://app.example" \44 -H "Referer: https://app.example/" \45 -H "Sec-Fetch-Site: same-site" -H "Sec-Fetch-Mode: cors" -H "Sec-Fetch-Dest: empty" \46 "https://api.example/v1/health"47 ```48 Match `Origin`/`Referer` to the site's real front-end origin — that's usually what the policy49 keys on.503. **Bracket/query params (prototype-pollution shapes) need `--globoff`:** curl treats `[]` as URL51 globs and dies with `curl: (3) bad range in position N`. Either:52 ```bash53 curl --globoff -sS "https://target/?__proto__%5Btest%5D=polluted"54 curl --globoff -sS -G "https://target/path" --data-urlencode "__proto__[test]=polluted"55 ```564. **Read the response like a signal, not a verdict:**57 - `403` + JSON access-policy / `CF-RAY` → retry with browser headers (step 2).58 - Still `403` and HTML contains `Cf-Mitigated: challenge` → a real browser or an authenticated59 session is required; **do not** brute-force the edge.60 - SPA route that returns `200` with an identical body size for every path → front-end router,61 not a real API hit — don't log it as an endpoint.625. **Feed curl requests to other tools:** once a request works, export/replay it exactly — save it63 for `sqlmap -r` (`tools-sqlmap`) or send it to Burp Repeater (`tools-burp-suite`) instead of64 re-deriving it.6566## Gotchas67- **`curl -I` sends HEAD** — some apps 405 or behave differently on HEAD; for a real GET with68 headers shown, use `curl -s -o /dev/null -D -`.69- **Windows `curl.exe` vs the alias** — in PowerShell `curl` may be an alias for70 `Invoke-WebRequest`; call `curl.exe` explicitly, and remember `--globoff` for `[]`.71- **Windows `httpx` collision** — the `httpx` command may resolve to the Python httpx library CLI,72 not ProjectDiscovery's httpx; check `httpx -version` before piping recon output into it.73- **A working browser-header bypass is recon, not a bypass finding** — it means the edge admits74 browser-shaped traffic; record it and continue, don't report it.75- **Timeouts are part of the command** — always `--max-time`; a hung curl in an automated loop76 silently stalls the whole pipeline.7778## Verify success79You have a reproducible, copy-pasteable request that returns the expected status/body through the80edge, saved verbatim in the engagement notes so any later tool (sqlmap, Repeater, a finding's81repro steps) uses the exact same bytes.8283## References84curl manual (`--globoff`, `--data-urlencode`); Cloudflare edge behavior. Pipeline context:85`automation-recon-pipeline`.8687---88_Portions adapted from [reverse-skill](https://github.com/zhaoxuya520/reverse-skill) by zhaoxuya520, MIT License._