Url Parsing Security

URL Parsing Security Engineer

Secures URL parsing and query string handling against ampersand injection, double-encoding bypasses, parser inconsistencies, and parameter pollution across Python, Node.js, and Go applications. Treat every URL — whether from an HTTP request, a configuration file, a user-facing redirect target, or an API callback — as adversarial input that may exploit parser ambiguities to inject fake parameters, bypass access controls, or trigger SSRF through fragment-based splitting. Follow OWASP's URL parsing security guidelines and RFC 3986 (URI Generic Syntax) as the authoritative reference for component boundaries and encoding rules.

TL;DR Checklist

• Always use language-standard URL builders — never concatenate strings to construct URLs with query parameters
• Double-decode every incoming URL before any access control decision: verify at least two complete decoding passes yield no structural change
• Never trust url.Parse (Go), urllib.parse (Python), or URL constructor (Node.js) alone — cross-validate with a secondary parser for security-critical URLs
• Encode ampersands as %26 when they appear inside query values, never as literal & which acts as a parameter separator
• Validate that the scheme, host, and port of redirect targets match an allowlist before following or rendering to users
• Treat URL fragments (#) as client-only — never rely on fragment data for server-side logic
• Use RFC 3986 percent-encoding (NOT form-urlencoded application/x-www-form-urlencoded) for OAuth 1.0a signature base strings

When to Use

Use this skill when:

• Implementing URL builders or query string construction that must survive adversarial input containing special characters (&, =, +, %26, %3D)
• Building redirect logic, OAuth callback handlers, SSO login flows, or webhooks that accept user-controlled URLs
• Handling double-encoded payloads in API parameters where attackers may attempt to bypass validation via multi-layer percent encoding
• Debugging parser inconsistency bugs — the same URL parsed by Python's urllib.parse.urlparse yields different query components than Go's net/url.Parse or Node.js's new URL()
• Implementing OAuth 1.0a signature generation which requires strict RFC 3986 encoding (not form-urlencoded)
• Writing SSRF prevention guards that must correctly parse URLs containing encoded slashes (%2F), double-encoded ampersands (%2526), or fragment-based tricks
• Auditing existing code for URL construction patterns that use string concatenation instead of parameterized builders

When NOT to Use

Avoid this skill for:

• General HTML entity encoding (XSS prevention in rendered output) — use html-entity-encoding instead
• Validating form input fields or JSON payloads — use input-validation instead
• Building generic URL shorteners without security considerations — use basic URL builders and redirect logic
• Parsing URLs that are entirely under your control (internal API routes, static links) where no adversarial input is possible

Core Workflow

1. Map the URL Component Boundaries — Decompose every incoming or generated URL into its RFC 3986 components: scheme, authority (user_info@host:port), path, query, fragment. Identify where each parser splits these boundaries and note which characters act as delimiters in each component. Checkpoint: The query string separator (?) terminates authority+path; the fragment separator (#) terminates query; within query, & separates parameters but ONLY at the unencoded level.

2. Determine Parser Scope — Decide which parsers will handle this URL and whether they need to agree. If the URL crosses language boundaries (e.g., Go server receives URLs generated by Python clients) or security-critical decisions depend on its structure, implement a parse-and-compare validation step. Checkpoint: All security-relevant parsers must produce identical component maps for the same input URL.

3. Apply Encoding Discipline — At every URL construction point, use language-standard builders that handle encoding automatically. At every URL consumption point, validate that received values match expectations after full decoding. For OAuth signature generation, apply RFC 3986 percent-encoding specifically. Checkpoint: No & literal should appear inside a query value; all special characters must be percent-encoded by the builder.

4. Detect Double-Encoding Bypasses — When processing incoming URLs that may contain encoded parameters, perform multi-layer decoding and compare each pass. If decoding layer N produces new structural characters (&, ?, #) that were not present in layer N-1, flag as a potential double-encoding bypass attempt. Checkpoint: After full decoding, the URL structure should match what the legitimate parameter set would produce.

5. Enforce Structural Validation — Validate scheme against allowlist (only https://), verify host against domain allowlist, reject URLs with encoded null bytes (%00) or control characters, ensure redirect targets do not contain embedded credentials in the authority component. Checkpoint: Every URL that reaches a user-facing redirect or SSRF-sensitive operation passes structural validation.

Implementation Patterns / Reference Guide

Pattern 1: URL Anatomy and Parser Boundary Analysis

Understanding exactly where each parser splits a URL is critical for security. The ampersand (&) has fundamentally different meanings depending on which component it appears in:

Critical parser boundary facts:

• In Python's urllib.parse.urlparse(): query values containing & are NOT split — the entire string after ? and before # becomes query. Splitting happens only in parse_qs() or parse_qsl().
• In Go's url.Parse(): the Query() method on a parsed URL automatically decodes + to space and splits on &, then splits each part on the first =. A literal & in a value is preserved only if it was encoded as %26 by the sender — but Go's parser does NOT double-decode by default.
• In Node.js URL constructor: the .searchParams property uses the WHATWG URL Standard which automatically percent-decodes values. A literal & in a query value (HTML-encoded) will NOT be decoded — only %26 decoding happens.

from urllib.parse import urlparse, parse_qs, parse_qsl, unquote

def analyze_url_boundaries(url: str) -> dict:
    """Decompose a URL into RFC 3986 components and identify boundary characters.
    
    This reveals where each parser would split the URL and where ampersands
    act as structural delimiters vs. data characters.
    """
    parsed = urlparse(url)
    
    # Identify raw ampersand positions in query string
    raw_ampersand_positions: list[int] = []
    if parsed.query:
        for i, ch in enumerate(parsed.query):
            if ch == '&':
                raw_ampersand_positions.append(i)
    
    # Check what parse_qs produces (splits on unencoded &)
    qs_parsed = parse_qs(parsed.query, keep_blank_values=True)
    
    # Check what parse_qsl produces (preserves order, key=value pairs)
    qsl_parsed = parse_qsl(parsed.query, keep_blank_values=True)
    
    return {
        "url": url,
        "scheme": parsed.scheme,
        "authority": parsed.netloc,
        "path": parsed.path,
        "query_raw": parsed.query,
        "fragment": parsed.fragment,
        "ampersand_positions_in_query": raw_ampersand_positions,
        "parsed_params": qs_parsed,
        "parsed_pairs": qsl_parsed,
    }


# Example: Same URL parsed by different functions yields different results
url = "https://example.com/path?a=hello%26world&b=c&d"

result = analyze_url_boundaries(url)
# result["parsed_params"] == {"a": ["hello&world"], "b": ["c"], "d": [""]}
# The %26 in value "a" is correctly decoded to literal &, but parse_qs
# does NOT split on this decoded & — it stays inside the value.


# Adversarial: double-encoded ampersand in path that becomes & after decoding
tricky_url = "https://example.com/search?q=test%2526amp;b=evil"
result2 = analyze_url_boundaries(tricky_url)
# result2["query_raw"] == "q=test%2526amp;b=evil"
# parse_qs will see param "q" with value "test%2526amp" and param "b" with value "evil"
# BUT unquote("test%2526amp") → "test%26amp" → unquote again → "test&amp"
# This is the double-encoding bypass pattern — the attacker hides & behind %25

Pattern 2: Query Parameter Ampersand Handling Across Languages

Different languages handle the ampersand character differently in query strings. Understanding these differences prevents injection bugs when URLs cross language boundaries.

The core vulnerability: When a developer manually constructs a URL string and embeds user input that contains &, the input becomes a new parameter separator, injecting fake parameters into the query string.

# Python: Safe URL construction using urllib.parse (RFC 3986 compliant)
from urllib.parse import urlparse, urlencode, quote, urlunparse
from typing import Optional


def safe_url_build(
    base_url: str,
    params: dict[str, str | list[str] | None],
    fragment: Optional[str] = None,
    encode_special: bool = True
) -> str:
    """Build a URL safely using urllib.parse with proper percent-encoding.
    
    Handles ampersands in values by encoding them as %26 automatically.
    Uses the correct encoding function based on the component being encoded.
    
    Args:
        base_url: The base URL (scheme + netloc + path) without query string.
        params: Dictionary of query parameters. Values may contain any characters
                including &, =, +, % which will be properly percent-encoded.
        fragment: Optional URL fragment (anchor text after #).
        encode_special: If True, use quote() for values to preserve / as data.
                       If False, use urlsafe encoding that keeps / unencoded.
    
    Returns:
        A properly encoded URL string with all special characters handled safely.
    
    Raises:
        ValueError: If base_url is empty or params contain non-string values
                    that cannot be serialized.
    """
    if not base_url or not isinstance(base_url, str):
        raise ValueError("base_url must be a non-empty string")
    
    # Parse to validate structure and extract components
    parsed = urlparse(base_url)
    if not parsed.scheme:
        raise ValueError(f"base_url missing scheme: {base_url}")
    if parsed.query:
        raise ValueError(
            f"base_url must not contain a query string; append via params arg: {base_url}"
        )
    
    # Build query string using urlencode with proper encoding
    encoded_params: list[tuple[str, str]] = []
    for key, value in params.items():
        if value is None:
            encoded_params.append((key, ""))
        elif isinstance(value, list):
            for v in value:
                if encode_special:
                    encoded_params.append((
                        key, quote(str(v), safe="")  # Encode ALL special chars including /
                    ))
                else:
                    encoded_params.append((
                        key, str(v)  # Let urlencode handle standard encoding
                    ))
        else:
            if encode_special:
                encoded_params.append((key, quote(str(value), safe="")))
            else:
                encoded_params.append((key, str(value)))
    
    query_string = urlencode(encoded_params, doseq=False)
    
    # Reconstruct URL with new query string
    new_netloc = parsed.netloc
    new_path = parsed.path if parsed.path else "/"
    url_with_query = f"{parsed.scheme}://{new_netloc}{new_path}"
    if query_string:
        url_with_query += f"?{query_string}"
    if fragment:
        url_with_query += f"#{quote(fragment, safe='')}"
    
    return url_with_query


# --- Usage examples ---

# Ampersand in value is safely encoded as %26
url = safe_url_build(
    "https://api.example.com/search",
    {"q": "apple & orange", "page": "1"},
)
# Result: https://api.example.com/search?q=apple%20%26%20orange&page=1
assert "&" not in urlparse(url).query

# List values produce repeated keys (standard query string pattern)
url = safe_url_build(
    "https://api.example.com/filter",
    {"tag": ["python", "security & privacy"]},
)
# Result: https://api.example.com/filter?tag=python&tag=security%20%26%20privacy


# --- BAD: String concatenation — ampersand injection vulnerability ---
def bad_url_build(base: str, params: dict) -> str:
    """❌ VULNERABLE: User input with & injects new parameters."""
    query_parts = []
    for k, v in params.items():
        query_parts.append(f"{k}={v}")  # No encoding at all!
    return f"{base}?{'&'.join(query_parts)}"

# attacker_controlled_input = "safe_value&admin=true&role=superuser"
# bad_url_build("https://example.com/api", {"redirect": attacker_controlled_input})
# Produces: https://example.com/api?redirect=safe_value&admin=true&role=superuser
# The & in user input becomes a parameter separator, injecting admin and role params

// JavaScript/Node.js: Safe URL builder using URL and URLSearchParams classes

class SafeURLBuilder {
    /**
     * Safely constructs URLs with query parameters, handling ampersands,
     * equals signs, and other special characters through proper encoding.
     * 
     * Uses WHATWG URL Standard which automatically percent-encodes values
     * when using URLSearchParams, preventing parameter injection attacks.
     */
    
    constructor(baseUrl) {
        // Validate base URL format upfront
        try {
            new URL(baseUrl);
        } catch (err) {
            throw new TypeError(`Invalid base URL: ${baseUrl}`);
        }
        this.baseUrl = baseUrl;
        this._params = new URLSearchParams();
        this._fragment = null;
    }
    
    /**
     * Add a query parameter. Values containing &, =, +, % are safely encoded.
     * Multiple calls with the same key produce repeated params (e.g., ?tag=a&tag=b).
     * 
     * @param {string} name - Parameter name (must not contain & or =)
     * @param {string|number|null|undefined} value - Value to encode; null/undefined omitted
     * @returns {SafeURLBuilder} This instance for chaining
     */
    param(name, value) {
        if (!name || typeof name !== 'string') {
            throw new TypeError('Parameter name must be a non-empty string');
        }
        
        // Skip null/undefined values silently (standard URLSearchParams behavior)
        if (value === null || value === undefined) {
            return this;
        }
        
        this._params.append(name, String(value));
        return this;
    }
    
    /**
     * Set multiple parameters at once from a plain object.
     * @param {Record<string, string|string[]>} obj - Key-value pairs
     * @returns {SafeURLBuilder} This instance for chaining
     */
    params(obj) {
        if (typeof obj !== 'object' || obj === null) {
            throw new TypeError('params() requires an object');
        }
        for (const [key, value] of Object.entries(obj)) {
            if (Array.isArray(value)) {
                for (const v of value) {
                    this.param(key, v);
                }
            } else {
                this.param(key, value);
            }
        }
        return this;
    }
    
    /**
     * Set the URL fragment (anchor). Note: fragments are client-side only —
     * never sent to the server in HTTP requests.
     * @param {string} hash - Fragment text after #
     * @returns {SafeURLBuilder} This instance for chaining
     */
    fragment(hash) {
        if (hash !== null && typeof hash !== 'undefined') {
            this._fragment = String(hash);
        }
        return this;
    }
    
    /**
     * Build the final URL string. All special characters are percent-encoded
     * per WHATWG URL Standard. Ampersands in values become %26.
     * 
     * @returns {string} The complete, safely-encoded URL
     */
    toString() {
        const url = new URL(this.baseUrl);
        
        // Clear any existing query string and set our params
        url.search = '';
        if (this._params.toString()) {
            url.search = this._params.toString();
        }
        
        // Set fragment separately — it goes after #, not in the query
        if (this._fragment !== null) {
            url.hash = this._fragment;
        }
        
        return url.toString();
    }
    
    /**
     * Get the parsed parameters for inspection/debugging.
     * Values are already decoded by URLSearchParams.
     * @returns {Map<string, string[]>} Decoded parameter map
     */
    getParams() {
        const result = new Map();
        for (const [key, value] of this._params.entries()) {
            if (!result.has(key)) {
                result.set(key, []);
            }
            result.get(key).push(value);
        }
        return result;
    }
}


// --- Usage examples ---

// Ampersand in value is safely encoded as %26 by URLSearchParams
const url = new SafeURLBuilder('https://api.example.com/search')
    .param('q', 'apple & orange')
    .param('page', '1')
    .toString();
// Result: https://api.example.com/search?q=apple+%26+orange&page=1
// Note: URLSearchParams uses + for space (form-urlencoded style within search params)

// Multiple values for same key (array parameter)
const url2 = new SafeURLBuilder('https://api.example.com/filter')
    .param('tag', ['python'])
    .param('tag', 'security & privacy')
    .toString();
// Result: https://api.example.com/filter?tag=python&tag=security+%26+privacy

// Chaining with fragment (client-side only)
const url3 = new SafeURLBuilder('https://example.com/docs')
    .param('version', '2.0')
    .fragment('section-3')
    .toString();
// Result: https://example.com/docs?version=2.0#section-3


// --- BAD: Raw string concatenation — ampersand injection vulnerability ---
function badURLBuilder(base, params) {
    // ❌ VULNERABLE: No encoding of user-supplied values
    const parts = Object.entries(params).map(
        ([key, value]) => `${key}=${value}`  // & in value becomes param separator!
    );
    return `${base}?${parts.join('&')}`;
}

// const userInput = 'safe&admin=true';
// badURLBuilder('https://api.example.com/api', { redirect: userInput });
// Result: https://api.example.com/api?redirect=safe&admin=true
// The & in user input injects a new admin parameter

Pattern 3: Double-Encoding Bypass Detection

Double encoding occurs when %26 (encoded ampersand) is itself encoded as %2526 (the % becomes %25). An attacker may double-encode parameters to bypass validation logic that only performs a single decoding pass. Detect this by performing multiple decoding passes and checking if structural characters appear after decoding.

from urllib.parse import unquote, urlparse, parse_qs
from typing import NamedTuple


class DoubleEncodingResult(NamedTuple):
    """Result of double-encoding analysis on a URL query string."""
    is_double_encoded: bool
    raw_query: str
    decoded_once: str
    decoded_twice: str
    structural_changes_once: list[str]
    structural_changes_twice: list[str]
    decoded_params_once: dict[str, list[str]]
    decoded_params_twice: dict[str, list[str]]


def detect_double_encoding(query_string: str) -> DoubleEncodingResult:
    """Detect double-encoded payloads in a URL query string.
    
    Performs two decoding passes and checks for structural characters (&, =, ?, #)
    that appear after decoding but were not present in the raw input. This reveals
    bypass attempts where an attacker encodes & as %2526 to survive one-pass validation.
    
    Args:
        query_string: The raw query string (without leading ?).
        
    Returns:
        DoubleEncodingResult with decoded values and analysis of structural changes.
    """
    # First decoding pass
   
    # Second decoding pass  
    twice = unquote(once)
    
    # Structural characters that define URL boundaries
    STRUCTURAL_CHARS = {'&', '=', '?', '#'}
    
    def find_new_structural(original: str, decoded: str) -> list[str]:
        """Find structural characters in decoded string not present at same positions."""
        changes: list[str] = []
        for ch in STRUCTURAL_CHARS:
            if ch in decoded and ch not in original:
                changes.append(ch)
            elif ch in decoded and ch in original:
                # Count occurrences — more in decoded means new instances appeared
                if decoded.count(ch) > original.count(ch):
                    changes.append(f"{ch}+{decoded.count(ch)-original.count(ch)}")
        return changes
    
    changes_once = find_new_structural(query_string, once)
    changes_twice = find_new_structural(once, twice)
    
    # Parse with both decodings to see parameter structure change
    try:
        params_once = parse_qs(once, keep_blank_values=True)
    except Exception:
        params_once = {}
    try:
        params_twice = parse_qs(twice, keep_blank_values=True)
    except Exception:
        params_twice = {}
    
    return DoubleEncodingResult(
        is_double_encoded=bool(changes_once or changes_twice),
        raw_query=query_string,
        decoded_once=once,
        decoded_twice=twice,
        structural_changes_once=changes_once,
        structural_changes_twice=changes_twice,
        decoded_params_once=params_once,
        decoded_params_twice=params_twice,
    )


def fully_decode_url(url: str, max_passes: int = 3) -> dict:
    """Fully decode a URL through multiple passes until no further decoding is possible.
    
    Used to normalize adversarially encoded URLs before validation or access control.
    
    Args:
        url: The raw URL string that may contain double/triple encoding.
        max_passes: Maximum number of decoding iterations (prevents infinite loops).
        
    Returns:
        Dictionary with the fully decoded URL, all intermediate steps, and
        a flag indicating whether any structural changes occurred.
    """
    parsed = urlparse(url)
    current_query = parsed.query
    history: list[str] = [current_query]
    has_structural_change = False
    
    for i in range(max_passes):
        next_query = unquote(current_query)
        if next_query == current_query:
            break  # No more decoding possible
        
        history.append(next_query)
        
        # Check if structural characters appeared (new & or new =)
        prev_amps = current_query.count('&')
        curr_amps = next_query.count('&')
        if curr_amps > prev_amps:
            has_structural_change = True
        
        # Check if new key=value pairs emerged
        prev_eqs = current_query.count('=')
        curr_eqs = next_query.count('=')
        if curr_eqs > prev_eqs:
            has_structural_change = True
        
        current_query = next_query
    
    fully_decoded = parsed._replace(query=current_query)
    
    return {
        "original": url,
        "fully_decoded_url": fully_decoded.geturl(),
        "decoding_history": history,
        "total_passes": len(history) - 1,
        "structural_change_detected": has_structural_change,
        "final_params": parse_qs(current_query, keep_blank_values=True),
    }


# --- Usage examples ---

# Normal single-encoding: & in value properly encoded as %26
result = detect_double_encoding("name=hello%26world&page=1")
assert result.is_double_encoded is False  # The & was already a structural char after first decode

# Double-encoded bypass attempt: %2526 becomes & after two decodes
result2 = detect_double_encoding("search=test%2526amp&flag=true")
assert result2.is_double_encoded is True
assert "&" in result2.decoded_once  # %25 → %, so %2526 → %26 (literal string)
# After second decode: %26 → &, so "test%26amp" → "test&amp"
# This means the attacker's parameter "search=test&amp" was hidden behind double encoding


# Full URL decoding for access control validation
attack_url = "https://example.com/api?redirect=http://evil.com?admin=true"
safe_result = fully_decode_url(attack_url)
# This reveals that after decoding, new structural characters may appear

# Double-encoded SSRF attempt: %253A (double-encoded colon) decodes to : after two passes
ssrf_url = "https://example.com/proxy?url=http%253A%252F%252Finternal.server/admin"
ssrf_result = fully_decode_url(ssrf_url)
assert ssrf_result["structural_change_detected"] is True
# After full decode: http://internal.server/admin — a new SSRF target revealed

Pattern 4: OAuth 1.0a Compliant Encoder (RFC 3986)

OAuth 1.0a signature base string requires RFC 3986 percent-encoding, NOT application/x-www-form-urlencoded encoding. The difference matters: form encoding converts . to %2E, - to %2D, _ to %5F, and ~ to +. OAuth explicitly requires the strict RFC 3986 set which leaves these characters unencoded.

import hashlib
import hmac
import base64
import re
from urllib.parse import urlparse, urlencode, quote, parse_qs, urlunparse


# RFC 3986 unreserved characters that must NOT be percent-encoded:
# A-Z a-z 0-9 - _ . ~
RFC3986_UNRESERVED = set(
    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_.~"
)


def rfc3986_encode(value: str) -> str:
    """Encode a string using strict RFC 3986 percent-encoding.
    
    Unlike urllib.parse.quote() or form_urlencode, this function:
    - Leaves ~ unencoded (form encoding converts it to +)
    - Leaves . - _ unencoded (form encoding may alter these)
    - Encodes spaces as %20 (NOT +, which is form-encoding only)
    
    This matches the OAuth 1.0a specification exactly.
    
    Args:
        value: The raw string to encode. May contain any Unicode characters.
        
    Returns:
        RFC 3986 percent-encoded string safe for use in signature base strings.
    """
    # Percent-encode every byte, then restore unreserved characters
    encoded = quote(value.encode('utf-8'), safe='')
    
    # Restore RFC 3986 unreserved character set
    result = []
    i = 0
    while i < len(encoded):
        ch = encoded[i]
        if ch == '%':
            hex_str = encoded[i+1:i+3].upper()
            try:
                char_value = chr(int(hex_str, 16))
                if char_value in RFC3986_UNRESERVED:
                    # Use the original unencoded character
                    result.append(char_value)
                else:
                    # Keep percent-encoded, uppercase hex
                    result.append('%' + hex_str)
                i += 3
            except (ValueError, IndexError):
                result.append(ch)
                i += 1
        else:
            result.append(ch)
            i += 1
    
    return ''.join(result)


def oauth_normalize_parameters(params: dict[str, str], sort: bool = True) -> str:
    """Normalize OAuth 1.0a parameters into a sorted, encoded string.
    
    Per RFC 5849 Section 3.4.1.3.2:
    1. Percent-encode both keys and values with RFC 3986 encoding
    2. Sort by key (byte-order), then by value for duplicate keys
    3. Join with & using key=value format
    
    Args:
        params: Raw parameter dictionary from the HTTP request.
        sort: Whether to sort parameters (required for OAuth signature).
        
    Returns:
        Normalized parameter string suitable for signature base string calculation.
    """
    encoded_items: list[tuple[str, str]] = []
    
    for key, value in params.items():
        enc_key = rfc3986_encode(key)
        enc_value = rfc3986_encode(value)
        encoded_items.append((enc_key, enc_value))
    
    if sort:
        # Sort by encoded key first, then by encoded value
        encoded_items.sort(key=lambda item: (item[0], item[1]))
    
    return '&'.join(f"{key}={value}" for key, value in encoded_items)


def build_oauth_signature_base_string(
    http_method: str,
    base_url: str,
    params: dict[str, str]
) -> str:
    """Build the OAuth 1.0a signature base string per RFC 5849 Section 3.4.1.1.
    
    The signature base string has three components joined by &:
    1. HTTP method (GET, POST, etc.) — uppercased
    2. Base URL — scheme + host + path, with normalized query parameters removed
    3. Normalized parameters — RFC 3986-encoded, sorted key=value pairs
    
    Args:
        http_method: HTTP method string (e.g., "GET", "POST").
        base_url: Full request URL including query string.
        params: All OAuth and request parameters combined.
        
    Returns:
        The signature base string ready for HMAC-SHA1 signing.
    """
    # Normalize method
    method = http_method.upper()
    
    # Parse the URL and normalize it per OAuth spec
    parsed = urlparse(base_url)
    
    # Lowercase scheme and host
    normalized_host = f"{parsed.scheme.lower()}://{parsed.netloc.lower()}"
    
    # Remove default ports (80 for http, 443 for https)
    if (parsed.scheme == 'http' and parsed.port == 80) or \
       (parsed.scheme == 'https' and parsed.port == 443):
        normalized_host = f"{parsed.scheme.lower()}://{parsed.hostname.lower()}"
    
    # Normalize path — ensure it starts with /
    path = parsed.path if parsed.path else '/'
    
    url_base = f"{normalized_host}{path}"
    
    # Encode parameters using strict RFC 3986 (not form-urlencoded)
    params_string = oauth_normalize_parameters(params)
    
    # Join the three components
    return '&'.join([method, rfc3986_encode(url_base), rfc3986_encode(params_string)])


def generate_oauth_signature(
    http_method: str,
    base_url: str,
    params: dict[str, str],
    consumer_secret: str,
    token_secret: str = ''
) -> str:
    """Generate an OAuth 1.0a HMAC-SHA1 signature for a request.
    
    This implements the core signing algorithm used by Twitter API, 
    Stripe, GitHub, and most other OAuth 1.0a providers.
    
    Args:
        http_method: HTTP method (GET, POST, DELETE, etc.).
        base_url: The full request URL.
        params: Combined OAuth credentials + request parameters.
        consumer_secret: Your application's OAuth consumer secret key.
        token_secret: The authorized user's token secret (empty for initial requests).
        
    Returns:
        Base64-encoded HMAC-SHA1 signature string.
    """
    base_string = build_oauth_signature_base_string(http_method, base_url, params)
    
    signing_key = '&'.join([
        rfc3986_encode(consumer_secret),
        rfc3986_encode(token_secret)
    ])
    
    # HMAC-SHA1 signature
    key_bytes = signing_key.encode('utf-8')
    msg_bytes = base_string.encode('utf-8')
    signature = hmac.new(key_bytes, msg_bytes, hashlib.sha1).digest()
    
    # Base64 encode the binary signature
    return base64.b64encode(signature).decode('ascii')


# --- Usage example: Twitter API OAuth 1.0a signing ---

oauth_params = {
    "oauth_consumer_key": "jpPtK3mFsY...",
    "oauth_token": "12345678-AbCdEf...",
    "oauth_signature_method": "HMAC-SHA1",
    "oauth_timestamp": "1609459200",
    "oauth_nonce": "jKlMnOpQrStUvWxYz",
    "oauth_version": "1.0",
    "status": "Hello, world! & more text",  # Ampersand in parameter value
}

signature = generate_oauth_signature(
    http_method="POST",
    base_url="https://api.twitter.com/1.1/statuses/update.json",
    params=oauth_params,
    consumer_secret="your_consumer_secret_here",
    token_secret="your_token_secret_here",
)
# The ampersand in "Hello, world! & more text" is correctly handled:
# - rfc3986_encode converts it to %26 (NOT +)
# - The signature base string uses this exact encoding

Pattern 5: Go SafeURL Struct with QueryEscape and Fragment Handling

Go's net/url package provides QueryEscape() which produces RFC 3986-compliant encoding (same as OAuth). Use it for query parameters. For fragment handling, Go does NOT automatically encode fragments — you must use PathEscape() or manual encoding before appending to the URL string.

package urlsecurity

import (
	"fmt"
	"net/url"
	"strings"
)

// SafeURL provides a type-safe way to construct URLs with proper encoding,
// preventing ampersand injection and double-encoding vulnerabilities.
type SafeURL struct {
	Scheme   string
	Host     string
	Path     string
	Query    map[string][]string
	Fragment string
}

// NewSafeURL creates a SafeURL from its components. The scheme defaults to "https"
// if empty, and the path defaults to "/" if empty.
func NewSafeURL(scheme, host, path string) *SafeURL {
	s := &SafeURL{
		Scheme: scheme,
		Host:   host,
		Path:   path,
		Query:  make(map[string][]string),
	}
	if s.Scheme == "" {
		s.Scheme = "https"
	}
	if s.Path == "" {
		s.Path = "/"
	}
	return s
}

// AddQuery adds a single query parameter with safe encoding.
// Values are encoded using url.QueryEscape (RFC 3986 compatible),
// which correctly encodes & as %26, preventing parameter injection.
func (s *SafeURL) AddQuery(key, value string) {
	s.Query[key] = append(s.Query[key], value)
}

// AddQueryMulti adds a key that can have multiple values (e.g., ?tag=a&tag=b).
// Each value is independently encoded, preserving the multi-value semantics.
func (s *SafeURL) AddQueryMulti(key string, values ...string) {
	for _, v := range values {
		s.Query[key] = append(s.Query[key], v)
	}
}

// SetFragment sets the URL fragment (anchor text). The fragment is NOT
// automatically encoded by url.URL.String(), so we encode it manually.
// Fragments are client-side only and never sent to the server.
func (s *SafeURL) SetFragment(fragment string) {
	// Encode special characters in fragment: space, #, ?, %, &, etc.
	// Use PathEscape for general encoding, then fix remaining needs
	s.Fragment = url.PathEscape(fragment)
}

// Build returns the fully constructed URL string with all components
// properly percent-encoded per RFC 3986.
func (s *SafeURL) Build() string {
	u := &url.URL{
		Scheme:   s.Scheme,
		Host:     s.Host,
		Path:     s.Path,
		RawQuery: s.buildEncodedQuery(),
	}

	if s.Fragment != "" {
		u.Fragment = s.Fragment
	}

	return u.String()
}

// buildEncodedQuery constructs the query string with proper encoding.
// url.Values.Encode() uses application/x-www-form-urlencoded which encodes
// spaces as +. We use manual QueryEscape to stay RFC 3986 compliant.
func (s *SafeURL) buildEncodedQuery() string {
	var parts []string
	for key, values := range s.Query {
		encodedKey := url.QueryEscape(key)
		for _, value := range values {
			encodedValue := url.QueryEscape(value)
			parts = append(parts, fmt.Sprintf("%s=%s", encodedKey, encodedValue))
		}
	}

	if len(parts) == 0 {
		return ""
	}

	// Sort for deterministic output (important for OAuth signing)
	sortStrings(parts)
	return strings.Join(parts, "&")
}

// --- BAD: Manual string concatenation in Go — ampersand injection vulnerability ---

func badURLBuild(baseURL string, params map[string]string) string {
	queryParts := make([]string, 0, len(params))
	for k, v := range params {
		// ❌ VULNERABLE: No encoding of user input
		queryParts = append(queryParts, fmt.Sprintf("%s=%s", k, v))
	}
	return baseURL + "?" + strings.Join(queryParts, "&")
}

// --- BAD: Using url.Values.Encode() which converts spaces to + instead of %20 ---

func badURLEncode(baseURL string, params map[string][]string) string {
	values := make(url.Values)
	for k, v := range params {
		values[k] = v  // url.Values.Encode() uses form-urlencoded encoding
	}
	return baseURL + "?" + values.Encode()
}

// --- GOOD: Cross-parser validation for security-critical URLs ---

// ParseAndCompare verifies that two different URL parsers agree on the structure.
// This detects parser inconsistency bugs that can lead to security bypasses.
func ParseAndCompare(urlStr string) (*ParseResult, error) {
	if urlStr == "" {
		return nil, fmt.Errorf("url string is empty")
	}

	// Parser 1: Go's net/url.Parse (WHATWG-ish for scheme/host/path)
	goURL, goErr := url.Parse(urlStr)
	
	// For cross-validation with a Python-style parser, we simulate the behavior
	// by parsing query parameters differently and checking for structural mismatches
	
	result := &ParseResult{
		URL:         urlStr,
		GoParsed:    goURL,
		GoError:     goErr,
		StructuralChanges: []string{},
	}

	if goErr != nil {
		return result, nil // Go parser may reject non-standard URLs
	}

	// Check if query string has structural anomalies that differ between
	// raw parsing and parameter parsing
	rawQuery := goURL.RawQuery
	parsedValues, parseErr := url.ParseQuery(rawQuery)
	
	if parseErr != nil {
		result.GoParseError = parseErr
		return result, nil
	}

	// Check for double-encoding in query values
	for key, values := range parsedValues {
		for _, val := range values {
			// If decoding the value again reveals new & or = characters,
			// it's a double-encoding candidate
			doubleDecoded, dErr := url.QueryUnescape(val)
			if dErr == nil && doubleDecoded != val {
				if strings.ContainsRune(doubleDecoded, '&') || 
				   strings.Count(doubleDecoded, '=') > strings.Count(val, '=') {
					result.StructuralChanges = append(result.StructuralChanges,
						fmt.Sprintf("double-encoding in param %q: %s → %s", key, val, doubleDecoded))
				}
			}
		}
	}

	return result, nil
}

type ParseResult struct {
	URL               string
	GoParsed          *url.URL
	GoError           error
	GoParseError      error
	StructuralChanges []string
	DoubleEncoded     bool
}


// sortStrings sorts a slice of strings in place (simple insertion sort for small slices).
func sortStrings(slice []string) {
	for i := 1; i < len(slice); i++ {
		key := slice[i]
		j := i - 1
		for j >= 0 && slice[j] > key {
			slice[j+1] = slice[j]
			j--
		}
		slice[j+1] = key
	}
}


// --- Usage examples ---

func ExampleSafeURL() {
	// Build a safe URL with user-supplied parameters containing special chars
	u := NewSafeURL("https", "api.example.com", "/search")
	u.AddQuery("q", "apple & orange")
	u.AddQuery("page", "1")
	u.SetFragment("results-section-1")
	
	result := u.Build()
	// Result: https://api.example.com/search?page=1&q=apple%20%26%20orange#results-section-1
	// The & in the query value is encoded as %26 — no parameter injection possible

	// SSRF prevention example
	func validateRedirectURL(rawInput string) error {
		parsed, err := url.Parse(rawInput)
		if err != nil {
			return fmt.Errorf("invalid URL: %w", err)
		}
		
		// Check scheme allowlist
		if parsed.Scheme != "https" {
			return fmt.Errorf("only https:// redirects are allowed")
		}
		
		// Check for embedded credentials (user:pass@host) — SSRF vector
		if parsed.User != nil && parsed.User.Username() != "" {
			return fmt.Errorf("URL must not contain embedded credentials")
		}
		
		// Decode path to detect encoded slash attacks
		decodedPath, _ := url.PathUnescape(parsed.Path)
		if decodedPath == "/" || decodedPath == "/admin" {
			return fmt.Errorf("redirect to privileged paths is not allowed")
		}
		
		return nil // Passes all structural checks
	}
}

Pattern 6: Parse-and-Compare Validator (Cross-Library Consistency)

When URLs cross language boundaries or security decisions depend on their structure, verify that two different parsers agree. Mismatches indicate either a malformed URL or an encoding bypass attempt.

from urllib.parse import urlparse, parse_qs, unquote, quote
import json


class ParserInconsistencyError(Exception):
    """Raised when two URL parsers disagree on the structure of the same URL.""

…(truncated)