Url Tools

URL Tools Skill

URL manipulation and HTTP request tools.

When to Use

• Encode or decode URL components
• Check HTTP headers
• Test API endpoints
• Download files
• Debug HTTP issues

URL Encoding

Encode URL

# Encode a string for URL use
python3 -c "import urllib.parse; print(urllib.parse.quote('hello world'))"
# Output: hello%20world

Decode URL

# Decode URL-encoded string
python3 -c "import urllib.parse; print(urllib.parse.unquote('hello%20world'))"
# Output: hello world

HTTP Operations

Check Headers

# Get response headers
curl -I https://example.com

# Get specific header
curl -sI https://example.com | grep -i content-type

Test API

# GET request
curl -s https://api.example.com/data

# POST request
curl -s -X POST https://api.example.com/data \
  -H "Content-Type: application/json" \
  -d '{"key":"value"}'

# With authentication
curl -s https://api.example.com/protected \
  -H "Authorization: Bearer TOKEN"

Download File

# Download with progress
curl -L -O https://example.com/file.zip

# Resume download
curl -L -C - -O https://example.com/large-file.zip

Debug Request

# Verbose output
curl -v https://example.com

# Show only headers
curl -sI https://example.com

# Timing
curl -w "@/dev/stdin" -s -o /dev/null https://example.com <<'EOF'
time_namelookup:  %{time_namelookup}
time_connect:     %{time_connect}
time_starttransfer: %{time_starttransfer}
time_total:       %{time_total}
EOF

URL Parsing

Extract Parts

# Using Python
python3 -c "
from urllib.parse import urlparse
url = 'https://user:pass@example.com:8080/path?query=value#frag'
p = urlparse(url)
print(f'scheme: {p.scheme}')
print(f'netloc: {p.netloc}')
print(f'host: {p.hostname}')
print(f'port: {p.port}')
print(f'path: {p.path}')
print(f'query: {p.query}')
"

Common curl Flags

Examples

Check if site is up:

curl -s -o /dev/null -w "%{http_code}" https://example.com

Get JSON and format:

curl -s https://api.example.com/data | python3 -m json.tool

Test webhook:

curl -s -X POST https://webhook.example.com/hook \
  -H "Content-Type: application/json" \
  -d '{"event":"test","timestamp":"'"$(date -Iseconds)"'"}'