Wstg Info 03

wstg-info-03

Test ID

WSTG-INFO-03

Test Name

Review Webserver Metafiles for Information Leakage

High-Level Description

Web servers contain various metafiles that can inadvertently expose sensitive information about the application structure, hidden directories, restricted areas, and internal configurations. Files such as robots.txt, sitemap.xml, security.txt, and META tags are often overlooked but can provide attackers with valuable reconnaissance data. This test involves systematically reviewing these metafiles to identify information leakage and potential attack vectors.

What to Check

Metafiles to Examine

• robots.txt - Spider/crawler directives
• sitemap.xml - Site structure mapping
• sitemap_index.xml - Sitemap index files
• security.txt - Security policy disclosure
• humans.txt - Team/contributor information
• crossdomain.xml - Flash cross-domain policy
• clientaccesspolicy.xml - Silverlight policy
• .well-known/ directory contents
• HTML META tags - Robot directives
• Open Graph and Twitter Card metadata
• ads.txt - Authorized digital sellers
• app-ads.txt - Mobile app advertising
• assetlinks.json - Android app links
• apple-app-site-association - iOS app links

Information to Identify

• Hidden or restricted directories
• Administrative panels and login pages
• API endpoints
• Backup directories
• Development/staging paths
• Internal IP addresses
• Technology stack indicators
• Team member names and roles
• Security contact information
• PGP/GPG keys

How to Test

Step 1: Retrieve and Analyze robots.txt

# Download robots.txt
curl -s https://target.com/robots.txt

# Save to file
curl -s https://target.com/robots.txt -o robots.txt

# Check for multiple user-agent sections
curl -s https://target.com/robots.txt | grep -i "user-agent\|disallow\|allow\|sitemap"

robots.txt Directives to Analyze

Common Sensitive Paths in robots.txt

/admin/
/administrator/
/backup/
/config/
/database/
/db/
/dev/
/internal/
/login/
/logs/
/private/
/secret/
/staging/
/test/
/tmp/
/upload/
/uploads/
/wp-admin/
/api/
/cgi-bin/

Step 2: Analyze Sitemaps

# Get main sitemap
curl -s https://target.com/sitemap.xml | xmllint --format -

# Check common sitemap locations
for path in sitemap.xml sitemap_index.xml sitemap1.xml sitemaps/sitemap.xml; do
    echo "=== $path ==="
    curl -s "https://target.com/$path" | head -20
done

# Extract all URLs from sitemap
curl -s https://target.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+'

# Find nested sitemaps
curl -s https://target.com/sitemap.xml | grep -i "sitemap"

Sitemap Types

Step 3: Check security.txt

# RFC 9116 compliant location
curl -s https://target.com/.well-known/security.txt

# Alternative location
curl -s https://target.com/security.txt

# Extract useful information
curl -s https://target.com/.well-known/security.txt | grep -i "contact\|encryption\|policy\|hiring\|acknowledgments"

security.txt Fields

Step 4: Examine humans.txt

# Get humans.txt
curl -s https://target.com/humans.txt

# Look for names and roles
curl -s https://target.com/humans.txt | grep -i "team\|developer\|designer\|lead\|manager"

Step 5: Scan .well-known Directory

# Common .well-known files
for file in \
    security.txt \
    openid-configuration \
    assetlinks.json \
    apple-app-site-association \
    change-password \
    dnt-policy.txt \
    host-meta \
    host-meta.json \
    mta-sts.txt \
    nodeinfo \
    webfinger \
    matrix/client \
    matrix/server \
    acme-challenge \
    pki-validation \
    traffic-advice; do
    echo "=== .well-known/$file ==="
    curl -s "https://target.com/.well-known/$file" | head -10
done

Step 6: Analyze HTML META Tags

# Get page source and extract META tags
curl -s https://target.com | grep -i "<meta"

# Look for robot directives
curl -s https://target.com | grep -i "name=\"robots\""

# Extract Open Graph data
curl -s https://target.com | grep -i "og:"

# Find generator information
curl -s https://target.com | grep -i "generator"

META Tag Analysis

<!-- Robot directives -->
<meta name="robots" content="noindex, nofollow" />
<meta name="googlebot" content="noindex" />

<!-- Generator (CMS info) -->
<meta name="generator" content="WordPress 6.0" />

<!-- Technology indicators -->
<meta name="csrf-token" content="..." />
<meta name="viewport" content="..." />

Step 7: Check Additional Metafiles

# Cross-domain policy files
curl -s https://target.com/crossdomain.xml
curl -s https://target.com/clientaccesspolicy.xml

# Advertising files
curl -s https://target.com/ads.txt
curl -s https://target.com/app-ads.txt

# App association files
curl -s https://target.com/.well-known/assetlinks.json
curl -s https://target.com/.well-known/apple-app-site-association
curl -s https://target.com/apple-app-site-association

Step 8: Google Dorking for Metafiles

# Find robots.txt via Google
site:target.com inurl:robots.txt

# Find sitemaps
site:target.com inurl:sitemap filetype:xml

# Find exposed directories from robots.txt
site:target.com inurl:admin OR inurl:backup OR inurl:config

Tools

Command-Line Tools

Automated Scanners

Online Tools

Example Commands/Payloads

Comprehensive Metafile Scan Script

#!/bin/bash
TARGET=$1

echo "=== METAFILE SCANNER ==="
echo "Target: $TARGET"
echo ""

# robots.txt
echo "[+] Checking robots.txt..."
curl -s "https://$TARGET/robots.txt" -o robots.txt
if [ -s robots.txt ]; then
    echo "Found robots.txt:"
    cat robots.txt
    echo ""
    echo "Disallowed paths:"
    grep -i "disallow" robots.txt | awk '{print $2}'
fi

echo ""

# sitemap.xml
echo "[+] Checking sitemap.xml..."
curl -s "https://$TARGET/sitemap.xml" -o sitemap.xml
if [ -s sitemap.xml ]; then
    echo "Found sitemap.xml"
    echo "URLs count: $(grep -c "<loc>" sitemap.xml)"
fi

echo ""

# security.txt
echo "[+] Checking security.txt..."
for path in ".well-known/security.txt" "security.txt"; do
    response=$(curl -s "https://$TARGET/$path")
    if [[ ! -z "$response" && ! "$response" =~ "404" ]]; then
        echo "Found at /$path:"
        echo "$response"
        break
    fi
done

echo ""

# humans.txt
echo "[+] Checking humans.txt..."
curl -s "https://$TARGET/humans.txt"

echo ""

# .well-known directory
echo "[+] Scanning .well-known/..."
for file in security.txt openid-configuration assetlinks.json apple-app-site-association; do
    status=$(curl -s -o /dev/null -w "%{http_code}" "https://$TARGET/.well-known/$file")
    if [ "$status" == "200" ]; then
        echo "Found: .well-known/$file"
    fi
done

Parsero - robots.txt Analyzer

# Install
pip install parsero

# Basic usage
parsero -u https://target.com

# Check disallowed entries
parsero -u https://target.com -sb

# Output to file
parsero -u https://target.com -o output.txt

Burp Suite Crawler Configuration

1. Target > Site map > Right-click target
2. Spider this host
3. Spider > Control > Start
4. Review discovered paths in Site map

OWASP ZAP Spider

# Using ZAP CLI
zap-cli quick-scan -s all -r https://target.com

# Spider specific URL
zap-cli spider https://target.com

Extract URLs from Sitemap (Python)

#!/usr/bin/env python3
import requests
import xml.etree.ElementTree as ET

def parse_sitemap(url):
    response = requests.get(url)
    root = ET.fromstring(response.content)

    # Handle namespace
    ns = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}

    urls = []
    for url_elem in root.findall('.//ns:loc', ns):
        urls.append(url_elem.text)

    return urls

# Usage
urls = parse_sitemap('https://target.com/sitemap.xml')
for url in urls:
    print(url)

Remediation Guide

1. robots.txt Best Practices

# Good - Generic exclusions
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/

# Bad - Reveals sensitive paths
User-agent: *
Disallow: /admin-secret-panel/
Disallow: /backup-20240115/
Disallow: /api/v2/internal/

Recommendations:

• Do not use robots.txt as a security mechanism
• Use authentication for sensitive areas instead
• Consider not listing truly sensitive paths at all
• Monitor for unauthorized access to "disallowed" paths

2. Sitemap Security

• Exclude sensitive URLs from sitemaps
• Keep sitemaps updated (remove deprecated pages)
• Don't include internal/development URLs
• Consider requiring authentication for full sitemaps

3. META Tag Hardening

<!-- Remove generator information -->
<!-- Bad -->
<meta name="generator" content="WordPress 6.0" />

<!-- Good - Remove entirely or use generic -->
<meta name="generator" content="Custom CMS" />

<!-- Restrict indexing for sensitive pages -->
<meta name="robots" content="noindex, nofollow" />

4. security.txt Guidelines (RFC 9116)

# Include useful information
Contact: mailto:security@example.com
Encryption: https://example.com/.well-known/pgp-key.txt
Policy: https://example.com/security-policy
Preferred-Languages: en
Expires: 2025-12-31T23:59:59.000Z

# Sign the file with PGP for authenticity

5. humans.txt Considerations

• Avoid exposing full names
• Don't include email addresses
• Use roles instead of personal details
• Consider removing if not necessary

6. .well-known Directory

• Audit all files in .well-known
• Remove unnecessary entries
• Restrict access to sensitive configurations
• Monitor for unauthorized additions

Risk Assessment

CVSS Score

Base Score: 5.3 (Medium)

CVSS Vector: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N

Severity Levels

CWE Categories

References

OWASP References

• OWASP WSTG - Review Webserver Metafiles

Standards

• RFC 9116 - security.txt
• RFC 5785 - .well-known URIs
• Google robots.txt Specification
• Sitemaps Protocol

Tools

• Parsero
• Gobuster

Checklist

[ ] robots.txt retrieved and analyzed
[ ] Disallowed paths documented
[ ] Sitemap.xml retrieved and parsed
[ ] All sitemap URLs extracted
[ ] security.txt checked (both locations)
[ ] humans.txt retrieved
[ ] .well-known directory scanned
[ ] HTML META tags analyzed
[ ] crossdomain.xml checked
[ ] clientaccesspolicy.xml checked
[ ] ads.txt checked
[ ] App association files checked
[ ] Hidden paths verified for accessibility
[ ] Sensitive information documented
[ ] Risk assessment completed
[ ] Remediation recommendations prepared