ADR-0015: Configurable Well-Known URI Handler

ADR-0015: Configurable Well-Known URI Handler

• Status: Accepted
• Date: 2025-08-17
• Deciders: Core Engineering Team
• Issues: #540
• Related: Security infrastructure for standardized web discovery

Context

ContextForge needed to support standardized well-known URIs as defined by RFC 8615 to enable proper web service discovery, security contact information, and crawler management. Well-known URIs are standardized endpoints that web services expose for automated discovery and security contact purposes.

The implementation needed to address:

• robots.txt for search engine crawler management (typically private API = disable crawling)
• security.txt for security contact information per RFC 9116
• Custom well-known files for additional service policies (AI usage, DNT policy, etc.)
• Security-first defaults appropriate for private API gateway deployment
• Configuration flexibility for different deployment scenarios
• Admin monitoring of well-known configuration status

Requirements

• Support standard well-known URIs (robots.txt, security.txt)
• Allow custom well-known files via configuration
• Default to private API security posture (no crawling)
• RFC 9116 compliant security.txt with automatic validation
• Configurable cache headers for performance
• Admin endpoint for configuration monitoring
• Environment-based configuration via standard patterns

Decision

We implemented a flexible /.well-known/* endpoint handler with the following design:

1. Router-Based Implementation

Created mcpgateway/routers/well_known.py with a dedicated FastAPI router:

@router.get("/.well-known/{filename:path}", include_in_schema=False)
async def get_well_known_file(filename: str, response: Response, request: Request):
    """Serve well-known URI files with configurable content and security defaults."""

Design decisions:

• Router isolation: Separate router for clean organization and testing
• Dynamic routing: Single endpoint handles all well-known URIs
• Security-first: Disabled by default, explicit enable required
• Schema exclusion: Not included in OpenAPI docs (reduces attack surface)

2. Configuration-Driven Content

Extended mcpgateway/config.py with well-known URI settings:

# Well-Known URI Configuration
well_known_enabled: bool = True
well_known_robots_txt: str = """User-agent: *
Disallow: /

# ContextForge is a private API gateway
# Public crawling is disabled by default"""

well_known_security_txt: str = ""
well_known_security_txt_enabled: bool = False
well_known_custom_files: str = "{}"  # JSON format
well_known_cache_max_age: int = 3600  # 1 hour

Design decisions:

• Private API defaults: robots.txt blocks all crawlers by default
• Explicit security.txt: Only enabled when content is provided
• JSON custom files: Flexible format for additional well-known files
• Configurable caching: Performance optimization with sensible defaults

3. RFC 9116 Security.txt Compliance

Implemented automatic security.txt validation and enhancement:

def validate_security_txt(content: str) -> Optional[str]:
    """Validate security.txt format and add required headers."""
    # Add Expires field if missing (6 months from now)
    # Add header comments for clarity
    # Preserve existing valid content

Design decisions:

• Auto-expires: Adds Expires header if missing (RFC requirement)
• Header comments: Adds generation timestamp and description
• Validation: Ensures RFC 9116 compliance
• Preservation: Maintains existing valid fields

4. Well-Known Registry

Implemented a registry for known well-known URIs with metadata:

WELL_KNOWN_REGISTRY = {
    "robots.txt": {
        "content_type": "text/plain",
        "description": "Robot exclusion standard",
        "rfc": "RFC 9309"
    },
    "security.txt": {
        "content_type": "text/plain",
        "description": "Security contact information",
        "rfc": "RFC 9116"
    },
    # ... additional standard URIs
}

Design decisions:

• Helpful errors: Provides descriptive 404 messages for known but unconfigured files
• Content-Type mapping: Ensures correct MIME types
• Documentation: Links to relevant RFCs and standards
• Extensibility: Easy to add new standard well-known URIs

5. Admin Monitoring

Added /admin/well-known endpoint for configuration visibility:

@router.get("/admin/well-known", response_model=dict)
async def get_well_known_status(user: str = Depends(require_auth)):
    """Returns configuration status and available well-known files."""

Design decisions:

• Authentication required: Admin endpoint requires JWT authentication
• Configuration visibility: Shows enabled files and cache settings
• Supported files list: Displays all known well-known URI types
• Status monitoring: Helps administrators verify configuration

Implementation Architecture

File Structure

mcpgateway/
├── config.py                 # Well-known configuration settings
├── routers/
│   └── well_known.py         # /.well-known/* endpoint handler
└── main.py                   # Router integration

tests/unit/mcpgateway/
└── test_well_known.py        # Comprehensive test coverage

.env.example                  # Configuration documentation

Request Flow

1. Request: GET /.well-known/robots.txt
2. Authentication: No auth required for well-known URIs (public by design)
3. Validation: Check if well-known endpoints are enabled
4. Routing: Match filename to configured content or registry
5. Headers: Add cache control and specific headers (X-Robots-Tag for robots.txt)
6. Response: Return PlainTextResponse with appropriate headers

Security Considerations

• No authentication: Well-known URIs are public by design per RFC 8615
• Content validation: security.txt content validated against RFC 9116
• Path traversal protection: Filename normalization prevents directory traversal
• Cache headers: Appropriate cache settings reduce server load
• Information disclosure: Default robots.txt reveals minimal information

Consequences

✅ Benefits

• Standards Compliance: Implements RFC 8615 (well-known URIs) and RFC 9116 (security.txt)
• Security Contact: Enables security researchers to find contact information
• Crawler Management: Proper robots.txt prevents unwanted search engine indexing
• Flexibility: Custom well-known files support organization-specific policies
• Performance: Configurable caching reduces server load for frequently accessed files
• Monitoring: Admin endpoint provides configuration visibility
• Private API Focused: Defaults appropriate for API gateway deployment

❌ Trade-offs

• Information Disclosure: Well-known URIs are public and may reveal service information
• Cache Headers: Public cache headers may not be appropriate for all deployments
• Configuration Complexity: Additional environment variables to manage
• Static Content: Well-known files are static and can't include dynamic information

🔄 Maintenance

• security.txt Updates: Requires periodic updates to contact information and expiration
• RFC Compliance: Monitor RFC updates for security.txt format changes
• Custom File Management: Organizations need to maintain custom well-known content
• Cache Tuning: May need cache duration adjustments based on usage patterns

Configuration Examples

Basic Private API (Default)

WELL_KNOWN_ENABLED=true
# robots.txt blocks all crawlers (default)
# security.txt disabled (default)

Public API with Security Contact

WELL_KNOWN_ENABLED=true
WELL_KNOWN_SECURITY_TXT="Contact: mailto:security@example.com\nContact: https://example.com/security\nPreferred-Languages: en"
WELL_KNOWN_ROBOTS_TXT="User-agent: *\nAllow: /health\nAllow: /docs\nDisallow: /"

Custom Policies

WELL_KNOWN_CUSTOM_FILES={"ai.txt": "AI Usage: Tool orchestration only", "dnt-policy.txt": "We honor Do Not Track headers"}

Alternatives Considered

Testing Strategy

Implemented comprehensive test coverage:

• Default robots.txt: Validates security-first defaults
• security.txt validation: Tests RFC 9116 compliance and auto-enhancement
• Custom files: Verifies JSON configuration parsing and serving
• 404 handling: Tests unknown files and helpful error messages
• Path normalization: Ensures path traversal protection
• Registry functionality: Validates well-known URI metadata

Future Enhancements

Potential improvements for future iterations:

• Dynamic content: Template variables (e.g., {{DOMAIN}}, {{CONTACT_EMAIL}})
• File upload API: Admin interface for uploading well-known files
• GPG signing: Digital signature support for security.txt
• Rate limiting: Specific limits for well-known endpoints
• Internationalization: Multi-language support for policy files
• A/B testing: Different content based on user agent or other criteria

Security Impact

Positive Security Impact

• Security contact: Enables responsible disclosure by security researchers
• Crawler control: Prevents unwanted indexing of private API endpoints
• Standards compliance: Follows established web security practices
• Information control: Explicit control over what information is disclosed

Security Considerations

• Information disclosure: Well-known URIs are intentionally public
• Content validation: Prevents serving malicious content through validation
• Cache control: Public caching may not be appropriate for all environments
• Admin endpoint: Configuration status requires authentication

Status

This well-known URI handler implementation is accepted and implemented as of version 0.7.0, providing standards-compliant web service discovery while maintaining security-first defaults appropriate for private API gateway deployments.