External Plugin mTLS Setup Guide

External Plugin mTLS Setup Guide

This guide covers how to set up mutual TLS (mTLS) authentication between ContextForge and external plugin servers.

Port Configuration

Standard port convention:

• Port 8000: Main plugin service (HTTP or HTTPS/mTLS)
• Port 9000: Health check endpoint (automatically starts on port+1000 when mTLS is enabled)

When mTLS is enabled, the plugin runtime automatically starts a separate HTTP-only health check server on port 9000 (configurable via port + 1000 formula). This allows health checks without requiring mTLS client certificates.

Certificate Generation

ContextForge includes Makefile targets to manage the complete certificate infrastructure for plugin mTLS.

Quick Start

# Generate complete mTLS infrastructure (recommended)
make certs-mcp-all

# This automatically:
# 1. Creates a Certificate Authority (CA)
# 2. Generates gateway client certificate
# 3. Reads plugins/external/config.yaml and generates server certificates for all external plugins

Certificate validity: Default is 825 days (~2.25 years)

Output structure:

certs/mcp/
├── ca/                           # Certificate Authority
│   ├── ca.key                   # CA private key (protect!)
│   └── ca.crt                   # CA certificate
├── gateway/                      # Gateway client certificates
│   ├── client.key               # Client private key
│   ├── client.crt               # Client certificate
│   └── ca.crt                   # Copy of CA cert
└── plugins/                      # Plugin server certificates
    └── PluginName/
        ├── server.key           # Server private key
        ├── server.crt           # Server certificate
        └── ca.crt               # Copy of CA cert

Makefile Targets

make certs-mcp-all

Generate complete mTLS infrastructure. This is the recommended command for setting up mTLS.

What it does:

1. Calls certs-mcp-ca to create the CA (if not exists)
2. Calls certs-mcp-gateway to create gateway client certificate (if not exists)
3. Reads plugins/external/config.yaml and generates certificates for all plugins with kind: external

Usage:

# Use default config file (plugins/external/config.yaml)
make certs-mcp-all

# Use custom config file
make certs-mcp-all MCP_PLUGIN_CONFIG=path/to/custom-config.yaml

# Custom certificate validity (in days)
make certs-mcp-all MCP_CERT_DAYS=365

# Combine both options
make certs-mcp-all MCP_PLUGIN_CONFIG=config.yaml MCP_CERT_DAYS=730

Config file format (plugins/external/config.yaml):

plugins:
  - name: "MyPlugin"           # Certificate will be created for this plugin
    kind: "external"           # Must be "external"
    mcp:
      proto: STREAMABLEHTTP
      url: http://127.0.0.1:8000/mcp

  - name: "AnotherPlugin"
    kind: "external"
    mcp:
      proto: STREAMABLEHTTP
      url: http://127.0.0.1:8001/mcp

Fallback behavior: If the config file doesn't exist or PyYAML is not installed, example certificates are generated for example-plugin-a and example-plugin-b.

make certs-mcp-ca

Generate the Certificate Authority (CA) for plugin mTLS. This is typically called automatically by other targets.

What it does:

• Creates certs/mcp/ca/ca.key (4096-bit RSA private key)
• Creates certs/mcp/ca/ca.crt (CA certificate)
• Sets file permissions: 600 for .key, 644 for .crt

Usage:

# Generate CA (one-time setup)
make certs-mcp-ca

# Custom validity
make certs-mcp-ca MCP_CERT_DAYS=1825

Safety: Won't overwrite existing CA. To regenerate, delete certs/mcp/ca/ first.

⚠️ Warning: The CA private key (ca.key) is critical. Protect it carefully!

make certs-mcp-gateway

Generate the gateway client certificate used by ContextForge to authenticate to plugin servers.

What it does:

• Depends on certs-mcp-ca (creates CA if needed)
• Creates certs/mcp/gateway/client.key (4096-bit RSA private key)
• Creates certs/mcp/gateway/client.crt (client certificate signed by CA)
• Copies ca.crt to certs/mcp/gateway/

Usage:

# Generate gateway client certificate
make certs-mcp-gateway

# Custom validity
make certs-mcp-gateway MCP_CERT_DAYS=365

Safety: Won't overwrite existing certificate.

make certs-mcp-plugin

Generate a server certificate for a specific plugin.

What it does:

• Depends on certs-mcp-ca (creates CA if needed)

• Creates certs/mcp/plugins/<PLUGIN_NAME>/server.key

• Creates certs/mcp/plugins/<PLUGIN_NAME>/server.crt with Subject Alternative Names (SANs):

  • DNS:<PLUGIN_NAME>
  • DNS:mcp-plugin-<PLUGIN_NAME>
  • DNS:localhost

• Copies ca.crt to plugin directory

Usage:

# Generate certificate for specific plugin
make certs-mcp-plugin PLUGIN_NAME=MyCustomPlugin

# Custom validity
make certs-mcp-plugin PLUGIN_NAME=MyPlugin MCP_CERT_DAYS=365

Required: PLUGIN_NAME parameter must be provided.

Use case: Add a new plugin after running certs-mcp-all, or generate certificates manually.

make certs-mcp-check

Check expiry dates of all MCP certificates.

What it does:

• Displays expiry dates for CA, gateway client, and all plugin certificates
• Shows remaining validity period

Usage:

make certs-mcp-check

Output example:

🔍  Checking MCP certificate expiry dates...

📋 CA Certificate:
   Expires: Jan 15 10:30:45 2027 GMT

📋 Gateway Client Certificate:
   Expires: Jan 15 10:31:22 2027 GMT

📋 Plugin Certificates:
   MyPlugin: Jan 15 10:32:10 2027 GMT
   AnotherPlugin: Jan 15 10:32:45 2027 GMT

Certificate Properties

All certificates generated include:

• Algorithm: RSA with SHA-256
• CA Key Size: 4096 bits
• Client/Server Key Size: 4096 bits
• Default Validity: 825 days
• Subject Alternative Names (plugins): DNS entries for plugin name and localhost

Important Notes

1. All ca.crt files are identical - They are copies of the root CA certificate distributed to each location for convenience

2. Safety features - Commands won't overwrite existing certificates. To regenerate, delete the target directory first

3. File permissions - Automatically set to secure values:

   • Private keys (.key): 600 (owner read/write only)
   • Certificates (.crt): 644 (world-readable)

4. Configuration variables:

   • MCP_CERT_DAYS: Certificate validity in days (default: 825)
   • MCP_PLUGIN_CONFIG: Path to plugin config file (default: plugins/external/config.yaml)

Configuration Options

You can configure mTLS using either YAML files or environment variables.

Option 1: YAML Configuration

Server Configuration (Plugin)

In your plugin config file (e.g., plugins/test.yaml):

plugins:
  - name: "ReplaceBadWordsPlugin"
    kind: "plugins.regex_filter.search_replace.SearchReplacePlugin"
    # ... plugin config ...

server_settings:
  host: "127.0.0.1"
  port: 8000
  tls:
    certfile: certs/mcp/plugins/ReplaceBadWordsPlugin/server.crt
    keyfile: certs/mcp/plugins/ReplaceBadWordsPlugin/server.key
    ca_bundle: certs/mcp/plugins/ReplaceBadWordsPlugin/ca.crt
    ssl_cert_reqs: 2  # 2 = CERT_REQUIRED (enforce client certificates)

Start the server (for testing):

PYTHONPATH=. PLUGINS_CONFIG_PATH="plugins/test.yaml" \
  python3 mcpgateway/plugins/framework/external/mcp/server/runtime.py

Client Configuration (Gateway)

In your gateway plugin config file (e.g., plugins/external/config-client.yaml):

plugins:
  - name: "ReplaceBadWordsPlugin"
    kind: "external"
    mcp:
      proto: STREAMABLEHTTP
      url: https://127.0.0.1:8000/mcp
      tls:
        certfile: certs/mcp/gateway/client.crt
        keyfile: certs/mcp/gateway/client.key
        ca_bundle: certs/mcp/gateway/ca.crt
        verify: true
        check_hostname: false

Option 2: Environment Variables

Server Environment Variables

# Server configuration
export PLUGINS_SERVER_HOST="127.0.0.1"
export PLUGINS_SERVER_PORT="8000"
export PLUGINS_SERVER_SSL_ENABLED="true"

# TLS/mTLS configuration
export PLUGINS_SERVER_SSL_KEYFILE="certs/mcp/plugins/ReplaceBadWordsPlugin/server.key"
export PLUGINS_SERVER_SSL_CERTFILE="certs/mcp/plugins/ReplaceBadWordsPlugin/server.crt"
export PLUGINS_SERVER_SSL_CA_CERTS="certs/mcp/plugins/ReplaceBadWordsPlugin/ca.crt"
export PLUGINS_SERVER_SSL_CERT_REQS="2"  # 2 = CERT_REQUIRED

Start the server (YAML without server_settings section for testing):

PYTHONPATH=. PLUGINS_CONFIG_PATH="plugins/test.yaml" \
  python3 mcpgateway/plugins/framework/external/mcp/server/runtime.py

Client Environment Variables

export PLUGINS_CLIENT_MTLS_CERTFILE="certs/mcp/gateway/client.crt"
export PLUGINS_CLIENT_MTLS_KEYFILE="certs/mcp/gateway/client.key"
export PLUGINS_CLIENT_MTLS_CA_BUNDLE="certs/mcp/gateway/ca.crt"
export PLUGINS_CLIENT_MTLS_VERIFY="true"
export PLUGINS_CLIENT_MTLS_CHECK_HOSTNAME="false"

Run your gateway code (YAML without tls section in mcp config).

Environment Variable Reference

Server Variables (Plugin)

!!! note PLUGINS_SERVER_UDS and TLS are mutually exclusive. UDS runs without TLS by design.

ssl_cert_reqs values:

• 0 = CERT_NONE - No client certificate required
• 1 = CERT_OPTIONAL - Client certificate requested but not required
• 2 = CERT_REQUIRED - Client certificate required (mTLS)

Client Variables (Gateway)

Testing mTLS

Test without TLS

# Server
PYTHONPATH=. PLUGINS_CONFIG_PATH="plugins/test.yaml" \
  PLUGINS_SERVER_HOST="127.0.0.1" \
  PLUGINS_SERVER_PORT="8000" \
  PLUGINS_SERVER_SSL_ENABLED="false" \
  python3 mcpgateway/plugins/framework/external/mcp/server/runtime.py &

# Client config should use: url: http://127.0.0.1:8000/mcp

Test with mTLS (YAML)

# Server (config has server_settings.tls section)
PYTHONPATH=. PLUGINS_CONFIG_PATH="plugins/test.mtls.yaml" \
  python3 mcpgateway/plugins/framework/external/mcp/server/runtime.py &

# Client (config has mcp.tls section)
python3 your_client.py

Test with mTLS (Environment Variables)

# Server (config has no server_settings section)
# Note: When mTLS is enabled, a health check server automatically starts on port 9000 (port+1000)
PYTHONPATH=. \
  PLUGINS_CONFIG_PATH="plugins/test.yaml" \
  PLUGINS_SERVER_HOST="127.0.0.1" \
  PLUGINS_SERVER_PORT="8000" \
  PLUGINS_SERVER_SSL_ENABLED="true" \
  PLUGINS_SERVER_SSL_KEYFILE="certs/mcp/plugins/ReplaceBadWordsPlugin/server.key" \
  PLUGINS_SERVER_SSL_CERTFILE="certs/mcp/plugins/ReplaceBadWordsPlugin/server.crt" \
  PLUGINS_SERVER_SSL_CA_CERTS="certs/mcp/plugins/ReplaceBadWordsPlugin/ca.crt" \
  PLUGINS_SERVER_SSL_CERT_REQS="2" \
  python3 mcpgateway/plugins/framework/external/mcp/server/runtime.py &

# Client (config has no mcp.tls section)
PLUGINS_CLIENT_MTLS_CERTFILE="certs/mcp/gateway/client.crt" \
  PLUGINS_CLIENT_MTLS_KEYFILE="certs/mcp/gateway/client.key" \
  PLUGINS_CLIENT_MTLS_CA_BUNDLE="certs/mcp/gateway/ca.crt" \
  PLUGINS_CLIENT_MTLS_VERIFY="true" \
  PLUGINS_CLIENT_MTLS_CHECK_HOSTNAME="false" \
  python3 your_client.py

How mTLS Works

1. Certificate Authority (CA): A single root CA (ca.crt) signs both client and server certificates
2. Server Certificate: Plugin server presents its certificate (server.crt) to clients
3. Client Certificate: Gateway presents its certificate (client.crt) to the plugin server
4. Mutual Verification: Both parties verify each other's certificates against the CA bundle
5. Secure Channel: After mutual authentication, all communication is encrypted

Configuration Priority

Environment variables take precedence over YAML configuration:

• If PLUGINS_SERVER_SSL_ENABLED=true, env vars override server_settings.tls
• If client env vars are set, they override mcp.tls in YAML

Hostname Verification (check_hostname)

Overview

check_hostname is a client-side only setting that verifies the server's certificate matches the hostname/IP you're connecting to.

How It Works

The client checks if the URL hostname matches entries in the server certificate's:

• Common Name (CN): CN=mcp-plugin-ReplaceBadWordsPlugin
• Subject Alternative Names (SANs): DNS names or IP addresses

Checking Certificate SANs

# View DNS and IP SANs in server certificate
openssl x509 -in certs/mcp/plugins/ReplaceBadWordsPlugin/server.crt -text -noout | grep -A 5 "Subject Alternative Name"

# Example output:
# X509v3 Subject Alternative Name:
#     DNS:ReplaceBadWordsPlugin, DNS:mcp-plugin-ReplaceBadWordsPlugin, DNS:localhost

Configuration Examples

Option 1: Use localhost with check_hostname: true

# Client config
mcp:
  url: https://localhost:8000/mcp
  tls:
    check_hostname: true  # Works because "localhost" is in DNS SANs

Or with environment variables:

export PLUGINS_CLIENT_MTLS_CHECK_HOSTNAME="true"
# Connect to: https://localhost:8000/mcp

Option 2: Use IP address with check_hostname: false

# Client config
mcp:
  url: https://127.0.0.1:8000/mcp
  tls:
    check_hostname: false  # Required because 127.0.0.1 is not in SANs

Or with environment variables:

export PLUGINS_CLIENT_MTLS_CHECK_HOSTNAME="false"
# Connect to: https://127.0.0.1:8000/mcp

Option 3: Add IP SANs to certificate (Advanced)

If you need check_hostname: true with IP addresses, regenerate certificates with IP SANs:

# Modify Makefile to add IP SANs when generating certificates
# Add to server.ext or openssl command:
# subjectAltName = DNS:localhost, DNS:plugin-name, IP:127.0.0.1, IP:0.0.0.0

Server-Side Hostname Verification

There is no check_hostname setting on the server side. The server only:

1. Verifies the client certificate is signed by the trusted CA
2. Checks if ssl_cert_reqs=2 (CERT_REQUIRED) to enforce client certificates

Testing Hostname Verification

Test 1: Valid hostname (should succeed)

# Server bound to 0.0.0.0 (accepts all interfaces)
PLUGINS_SERVER_HOST="0.0.0.0" ...

# Client connecting to localhost with hostname check
export PLUGINS_CLIENT_MTLS_CHECK_HOSTNAME="true"
# URL: https://localhost:8000/mcp
# Result: ✅ Success (localhost is in DNS SANs)

Test 2: IP address with hostname check (should fail)

# Client connecting to IP with hostname check
export PLUGINS_CLIENT_MTLS_CHECK_HOSTNAME="true"
# URL: https://127.0.0.1:8000/mcp
# Result: ❌ Fails with "IP address mismatch, certificate is not valid for '127.0.0.1'"

Troubleshooting

Connection Refused

• Ensure server is running: lsof -i :8000
• Check server logs for startup errors
• Verify server is bound to correct interface (0.0.0.0 for all, 127.0.0.1 for localhost only)
• Note: When mTLS is enabled, a health check server also runs on port 9000 (port+1000)

Certificate Verification Failed

• Verify CA bundle matches on both sides: md5 certs/**/ca.crt
• Check certificate paths are correct
• Ensure certificates haven't expired: openssl x509 -in cert.crt -noout -dates

Hostname Verification Failed

Error: certificate verify failed: IP address mismatch or Hostname mismatch

Solutions:

1. Use hostname from SANs: Connect to https://localhost:8000 instead of https://127.0.0.1:8000
2. Disable hostname check: Set check_hostname: false or PLUGINS_CLIENT_MTLS_CHECK_HOSTNAME="false"
3. Add IP to SANs: Regenerate certificates with IP SANs included

mTLS Deployment Hardening Guidelines

For production deployments, follow these security best practices to ensure robust mTLS configuration:

Implementation Checklist

When deploying plugin mTLS in production:

1. Generate Certificates:

   • Local/Docker: Use make certs-mcp-all to create complete certificate infrastructure
   • Kubernetes: Deploy cert-manager and configure Certificate resources for automated issuance and renewal

2. Verify Expiration:

   • Local/Docker: Run make certs-mcp-check regularly to monitor certificate validity
   • Kubernetes: cert-manager automatically monitors and renews certificates before expiration

3. Secure Private Keys: Ensure all .key files have 600 permissions and are owned by service accounts (or stored in Kubernetes Secrets with appropriate RBAC)

4. Enable Hostname Verification: Set check_hostname: true or PLUGINS_CLIENT_MTLS_CHECK_HOSTNAME="true" unless using IP addresses

5. Configure Health Checks: Bind health servers to 127.0.0.1 to prevent external access

6. Enforce mTLS: Set PLUGINS_SERVER_SSL_CERT_REQS="2" to require client certificates

7. Monitor Logs: Enable LOG_LEVEL=DEBUG temporarily during initial deployment to verify handshakes

8. Plan Rotation:

   • Local/Docker: Schedule certificate rotation every 90-180 days using MCP_CERT_DAYS parameter
   • Kubernetes: Configure cert-manager Certificate resources with appropriate renewBefore duration (typically 30 days before expiration)

Security Validation

After deployment, verify your mTLS configuration:

# 1. Check certificate expiration dates
make certs-mcp-check

# 2. Verify file permissions on private keys
find certs/mcp -name "*.key" -exec ls -la {} \;

# 3. Test certificate verification
openssl verify -CAfile certs/mcp/ca/ca.crt certs/mcp/gateway/client.crt

# 4. Confirm TLS version enforcement
openssl s_client -connect localhost:8000 -tls1_1 < /dev/null
# Should fail with "no protocols available" or similar

# 5. Test hostname verification (should succeed)
curl --cert certs/mcp/gateway/client.pem \
     --cacert certs/mcp/gateway/ca.crt \
     https://localhost:8000/health

# 6. Test without client cert (should fail if ssl_cert_reqs=2)
curl --cacert certs/mcp/gateway/ca.crt \
     https://localhost:8000/health