Internal Observability System

Internal Observability System

ContextForge includes a built-in observability system that provides comprehensive performance monitoring, error tracking, and analytics without requiring external observability platforms. All trace data is stored in your database (SQLite/PostgreSQL/MariaDB) and visualized through the Admin UI.

Overview

The internal observability system captures detailed performance metrics and traces for:

• Tools - Invocation frequency, performance metrics, error rates
• Prompts - Rendering frequency, latency percentiles, error tracking
• Resources - Fetch frequency, performance metrics, error tracking
• HTTP Requests - Complete request/response tracing with timing

Unlike OpenTelemetry (which sends traces to external systems like Phoenix or Jaeger), the internal observability system is self-contained, making it ideal for:

• Development and testing environments
• Organizations that prefer self-hosted solutions
• Scenarios where external observability platforms are not available
• Quick performance analysis without additional infrastructure

Key Features

Performance Analytics

• Latency Percentiles: p50, p90, p95, p99 metrics for detailed performance analysis
• Duration Tracking: Millisecond-precision timing for all operations
• Throughput Metrics: Request counts and rates over time
• Comparative Analysis: Side-by-side comparison of multiple resources

Error Tracking

• Error Rate Monitoring: Percentage of failed operations with health indicators
• Error-Prone Analysis: Identify resources with highest failure rates
• Status Code Tracking: HTTP response codes and error messages
• Root Cause Analysis: Detailed traces with full context

Interactive Dashboards

• Summary Cards: At-a-glance health status, most used, slowest, and most error-prone resources
• Performance Charts: Interactive visualizations using Chart.js
• Time-Based Filtering: Analyze performance over custom time ranges
• Auto-Refresh: Dashboards update every 60 seconds automatically

Trace Visualization

• Gantt Chart Timeline: Visual representation of span execution order and timing
• Flame Graphs: Hierarchical view of nested operations
• Trace Details: Complete trace metadata, attributes, and context
• Span Explorer: Drill down into individual operations

Quick Start

1. Enable Observability

Add to your .env file:

# Enable internal observability
OBSERVABILITY_ENABLED=true

# Automatically trace HTTP requests
OBSERVABILITY_TRACE_HTTP_REQUESTS=true

# Retention and limits
OBSERVABILITY_TRACE_RETENTION_DAYS=7
OBSERVABILITY_MAX_TRACES=100000

# Trace sampling (1.0 = 100%, 0.1 = 10%)
OBSERVABILITY_SAMPLE_RATE=1.0

# Include paths (JSON array of regex patterns)
OBSERVABILITY_INCLUDE_PATHS=["^/rpc/?$","^/sse$","^/message$","^/mcp(?:/|$)","^/servers/[^/]+/mcp/?$","^/servers/[^/]+/sse$","^/servers/[^/]+/message$","^/a2a(?:/|$)"]

# Exclude paths (JSON array of regex patterns, applied after include patterns)
OBSERVABILITY_EXCLUDE_PATHS=["/health","/healthz","/ready","/metrics","/static/.*"]

# Enable metrics and events
OBSERVABILITY_METRICS_ENABLED=true
OBSERVABILITY_EVENTS_ENABLED=true

2. Start ContextForge

# With environment variables
export OBSERVABILITY_ENABLED=true
mcpgateway

# Or start development server
make dev

3. Access Admin UI

Navigate to the Observability section in the Admin UI:

http://localhost:4444/admin/observability

Configuration Reference

Core Settings

Retention & Limits

Sampling & Filtering

Feature Flags

Admin UI Dashboards

Tools Dashboard

Path: /admin/observability/tools

Provides comprehensive analytics for MCP tool invocations:

Summary Cards

• Overall Health: Success rate with color-coded status

  • Green: <5% errors (healthy)
  • Yellow: 5-20% errors (degraded)
  • Red: >20% errors (unhealthy)

• Most Used: Top tool by invocation count

• Slowest: Tool with highest p99 latency

• Most Error-Prone: Tool with highest error rate

Performance Charts

1. Tool Usage Chart: Bar chart showing invocation counts
2. Average Latency Chart: Bar chart with millisecond precision
3. Error Rate Chart: Percentage visualization with color coding
4. Top N Error-Prone Tools: Focused view of problematic tools

Detailed Metrics Table

For each tool:

• Invocation Count: Total number of calls
• Latency Percentiles: p50, p90, p95, p99 in milliseconds
• Error Rate: Percentage with color-coded status
• Last Used: Timestamp of most recent invocation

Filtering Options

• Time Range: Last 1 hour, 24 hours, 7 days, 30 days
• Result Limit: Top 10, 20, 50, or 100 tools
• Auto-Refresh: 60-second automatic updates

Prompts Dashboard

Path: /admin/observability/prompts

Analyzes MCP prompt rendering performance:

Summary Cards

• Overall Health: Rendering success rate
• Most Used: Most frequently rendered prompt
• Slowest: Prompt with highest p99 latency
• Most Error-Prone: Prompt with highest failure rate

Performance Charts

1. Prompt Render Frequency: Usage distribution
2. Average Latency: Rendering performance
3. Error Rate: Failure rate analysis
4. Top N Error-Prone Prompts: Problem identification

Detailed Metrics

• Render Count: Total rendering operations
• Latency Percentiles: p50, p90, p95, p99 metrics
• Error Rate: Failure percentage with status
• Last Rendered: Most recent usage timestamp

Resources Dashboard

Path: /admin/observability/resources

Monitors MCP resource fetch operations:

Summary Cards

• Overall Health: Fetch success rate
• Most Used: Most accessed resource
• Slowest: Resource with highest p99 latency
• Most Error-Prone: Resource with highest error rate

Performance Charts

1. Resource Fetch Frequency: Access patterns
2. Average Latency: Fetch performance
3. Error Rate: Failure analysis
4. Top N Error-Prone Resources: Issue detection

Detailed Metrics

• Fetch Count: Total access operations
• Latency Percentiles: p50, p90, p95, p99 metrics
• Error Rate: Failure rate with health status
• Last Fetched: Recent access timestamp

Trace Visualization

Trace List

Path: /admin/observability/traces

Browse all captured traces with:

• Trace ID: Unique identifier
• Operation Name: Human-readable description
• Start Time: When the trace began
• Duration: Total execution time
• Status: Success/error indicator
• HTTP Details: Method, URL, status code

Trace Detail View

Path: /admin/observability/traces/{trace_id}

Comprehensive trace analysis:

Trace Metadata

• Trace ID: Unique identifier (W3C format)
• Name: Operation description
• Status: Overall outcome
• Duration: Total execution time
• HTTP Context: Method, URL, status code, user agent
• User Context: Email, IP address
• Timestamps: Start, end, created times

Gantt Chart Timeline

Visual representation showing:

• Span Execution Order: Chronological flow
• Nested Operations: Parent-child relationships
• Duration Bars: Relative timing visualization
• Overlap Detection: Concurrent operations

Flame Graph

Hierarchical view displaying:

• Call Stack: Nested span relationships
• Time Distribution: Width represents duration
• Critical Path: Longest execution chains
• Bottleneck Identification: Performance hotspots

Spans Table

Detailed span information:

• Span ID: Unique identifier
• Name: Operation description
• Kind: Span type (internal, server, client)
• Start/End Time: Execution window
• Duration: Millisecond precision
• Status: Success/error indicator
• Resource Info: Type, name, ID
• Attributes: Custom metadata

Performance Metrics

Latency Percentiles

The system calculates accurate percentiles using database aggregation:

• p50 (Median): 50% of requests complete faster
• p90: 90% of requests complete faster
• p95: 95% of requests complete faster
• p99: 99% of requests complete faster

These metrics help identify performance outliers and establish SLAs.

Health Status Indicators

Color-coded status based on error rates:

Green  (<5% errors)   - Healthy
Yellow (5-20% errors) - Degraded
Red    (>20% errors)  - Unhealthy

Metrics Calculation

All metrics are calculated dynamically based on your selected time range:

• Real-time aggregation from trace database
• No pre-computation or caching delays
• Accurate percentile calculations using SQLite/PostgreSQL functions
• Efficient indexing for fast queries

Data Retention

Automatic Cleanup

Traces older than OBSERVABILITY_TRACE_RETENTION_DAYS are automatically deleted:

# Retain traces for 7 days (default)
OBSERVABILITY_TRACE_RETENTION_DAYS=7

# Extend retention to 30 days
OBSERVABILITY_TRACE_RETENTION_DAYS=30

Size Limits

Prevent unbounded growth with OBSERVABILITY_MAX_TRACES:

# Store up to 100,000 traces (default)
OBSERVABILITY_MAX_TRACES=100000

# Increase for high-volume environments
OBSERVABILITY_MAX_TRACES=1000000

When the limit is reached, oldest traces are deleted first.

Manual Cleanup

Use the CLI for manual trace management:

# Delete traces older than 7 days
mcpgateway observability cleanup --days 7

# Delete specific trace by ID
mcpgateway observability delete-trace <trace_id>

# Clear all traces (use with caution!)
mcpgateway observability clear-all

Sampling Strategies

Full Sampling (Development)

Capture all requests for complete visibility:

OBSERVABILITY_SAMPLE_RATE=1.0  # 100% sampling

Partial Sampling (Production)

Reduce overhead while maintaining visibility:

# Sample 10% of requests
OBSERVABILITY_SAMPLE_RATE=0.1

# Sample 1% of requests (high volume)
OBSERVABILITY_SAMPLE_RATE=0.01

Path Exclusion

Tune which paths are traced:

# Limit tracing to MCP/A2A endpoints (default include list)
OBSERVABILITY_INCLUDE_PATHS=["^/rpc/?$","^/sse$","^/message$","^/mcp(?:/|$)","^/servers/[^/]+/mcp/?$","^/servers/[^/]+/sse$","^/servers/[^/]+/message$","^/a2a(?:/|$)"]

# Trace all endpoints (leave include list empty)
OBSERVABILITY_INCLUDE_PATHS=[]

# Default exclusions
OBSERVABILITY_EXCLUDE_PATHS=["/health","/healthz","/ready","/metrics","/static/.*"]

# Custom exclusions (regex patterns applied after includes)
OBSERVABILITY_EXCLUDE_PATHS=["/health.*","/metrics.*","/static/.*","/admin/assets/.*"]

Database Schema

ObservabilityTrace Table

Stores complete request traces:

CREATE TABLE observability_traces (
    trace_id VARCHAR(36) PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    start_time TIMESTAMP NOT NULL,
    end_time TIMESTAMP,
    duration_ms FLOAT,
    status VARCHAR(20) DEFAULT 'unset',
    status_message TEXT,
    http_method VARCHAR(10),
    http_url VARCHAR(767),
    http_status_code INTEGER,
    user_email VARCHAR(255),
    user_agent TEXT,
    ip_address VARCHAR(45),
    attributes JSON,
    resource_attributes JSON,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

ObservabilitySpan Table

Stores individual operations within traces:

CREATE TABLE observability_spans (
    span_id VARCHAR(36) PRIMARY KEY,
    trace_id VARCHAR(36) NOT NULL,
    parent_span_id VARCHAR(36),
    name VARCHAR(255) NOT NULL,
    kind VARCHAR(20) DEFAULT 'internal',
    start_time TIMESTAMP NOT NULL,
    end_time TIMESTAMP,
    duration_ms FLOAT,
    status VARCHAR(20) DEFAULT 'unset',
    status_message TEXT,
    attributes JSON,
    resource_name VARCHAR(255),
    resource_type VARCHAR(50),
    resource_id VARCHAR(36),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (trace_id) REFERENCES observability_traces(trace_id),
    FOREIGN KEY (parent_span_id) REFERENCES observability_spans(span_id)
);

Performance Indexes

Optimized for fast queries:

-- Trace indexes
CREATE INDEX idx_observability_traces_start_time ON observability_traces(start_time);
CREATE INDEX idx_observability_traces_user_email ON observability_traces(user_email);
CREATE INDEX idx_observability_traces_status ON observability_traces(status);
CREATE INDEX idx_observability_traces_http_status_code ON observability_traces(http_status_code);

-- Span indexes
CREATE INDEX idx_observability_spans_trace_id ON observability_spans(trace_id);
CREATE INDEX idx_observability_spans_parent_span_id ON observability_spans(parent_span_id);
CREATE INDEX idx_observability_spans_start_time ON observability_spans(start_time);
CREATE INDEX idx_observability_spans_resource_type ON observability_spans(resource_type);
CREATE INDEX idx_observability_spans_resource_name ON observability_spans(resource_name);

REST API

List Traces

GET /observability/traces

Query parameters:

• start_time: Filter traces after this timestamp (ISO 8601)
• end_time: Filter traces before this timestamp
• min_duration_ms: Minimum duration in milliseconds
• max_duration_ms: Maximum duration in milliseconds
• status: Filter by status (ok, error)
• http_status_code: Filter by HTTP status code
• http_method: Filter by HTTP method
• user_email: Filter by user email
• limit: Maximum results (default: 100)
• offset: Result offset (default: 0)

Example:

curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:4444/observability/traces?limit=10&status=error"

Get Trace Details

GET /observability/traces/{trace_id}

Returns complete trace with all spans, events, and metrics.

Example:

curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:4444/observability/traces/550e8400-e29b-41d4-a716-446655440000"

Query Tool Metrics

GET /observability/tools/metrics

Query parameters:

• time_range: Time window (1h, 24h, 7d, 30d)
• limit: Number of tools to return

Example:

curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:4444/observability/tools/metrics?time_range=24h&limit=20"

Query Prompt Metrics

GET /observability/prompts/metrics

Same parameters as tool metrics.

Query Resource Metrics

GET /observability/resources/metrics

Same parameters as tool metrics.

Comparison: Internal vs OpenTelemetry

When to Use Each

Use Internal Observability when:

• You want zero external dependencies
• Database storage is acceptable
• Admin UI visualization is sufficient
• Deployment simplicity is a priority
• You're in development/testing mode

Use OpenTelemetry when:

• You need distributed tracing across multiple services
• You want vendor-agnostic standard
• You have existing observability infrastructure
• You need advanced APM features
• You're in production with high scale

Use Both when:

• You want local debugging with external production monitoring
• You need different retention policies
• You want redundancy in observability data

Production Considerations

Performance Impact

The internal observability system is designed for minimal overhead:

• Database Writes: Async, batched when possible
• Indexing: Optimized indexes for fast queries
• Sampling: Reduce load with configurable sample rates
• Cleanup: Automatic retention management

Scaling Recommendations

For high-volume deployments:

# Reduce sampling rate
OBSERVABILITY_SAMPLE_RATE=0.1  # 10% sampling

# Aggressive retention
OBSERVABILITY_TRACE_RETENTION_DAYS=3

# Exclude high-frequency paths
OBSERVABILITY_EXCLUDE_PATHS=["/health.*","/metrics.*","/static/.*"]

# Disable HTTP request tracing (manual traces only)
OBSERVABILITY_TRACE_HTTP_REQUESTS=false

Database Considerations

SQLite

Suitable for:

• Development and testing
• Single-instance deployments
• Low to medium traffic

Limitations:

• Write concurrency limits
• File-based storage

PostgreSQL

Recommended for:

• Production deployments
• High-volume environments
• Multi-instance setups

Benefits:

• Superior write concurrency
• Advanced indexing
• Better query performance

MariaDB/MySQL

Alternative production option:

• Good write performance
• Wide deployment support
• Compatible with PostgreSQL features

Monitoring the Monitor

Track observability system health:

# Check trace count
SELECT COUNT(*) FROM observability_traces;

# Check database size
SELECT pg_size_pretty(pg_total_relation_size('observability_traces'));
SELECT pg_size_pretty(pg_total_relation_size('observability_spans'));

# Check oldest trace
SELECT MIN(start_time) FROM observability_traces;

# Check cleanup effectiveness
SELECT COUNT(*) FROM observability_traces
WHERE start_time < NOW() - INTERVAL '7 days';

Troubleshooting

No Traces Appearing

1. Verify observability is enabled:

   echo $OBSERVABILITY_ENABLED  # Should be "true"
   

2. Check sampling rate:

   echo $OBSERVABILITY_SAMPLE_RATE  # Should be > 0.0
   

3. Review included paths:

   echo $OBSERVABILITY_INCLUDE_PATHS
   # Ensure your test path is included
   

4. Review excluded paths:

   echo $OBSERVABILITY_EXCLUDE_PATHS
   # Ensure your test path is not excluded
   

5. Check database connection:

   # Verify database is accessible
   mcpgateway db-check
   

6. Enable debug logging:

   export LOG_LEVEL=DEBUG
   mcpgateway
   # Look for observability-related log messages
   

High Database Size

1. Reduce retention period:

   OBSERVABILITY_TRACE_RETENTION_DAYS=3
   

2. Lower maximum traces:

   OBSERVABILITY_MAX_TRACES=10000
   

3. Increase sampling threshold:

   OBSERVABILITY_SAMPLE_RATE=0.1
   

4. Manually cleanup:

   mcpgateway observability cleanup --days 1
   

Slow Dashboard Loading

1. Reduce query time range:

   • Use shorter time windows (1 hour instead of 30 days)

2. Limit result count:

   • Query top 10 instead of top 100

3. Add database indexes (if custom deployment):

   CREATE INDEX idx_custom ON observability_spans(resource_type, start_time);
   

4. Optimize database:

   # PostgreSQL
   VACUUM ANALYZE observability_traces;
   VACUUM ANALYZE observability_spans;
   
   # SQLite
   VACUUM;
   

Missing Spans or Metrics

1. Check span creation:

   • Verify tool/prompt/resource operations are completing
   • Look for errors in application logs

2. Verify metrics enabled:

   echo $OBSERVABILITY_METRICS_ENABLED  # Should be "true"
   

3. Check events enabled:

   echo $OBSERVABILITY_EVENTS_ENABLED  # Should be "true"
   

Best Practices

Development

# Full tracing, short retention
OBSERVABILITY_ENABLED=true
OBSERVABILITY_SAMPLE_RATE=1.0
OBSERVABILITY_TRACE_RETENTION_DAYS=1
OBSERVABILITY_MAX_TRACES=10000

Staging

# Partial tracing, moderate retention
OBSERVABILITY_ENABLED=true
OBSERVABILITY_SAMPLE_RATE=0.5
OBSERVABILITY_TRACE_RETENTION_DAYS=7
OBSERVABILITY_MAX_TRACES=100000

Production

# Sampled tracing, longer retention
OBSERVABILITY_ENABLED=true
OBSERVABILITY_SAMPLE_RATE=0.1
OBSERVABILITY_TRACE_RETENTION_DAYS=14
OBSERVABILITY_MAX_TRACES=1000000
OBSERVABILITY_EXCLUDE_PATHS=["/health.*","/metrics.*"]

Next Steps

• Review Configuration Reference for all observability settings
• Explore OpenTelemetry Integration for external monitoring
• Set up Phoenix Integration for AI-specific observability
• Configure Prometheus Metrics for time-series monitoring
• Implement Custom Dashboards based on your metrics

Related Documentation

• Configuration Reference - Environment variable configuration
• OpenTelemetry Observability - External tracing backends
• Phoenix Integration - AI/LLM observability
• Admin UI Documentation - Customizing the Admin UI
• Database Configuration - Database setup and tuning