Otel Demo Architecture

OpenTelemetry Demo Architecture

Quick reference for the OpenTelemetry Demo microservices system. Focus on service dependencies, critical paths, and common failure patterns.

For detailed observability queries: See the Observability Query Guides section below for comprehensive metrics, traces, and logs references.

Service Dependency Matrix

Critical Service Paths

User Request Flow:

Internet → frontend-proxy:ENVOY_PORT → frontend → [cart, product-catalog, recommendation, checkout]

Checkout/Order Flow:

checkout → cart (gRPC)
        → currency (gRPC)
        → payment (gRPC)
        → product-catalog (gRPC)
        → shipping (HTTP) → quote (HTTP)
        → email (HTTP)
        → kafka → [accounting, fraud-detection]

Telemetry Flow:

All services → otel-collector:4317(gRPC)/4318(HTTP) → [jaeger, prometheus, tempo, opensearch]
                                                     → grafana (visualization)

Observability Stack Details

Metrics Pipeline

Services → otel-collector (OTLP) → prometheus (scrape/remote-write) → grafana

Traces Pipeline

Services → otel-collector (OTLP) → [jaeger, tempo] → grafana

Logs Pipeline

Services → docker json-file → alloy → loki → grafana

Log Configuration

All services use json-file driver:

• Max size: 5M
• Max files: 2
• Auto-rotation

Observability Query Guides

Detailed reference guides for querying observability data:

Metrics Guide

See metrics-guide.md for:

• Complete Prometheus metrics catalog - All 260+ available metrics organized by category
• Service-specific metrics - Application metrics for each demo service
• HTTP/RPC metrics - Request rates, latencies, error rates
• Runtime metrics - Go, .NET, JVM, Node.js, Python runtime instrumentation
• Container metrics - CPU, memory, network, disk I/O
• Feature flag metrics - Flag evaluation and impression tracking
• OTEL Collector metrics - Pipeline health and throughput
• Common label patterns - Service identification, filtering, aggregation
• Query patterns - Request rates, error rates, percentiles, top-N

When to use: Building dashboards, investigating performance issues, analyzing resource utilization, monitoring service health.

Traces Guide

See traces-guide.md for:

• TraceQL syntax and patterns - Complete query language reference
• Resource attributes - Service, host, process, runtime identification (27 attributes)
• Span attributes - HTTP, gRPC, database, application-specific (150+ attributes)
• Event attributes - Exceptions, feature flags, business events
• Intrinsic attributes - Duration, status, kind, instrumentation
• Service-specific attributes - Cart, product, payment, shipping, ad service patterns
• Common query patterns - Error investigation, performance analysis, feature flag impact

When to use: Debugging distributed transactions, investigating latency issues, understanding service dependencies, tracing business transactions.

Logs Guide

See logs-guide.md for:

• LogQL syntax and patterns - Complete query language reference
• Available labels - service_name, container, project (5 labels)
• Service identification - All 16 application and infrastructure services
• Common query patterns - Errors, HTTP requests, performance, business events
• Log parsing - JSON parsing, pattern extraction, label filtering
• Aggregations - Count, rate, percentiles, error percentages
• Multi-service analysis - Cross-service errors, communication tracing

When to use: Investigating errors, debugging application logic, analyzing request patterns, root cause analysis.