Monitoring Observability

Monitoring and Observability

Production systems require visibility into their behavior. Observability goes beyond simple monitoring by enabling you to ask arbitrary questions about system state using logs, metrics, and traces. This guide covers instrumentation, collection, visualization, alerting, and the operational patterns that prevent alert fatigue while keeping systems reliable.

The Three Pillars

OpenTelemetry SDK Setup

OpenTelemetry provides a vendor-neutral API for emitting all three signals. Instrument once, export anywhere.

Node.js Auto-Instrumentation

// tracing.ts -- Load BEFORE application code
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { Resource } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'order-service',
    [ATTR_SERVICE_VERSION]: process.env.APP_VERSION || '0.0.0',
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter(),
    exportIntervalMillis: 15000,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false },
      '@opentelemetry/instrumentation-http': {
        ignoreIncomingPaths: ['/healthz', '/readyz', '/metrics'],
      },
    }),
  ],
});

sdk.start();
process.on('SIGTERM', () => sdk.shutdown().finally(() => process.exit(0)));

Manual Span Creation

import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('order-service', '1.0.0');

async function processOrder(orderId: string): Promise<Order> {
  return tracer.startActiveSpan('processOrder', async (span) => {
    try {
      span.setAttribute('order.id', orderId);

      const order = await tracer.startActiveSpan('db.fetchOrder', async (dbSpan) => {
        dbSpan.setAttribute('db.system', 'postgresql');
        const result = await db.orders.findById(orderId);
        dbSpan.end();
        return result;
      });

      span.addEvent('order.validated', { 'order.total': order.total });
      await chargePayment(order);
      span.setStatus({ code: SpanStatusCode.OK });
      return order;
    } catch (error) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
      span.recordException(error);
      throw error;
    } finally {
      span.end();
    }
  });
}

Prometheus Metrics

Prometheus uses a pull model -- it scrapes HTTP endpoints at intervals.

Express Application Metrics

import express from 'express';
import promClient from 'prom-client';

const app = express();
promClient.collectDefaultMetrics({ prefix: 'orderservice_' });

const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
});

const httpRequestsTotal = new promClient.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status_code'],
});

app.use((req, res, next) => {
  const end = httpRequestDuration.startTimer();
  res.on('finish', () => {
    const labels = { method: req.method, route: req.route?.path || req.path, status_code: res.statusCode.toString() };
    end(labels);
    httpRequestsTotal.inc(labels);
  });
  next();
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  res.end(await promClient.register.metrics());
});

Grafana Dashboard

Define dashboards as JSON for version control. Key panels for any service overview:

{
  "dashboard": {
    "title": "Order Service Overview",
    "refresh": "30s",
    "panels": [
      {
        "title": "Request Rate",
        "type": "timeseries",
        "targets": [{
          "expr": "sum(rate(http_requests_total{service=\"order-service\"}[5m])) by (status_code)",
          "legendFormat": "{{status_code}}"
        }],
        "fieldConfig": { "defaults": { "unit": "reqps" } }
      },
      {
        "title": "P99 Latency",
        "type": "timeseries",
        "targets": [{
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=\"order-service\"}[5m])) by (le, route))",
          "legendFormat": "{{route}}"
        }],
        "fieldConfig": { "defaults": { "unit": "s" } }
      },
      {
        "title": "Error Rate (SLO: 99.9%)",
        "type": "stat",
        "targets": [{
          "expr": "1 - (sum(rate(http_requests_total{service=\"order-service\", status_code=~\"5..\"}[1h])) / sum(rate(http_requests_total{service=\"order-service\"}[1h])))",
          "instant": true
        }],
        "fieldConfig": { "defaults": { "unit": "percentunit" } }
      }
    ]
  }
}

SLO-Based Alerting

Alert on Service Level Objectives, not raw thresholds. This ties alerting to user-visible impact.

Multi-Window Burn Rate Alerts

# alerts/slo-alerts.yml
groups:
  - name: slo-burn-rate
    rules:
      # Fast burn -- 2% of 30-day budget in 1 hour
      - alert: HighErrorBudgetBurnRate_Fast
        expr: |
          (
            sum(rate(http_requests_total{status_code=~"5..", service="order-service"}[1h]))
            / sum(rate(http_requests_total{service="order-service"}[1h]))
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_requests_total{status_code=~"5..", service="order-service"}[5m]))
            / sum(rate(http_requests_total{service="order-service"}[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Burning error budget 14.4x faster than sustainable"
          runbook: "https://wiki.internal/runbooks/order-service-high-errors"

      # Slow burn -- 10% of budget in 3 days
      - alert: HighErrorBudgetBurnRate_Slow
        expr: |
          (
            sum(rate(http_requests_total{status_code=~"5..", service="order-service"}[6h]))
            / sum(rate(http_requests_total{service="order-service"}[6h]))
          ) > (1.0 * 0.001)
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Slowly burning error budget"

      # Latency SLO
      - alert: LatencySLOBreach
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket{service="order-service"}[5m])) by (le)
          ) > 0.3
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "P99 latency exceeds 300ms SLO"

Log Aggregation Patterns

Structured Logging with Trace Correlation

import pino from 'pino';
import { context, trace } from '@opentelemetry/api';

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  mixin() {
    const span = trace.getSpan(context.active());
    if (span) {
      const { traceId, spanId } = span.spanContext();
      return { traceId, spanId };
    }
    return {};
  },
  redact: {
    paths: ['req.headers.authorization', 'password', 'ssn', 'creditCard'],
    censor: '[REDACTED]',
  },
});

// Good: structured with context
logger.info({ orderId, userId, total: order.total }, 'Order placed successfully');

// Bad: unstructured string interpolation -- not queryable, PII leaks
// logger.info(`Order ${orderId} placed by user ${userId} for $${total}`);

Loki Query Examples

# Find errors for a specific trace
{service="order-service"} |= "error" | json | traceId="abc123def456"

# Error rate by service
sum(rate({service=~".+"} |= "error" [5m])) by (service)

# Slow database queries
{service="order-service"} | json | db_duration_ms > 500

Distributed Tracing with Context Propagation

Client -> API Gateway -> Order Service -> Payment Service -> Database
  |           |              |                |
  trace_id=abc trace_id=abc  trace_id=abc     trace_id=abc
  span_id=001  span_id=002   span_id=003      span_id=004
               parent=001    parent=002       parent=003

import { context, propagation } from '@opentelemetry/api';

// Outgoing: inject trace context into headers
async function callPaymentService(order: Order): Promise<PaymentResult> {
  const headers: Record<string, string> = {};
  propagation.inject(context.active(), headers);

  return fetch('https://payment-service/charge', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', ...headers },
    body: JSON.stringify({ orderId: order.id, amount: order.total }),
  }).then(r => r.json());
}

// Incoming via message queue: extract manually
function processMessage(message: QueueMessage) {
  const parentContext = propagation.extract(context.active(), message.headers);
  context.with(parentContext, () => {
    tracer.startActiveSpan('processMessage', (span) => {
      // spans here are children of the original trace
      span.end();
    });
  });
}

APM Business Metrics

import { metrics } from '@opentelemetry/api';

const meter = metrics.getMeter('order-service', '1.0.0');

const orderCounter = meter.createCounter('orders_total', {
  description: 'Total orders processed',
});
const orderValue = meter.createHistogram('order_value_dollars', {
  description: 'Distribution of order values',
});

async function completeOrder(order: Order) {
  orderCounter.add(1, { payment_method: order.paymentMethod, region: order.region });
  orderValue.record(order.total, { currency: order.currency });
}

Alert Fatigue Prevention

Anti-Patterns

Observability Maturity Checklist

Level 1: Foundations

• Structured JSON logging with consistent field names across services
• Health check endpoints (/healthz, /readyz) on all services
• Prometheus metrics endpoint exposed on all services
• Default runtime metrics collected (memory, CPU, GC, event loop)
• Centralized log aggregation (ELK, Loki, or cloud-native)

Level 2: Instrumentation

• OpenTelemetry SDK integrated with auto-instrumentation
• Custom business metrics defined (orders/sec, revenue, conversion)
• Distributed tracing with context propagation across services
• Trace context injected into log entries (traceId, spanId)
• Grafana dashboards with RED metrics (Rate, Errors, Duration)

Level 3: Alerting

• SLOs defined for each critical user journey
• Multi-window burn rate alerts replacing threshold alerts
• Every alert has a linked runbook with remediation steps
• Alert routing configured by severity (page / ticket / dashboard)
• Alert fatigue review conducted monthly
• absent() alerts for metrics that stop reporting

Level 4: Advanced

• Tail-based trace sampling to capture errors and slow requests
• Exemplars linking metrics to specific traces
• Error budget tracking visible on team dashboards
• Anomaly detection on key business metrics
• Observability costs tracked and optimized