Observability Patterns

Observability Patterns for Spring Boot

Structured Logging

Logback JSON (Production)

<springProfile name="!local">
  <appender name="JSON_CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      <customFields>{"service":"order-service","env":"${SPRING_PROFILES_ACTIVE}"}</customFields>
    </encoder>
  </appender>
</springProfile>

Log Levels

Rules

• SLF4J placeholders: log.info("Order created orderId={}", order.id()) — never concatenation.
• MDC for context: MDC.put("orderId", id). Clear in finally.
• Never log PII. Log identifiers only.

Reactive Context Propagation

public Mono<Order> createReactive(CreateOrderCommand cmd) {
    return Mono.deferContextual(ctx -> {
        MDC.put("traceId", ctx.getOrDefault("traceId", "none"));
        MDC.put("orderId", cmd.orderId());
        return orderRepository.save(Order.from(cmd))
            .doOnNext(o -> log.info("Order created orderId={}", o.id()))
            .doOnError(e -> log.error("Order creation failed", e))
            .doFinally(sig -> MDC.clear());
    });
}

Distributed Tracing

management:
  tracing:
    sampling:
      probability: 1.0  # 100% dev; 0.1 (10%) prod
  otlp:
    tracing:
      endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4318}/v1/traces

Custom spans: tracer.nextSpan().name("order.enrich").tag(...). Use WebClient.builder().observationRegistry(registry) for automatic trace propagation.

Custom Metrics

• Counter: Counter.builder("orders.created.total").register(meterRegistry) — totals
• Timer: Timer.builder("orders.creation.duration").register(meterRegistry).record(() -> work()) — latency
• Gauge: Gauge.builder("orders.pending.count", this, s -> s.count()).register(meterRegistry) — current value
• @Timed on controllers, @Observed on business methods

Prometheus Config

management:
  endpoints.web.exposure.include: health,metrics,prometheus
  endpoint.health.probes.enabled: true
  metrics.distribution:
    percentiles-histogram:
      http.server.requests: true
    slo:
      http.server.requests: 50ms,200ms,500ms,1s

Health Indicators & Kubernetes Probes

@Component
public class ExternalServiceHealthIndicator implements HealthIndicator {
    public Health health() {
        boolean ok = client.ping();
        return ok ? Health.up().build() : Health.down().withDetail("reason", "ping failed").build();
    }
}

# Kubernetes probes
livenessProbe:
  httpGet: { path: /actuator/health/liveness, port: 8080 }
  initialDelaySeconds: 30
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 8080 }
  initialDelaySeconds: 10
  failureThreshold: 3

Liveness = app alive (restart if fails). Readiness = can accept traffic (db, redis).

Alerting Thresholds

Rules

• Never log PII — mask emails, never log passwords/tokens/credentials
• SLF4J placeholders (log.info("x={}", x)) — never string concatenation
• Clear MDC in reactive doFinally blocks — prevent context leakage
• Set timeouts on health check external calls (2-5s) — prevent probe hangs

References

• references/logging.md — Full Logback config (local + JSON), reactive MDC propagation, WebFilter correlation, sensitive data rules
• references/tracing-metrics.md — Dependencies, custom spans, Observation API, Micrometer auto-instrumentation, Prometheus alerts, reactive health indicators
• references/detailed-patterns.md — Legacy combined reference (all patterns in one file)

Related Skills

• spring-webflux-patterns — WebFilter setup, production actuator defaults
• pentest — PII logging detection (OWASP A09)
• testing-workflow — Verification pipeline includes observability checks