# Logging Observability

> Production logging with SLF4J/Logback, structured JSON logs, MDC for request tracing, Micrometer metrics, and health indicators for Spring Boot applications.

- Skill: `gazolla/logging-observability` (Agent Skill)
- Install (CLI): `npx skillmds@latest add gazolla/logging-observability`
- Raw SKILL.md: https://api.skillmd.com/api/skills/gazolla/logging-observability/raw
- Safety review: pending (external: skill-scanner PASS, skillspector CAUTION)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: gazolla (https://skillmd.com/u/gazolla)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/gazolla/logging-observability

---


# Logging & Observability for Spring Boot

Production-grade logging, metrics, and health monitoring for Spring Boot 3.x applications.

## Use this skill when

- Setting up structured logging for a Spring Boot application
- Adding request tracing with correlation IDs
- Configuring JSON log output for production (ELK, Datadog, CloudWatch)
- Adding Micrometer metrics (counters, timers, gauges)
- Creating custom health indicators
- Configuring Spring Boot Actuator endpoints
- Implementing performance logging or method-level tracing
- Masking sensitive data in log output
- User mentions "logging", "metrics", "observability", "tracing", or "monitoring"

## Do not use this skill when

- User needs distributed tracing with Zipkin/Jaeger (use a dedicated tracing skill)
- User needs full APM solution (Datadog agent, New Relic)
- User needs log aggregation infrastructure setup (ELK stack deployment)
- User only needs basic `System.out.println` debugging (use java-fundamentals)

## Dependencies (pom.xml)

```xml
<dependencies>
    <!-- Logging (included via spring-boot-starter) -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>

    <!-- Structured JSON logging -->
    <dependency>
        <groupId>net.logstash.logback</groupId>
        <artifactId>logstash-logback-encoder</artifactId>
        <version>7.4</version>
    </dependency>

    <!-- Actuator for health/metrics endpoints -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

    <!-- Micrometer Prometheus registry (optional, for /prometheus endpoint) -->
    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>

    <!-- AOP for logging aspects -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-aop</artifactId>
    </dependency>
</dependencies>
```

## Logging Levels

| Level   | When to Use | Example |
|---------|-------------|---------|
| `ERROR` | System is broken, requires immediate attention. Failed operations that impact users. | Database connection lost, payment processing failed, unrecoverable state |
| `WARN`  | Unexpected situation, system can recover. Potential issues that may need investigation. | Retry succeeded after failure, deprecated API called, cache miss on expected hit |
| `INFO`  | Key business events and application lifecycle. What happened at a high level. | User registered, order placed, application started, scheduled job completed |
| `DEBUG` | Detailed flow for troubleshooting. Only enabled in development or when diagnosing issues. | Method entry/exit, query parameters, intermediate computation results |
| `TRACE` | Very fine-grained diagnostic info. Rarely enabled even in development. | Full request/response bodies, loop iterations, byte-level data |

## Logger Creation

```java
// Option 1: SLF4J direct (preferred — no extra dependency)
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class OrderService {
    private static final Logger log = LoggerFactory.getLogger(OrderService.class);
}

// Option 2: Lombok @Slf4j (if Lombok is in the project)
import lombok.extern.slf4j.Slf4j;

@Slf4j
public class OrderService {
    // log field is auto-generated
}
```

## Parameterized Logging

Always use parameterized messages. Never concatenate strings.

```java
// CORRECT: parameterized — no string concatenation cost if level is disabled
log.info("Order {} placed by user {} for amount {}", orderId, userId, amount);

// WRONG: string concatenation — always evaluated even if INFO is disabled
log.info("Order " + orderId + " placed by user " + userId);

// Logging exceptions — exception is ALWAYS the last argument, no placeholder
log.error("Failed to process order {}", orderId, exception);

// Guard expensive operations
if (log.isDebugEnabled()) {
    log.debug("Full order details: {}", order.toDetailedString());
}
```

## Structured Logging with JSON

Use `logstash-logback-encoder` to produce JSON logs consumable by ELK, Datadog, CloudWatch.

```java
import static net.logstash.logback.argument.StructuredArguments.*;

// Adds key-value pairs to the JSON log entry
log.info("Order placed", kv("orderId", orderId), kv("userId", userId), kv("amount", amount));

// Output: {"message":"Order placed","orderId":"ORD-123","userId":"USR-456","amount":99.99,...}

// keyValue() — includes in message AND JSON fields
log.info("Processing {}", keyValue("orderId", orderId));
// Output message: "Processing orderId=ORD-123" + JSON field "orderId":"ORD-123"

// value() — includes in message only, no JSON field
log.info("Processing order {}", value("orderId", orderId));
// Output message: "Processing order ORD-123"
```

## MDC (Mapped Diagnostic Context)

MDC attaches key-value pairs to the current thread. Every log statement on that thread automatically includes the MDC values.

```java
import org.slf4j.MDC;

// Set at request entry point (filter or interceptor)
MDC.put("correlationId", UUID.randomUUID().toString());
MDC.put("userId", authenticatedUser.getId());

try {
    // All log statements in this thread now include correlationId and userId
    log.info("Processing request");  // correlationId and userId are in the log output
    orderService.placeOrder(request);
} finally {
    MDC.clear();  // ALWAYS clear in a finally block to prevent thread pool leaks
}
```

### Correlation ID Pattern

Every HTTP request gets a unique correlation ID. Pass it through all service calls.

```java
// In a servlet filter (see examples/RequestLoggingFilter.java):
String correlationId = request.getHeader("X-Correlation-ID");
if (correlationId == null || correlationId.isBlank()) {
    correlationId = UUID.randomUUID().toString();
}
MDC.put("correlationId", correlationId);

// In Logback config, include it in every log line:
// Pattern: %d{ISO8601} [%X{correlationId}] %-5level %logger{36} - %msg%n
// JSON: automatically included by logstash-logback-encoder when in MDC
```

## Logback Configuration

Place `logback-spring.xml` in `src/main/resources/`. Spring Boot profiles control which appenders are active.

```xml
<!-- Dev profile: colorized console output -->
<springProfile name="dev">
    <root level="DEBUG">
        <appender-ref ref="CONSOLE" />
    </root>
</springProfile>

<!-- Prod profile: JSON to stdout + rolling file -->
<springProfile name="prod">
    <root level="INFO">
        <appender-ref ref="JSON_CONSOLE" />
        <appender-ref ref="ASYNC_FILE" />
    </root>
</springProfile>
```

See `examples/logback-spring.xml` for the full configuration.

## Micrometer Metrics

Micrometer is the metrics facade for Spring Boot (like SLF4J is for logging).

### Counters — track totals

```java
private final Counter orderCounter;

public OrderService(MeterRegistry registry) {
    this.orderCounter = Counter.builder("orders.placed.total")
        .description("Total orders placed")
        .tag("type", "standard")
        .register(registry);
}

public void placeOrder(Order order) {
    orderCounter.increment();
}
```

### Timers — track duration

```java
private final Timer orderTimer;

public OrderService(MeterRegistry registry) {
    this.orderTimer = Timer.builder("orders.processing.duration")
        .description("Order processing time")
        .publishPercentiles(0.5, 0.95, 0.99)
        .register(registry);
}

public void placeOrder(Order order) {
    orderTimer.record(() -> {
        // processing logic
    });
}
```

### Gauges — track current value

```java
private final AtomicInteger activeOrders = new AtomicInteger(0);

public OrderService(MeterRegistry registry) {
    Gauge.builder("orders.active.count", activeOrders, AtomicInteger::get)
        .description("Currently active orders")
        .register(registry);
}
```

## Custom Health Indicators

```java
@Component
public class DatabaseHealthIndicator extends AbstractHealthIndicator {

    @Override
    protected void doHealthCheck(Health.Builder builder) {
        // Check external dependency
        if (isDatabaseReachable()) {
            builder.up()
                .withDetail("database", "PostgreSQL")
                .withDetail("responseTime", "12ms");
        } else {
            builder.down()
                .withDetail("error", "Cannot connect to database");
        }
    }
}
```

Shows at `GET /actuator/health`:
```json
{
  "status": "UP",
  "components": {
    "database": {
      "status": "UP",
      "details": { "database": "PostgreSQL", "responseTime": "12ms" }
    }
  }
}
```

## Spring Boot Actuator Endpoints

```yaml
management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus,loggers
  endpoint:
    health:
      show-details: when_authorized
```

| Endpoint | Purpose |
|----------|---------|
| `/actuator/health` | Application health status |
| `/actuator/metrics` | List all metric names |
| `/actuator/metrics/{name}` | Specific metric value |
| `/actuator/prometheus` | Prometheus scrape endpoint |
| `/actuator/loggers` | View and change log levels at runtime |
| `/actuator/info` | Application information |

### Change log level at runtime

```bash
# View current level
curl http://localhost:8080/actuator/loggers/com.example.app

# Change to DEBUG (no restart needed)
curl -X POST http://localhost:8080/actuator/loggers/com.example.app \
  -H 'Content-Type: application/json' \
  -d '{"configuredLevel": "DEBUG"}'
```

## Performance Logging Patterns

```java
// Timed annotation (Micrometer) — auto-creates timer metric
@Timed(value = "service.orders.find", description = "Time to find orders")
public List<Order> findOrders(SearchCriteria criteria) {
    // method body
}

// Manual stopwatch for complex flows
long start = System.nanoTime();
try {
    // operation
} finally {
    long durationMs = (System.nanoTime() - start) / 1_000_000;
    log.info("Operation completed in {}ms", durationMs);
}
```

## Sensitive Data Masking

Never log passwords, tokens, credit card numbers, SSNs, or PII.

```java
// WRONG: logs the full credit card number
log.info("Payment with card {}", cardNumber);

// CORRECT: mask sensitive data
log.info("Payment with card {}", maskCard(cardNumber));

public static String maskCard(String card) {
    if (card == null || card.length() < 4) return "****";
    return "****-****-****-" + card.substring(card.length() - 4);
}

// WRONG: logs the entire request body which may contain passwords
log.debug("Request body: {}", requestBody);

// CORRECT: log only safe fields
log.debug("Login attempt for user {}", request.getUsername());
```

### Logback Masking Pattern (logback-spring.xml)

```xml
<!-- Replace patterns matching sensitive data in log messages -->
<conversionRule conversionWord="maskedMsg"
    converterClass="com.example.app.logging.SensitiveDataMaskingConverter" />
```

## Code Quality Checklist

- [ ] Loggers use `LoggerFactory.getLogger(ClassName.class)` or `@Slf4j`
- [ ] All log messages use parameterized format (no string concatenation)
- [ ] Exceptions logged with `log.error("message", exception)` (exception as last arg)
- [ ] MDC cleared in `finally` blocks to prevent thread pool leaks
- [ ] Correlation ID propagated through all service layers
- [ ] `logback-spring.xml` uses Spring profiles (dev: console, prod: JSON)
- [ ] No sensitive data (passwords, tokens, PII) in log messages
- [ ] Business metrics tracked with Micrometer counters/timers
- [ ] Custom health indicators for all external dependencies
- [ ] Actuator endpoints secured in production
- [ ] `DEBUG`/`TRACE` guarded with `log.isDebugEnabled()` for expensive operations
- [ ] Rolling file policy configured to prevent disk space exhaustion
- [ ] Async appender used in production for high-throughput logging

## References

- See `references/logging-levels-guide.md` for detailed level guidance
- See `references/actuator-endpoints.md` for full Actuator reference
- See `examples/` for complete code examples