# Spring AI Observability

> A complete guide for observability in Spring AI applications, covering monitoring, metrics, tracing, logging, and debugging.

- Skill: `mat-garcia/spring-ai-observability` (Agent Skill)
- Install (CLI): `npx skillmds@latest add mat-garcia/spring-ai-observability`
- Raw SKILL.md: https://api.skillmd.com/api/skills/mat-garcia/spring-ai-observability/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: Complete terms in LICENSE.txt
- Author: mat-garcia (https://skillmd.com/u/mat-garcia)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/mat-garcia/spring-ai-observability

---


# Spring AI - Observability, Monitoring & Observability

## Description

Complete guide for observability in Spring AI applications. Covers monitoring, metrics, tracing, logging, and debugging techniques for production AI systems.

## When to Use

- Monitoring LLM performance
- Tracking token usage and costs
- Distributed tracing
- Error analysis
- Performance debugging
- Usage analytics
- SLA monitoring
- Audit logging

## Topics Covered

### 1. Metrics & Monitoring

- **Request metrics**: Count, latency, errors
- **Token metrics**: Input/output tokens, total
- **Cost tracking**: Per-request costs
- **Model metrics**: Model selection, latency
- **Throughput**: Requests per second
- **Error rates**: By provider, model, type

### 2. Distributed Tracing

- **OpenTelemetry integration**: Standard tracing
- **Trace propagation**: Context across services
- **Span creation**: Request lifecycle
- **Baggage**: Propagating context
- **Exporters**: Send to observability platforms

### 3. Logging

- **Request logging**: Input/output content (carefully!)
- **Error logging**: Exception tracking
- **Performance logging**: Latency, tokens
- **Debug logging**: Detailed execution flow
- **Structured logging**: JSON format
- **Log levels**: Debug, info, warn, error

### 4. Chat Memory Tracking

- **Conversation logging**: Message history
- **Usage tracking**: How many messages
- **Memory statistics**: Size, tokens
- **Performance impact**: Memory overhead

### 5. Vector Store Observability

- **Query metrics**: Search latency
- **Result quality**: Retrieved document count
- **Storage metrics**: Size, document count
- **Similarity distribution**: Score analysis
- **Retrieval performance**: Top-K latency

### 6. Tool/Function Execution

- **Tool call metrics**: Count per tool
- **Execution time**: Per tool latency
- **Success rate**: Tool call success
- **Error tracking**: Tool errors
- **Tool usage patterns**: Analytics

### 7. Cost Analysis

- **Per-model costs**: Track by model
- **Per-request costs**: Granular tracking
- **Batch costs**: Bulk operation costs
- **Cost anomalies**: Alert on spikes
- **Budget tracking**: Spending trends

## Code Patterns

### Basic Metrics with Micrometer

```java
@Service
public class MetricsService {
    private final MeterRegistry meterRegistry;
    private final Counter requestCounter;
    private final Timer requestTimer;

    @Autowired
    public MetricsService(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
        this.requestCounter = Counter.builder("ai.request.count")
            .description("Total AI requests")
            .register(meterRegistry);
        this.requestTimer = Timer.builder("ai.request.duration")
            .description("AI request duration")
            .register(meterRegistry);
    }

    public String trackChat(String message) {
        return requestTimer.recordCallable(() -> {
            requestCounter.increment();
            return performChat(message);
        });
    }
}
```

### Token Usage Tracking

```java
@Service
public class TokenTrackingService {
    private final MeterRegistry meterRegistry;
    private final ChatClient chatClient;

    @Autowired
    public TokenTrackingService(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
    }

    public String chatWithTracking(String message) {
        ChatResponse response = chatClient.prompt()
            .user(message)
            .call()
            .chatResponse();

        UsageInfo usage = response.getMetadata().getUsageInformation();

        meterRegistry.counter("ai.tokens.input",
            "model", response.getMetadata().getModel())
            .increment(usage.getPromptTokens());

        meterRegistry.counter("ai.tokens.output",
            "model", response.getMetadata().getModel())
            .increment(usage.getGenerationTokens());

        return response.getResult().getOutput().getContent();
    }
}
```

### Cost Calculation

```java
@Service
public class CostCalculationService {
    private final MeterRegistry meterRegistry;

    // Cost per 1000 tokens (example rates)
    private static final Map<String, Double> INPUT_COST = Map.of(
        "gpt-4o", 0.005,
        "gpt-4", 0.03,
        "gpt-3.5-turbo", 0.0005
    );

    private static final Map<String, Double> OUTPUT_COST = Map.of(
        "gpt-4o", 0.015,
        "gpt-4", 0.06,
        "gpt-3.5-turbo", 0.0015
    );

    public double calculateRequestCost(String model, int inputTokens, int outputTokens) {
        double inputCost = (inputTokens / 1000.0) * INPUT_COST.getOrDefault(model, 0.0);
        double outputCost = (outputTokens / 1000.0) * OUTPUT_COST.getOrDefault(model, 0.0);
        double total = inputCost + outputCost;

        meterRegistry.counter("ai.cost.total",
            "model", model)
            .increment(total);

        return total;
    }
}
```

### OpenTelemetry Tracing

```java
@Configuration
public class TracingConfig {
    @Bean
    public OpenTelemetry openTelemetry() {
        return GlobalOpenTelemetry.get();
    }

    @Bean
    public Tracer tracer(OpenTelemetry openTelemetry) {
        return openTelemetry.getTracer("spring-ai");
    }
}

@Service
public class TracedChatService {
    private final Tracer tracer;
    private final ChatClient chatClient;

    @Autowired
    public TracedChatService(Tracer tracer) {
        this.tracer = tracer;
    }

    public String chat(String message) {
        Span span = tracer.spanBuilder("ai.chat.request")
            .setAttribute("message.length", message.length())
            .startSpan();

        try (Scope scope = span.makeCurrent()) {
            String response = chatClient.prompt()
                .user(message)
                .call()
                .content();

            span.setAttribute("response.length", response.length());
            return response;
        } finally {
            span.end();
        }
    }
}
```

### Structured Logging

```java
@Service
public class StructuredLoggingService {
    private static final Logger logger = LoggerFactory.getLogger(
        StructuredLoggingService.class
    );

    public String chatWithLogging(String userId, String message) {
        MDC.put("user_id", userId);
        MDC.put("message_id", UUID.randomUUID().toString());

        long startTime = System.currentTimeMillis();

        try {
            String response = performChat(message);

            long duration = System.currentTimeMillis() - startTime;

            logger.info("Chat request completed",
                "duration_ms", duration,
                "status", "success",
                "message_length", message.length(),
                "response_length", response.length()
            );

            return response;
        } catch (Exception e) {
            logger.error("Chat request failed",
                "error", e.getMessage(),
                "error_type", e.getClass().getSimpleName()
            );
            throw e;
        } finally {
            MDC.clear();
        }
    }
}
```

### RAG Metrics

```java
@Service
public class RagMetricsService {
    private final MeterRegistry meterRegistry;
    private final VectorStore vectorStore;

    public List<Document> searchWithMetrics(String query, int k) {
        Timer.Sample sample = Timer.start();

        List<Document> results = vectorStore.similaritySearch(query, k);

        sample.stop(Timer.builder("rag.search.latency")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(meterRegistry));

        meterRegistry.gauge("rag.results.count",
            results.size());

        // Similarity distribution
        results.forEach(doc -> {
            double similarity = extractSimilarity(doc);
            meterRegistry.recordDouble("rag.similarity.score",
                similarity);
        });

        return results;
    }
}
```

### Tool Execution Monitoring

```java
@Service
public class ToolMonitoringService {
    private final MeterRegistry meterRegistry;

    public <T> T executeToolWithMonitoring(
            String toolName,
            Callable<T> toolFn) {

        Timer.Sample sample = Timer.start();

        try {
            T result = toolFn.call();

            sample.stop(Timer.builder("tool.execution.time")
                .tag("tool", toolName)
                .tag("status", "success")
                .register(meterRegistry));

            meterRegistry.counter("tool.execution.success",
                "tool", toolName).increment();

            return result;
        } catch (Exception e) {
            sample.stop(Timer.builder("tool.execution.time")
                .tag("tool", toolName)
                .tag("status", "error")
                .register(meterRegistry));

            meterRegistry.counter("tool.execution.error",
                "tool", toolName,
                "error_type", e.getClass().getSimpleName())
                .increment();

            throw new RuntimeException(e);
        }
    }
}
```

### Conversation Analytics

```java
@Service
public class ConversationAnalyticsService {
    private final MeterRegistry meterRegistry;
    private final ChatMemoryStore memoryStore;

    public void analyzeConversation(String conversationId) {
        List<Message> messages = memoryStore.get(conversationId);

        // Track message count
        meterRegistry.gauge("conversation.message.count",
            messages.size());

        // Track token count
        long totalTokens = messages.stream()
            .mapToLong(msg -> countTokens(msg.getContent()))
            .sum();

        meterRegistry.gauge("conversation.token.count",
            totalTokens);

        // Track turn count
        int turns = messages.size() / 2;
        meterRegistry.gauge("conversation.turn.count",
            turns);
    }
}
```

## Configuration

### Micrometer Setup

```java
@Configuration
public class MetricsConfiguration {
    @Bean
    public MeterRegistryCustomizer meterRegistryCustomizer() {
        return registry -> {
            registry.config().commonTags(
                "application", "spring-ai",
                "environment", "production"
            );
        };
    }
}
```

### OpenTelemetry Exporter

```yaml
otel:
  exporter:
    otlp:
      endpoint: http://localhost:4317
  resource:
    attributes:
      service.name: spring-ai-app
      service.version: 1.0.0
```

### Logging Configuration

```properties
# Spring AI logging
logging.level.org.springframework.ai=DEBUG
logging.level.org.springframework.ai.chat=INFO

# Structured logging with SLF4J
logging.pattern.console=%d{ISO8601} - %msg%n
logging.pattern.file=%d{ISO8601} [%thread] %-5level %logger{36} - %msg%n
```

## Dashboard Examples

### Grafana Dashboard Queries

```
# Request rate
rate(ai_request_count[5m])

# Average latency
histogram_quantile(0.95, ai_request_duration_seconds)

# Token usage
increase(ai_tokens_total[1h])

# Error rate
rate(ai_request_errors_total[5m])
```

## Best Practices

- Monitor token usage for cost control
- Track model performance variations
- Set up alerts for anomalies
- Sample logging (avoid full content)
- Implement retention policies
- Monitor RAG quality metrics
- Track tool success rates
- Use structured logging
- Export traces for analysis
- Secure sensitive data

## Related Skills

- `chat-models/SKILL.md` - Chat monitoring
- `rag-retrieval/SKILL.md` - RAG metrics
- `tools-agents/SKILL.md` - Tool monitoring

## References

- Micrometer: https://micrometer.io
- OpenTelemetry: https://opentelemetry.io
- Spring documentation: `/pages/api/observability.adoc`