Spring AI Observability

Spring AI - Observability, Monitoring & Observability

Description

Complete guide for observability in Spring AI applications. Covers monitoring, metrics, tracing, logging, and debugging techniques for production AI systems.

When to Use

• Monitoring LLM performance
• Tracking token usage and costs
• Distributed tracing
• Error analysis
• Performance debugging
• Usage analytics
• SLA monitoring
• Audit logging

Topics Covered

1. Metrics & Monitoring

• Request metrics: Count, latency, errors
• Token metrics: Input/output tokens, total
• Cost tracking: Per-request costs
• Model metrics: Model selection, latency
• Throughput: Requests per second
• Error rates: By provider, model, type

2. Distributed Tracing

• OpenTelemetry integration: Standard tracing
• Trace propagation: Context across services
• Span creation: Request lifecycle
• Baggage: Propagating context
• Exporters: Send to observability platforms

3. Logging

• Request logging: Input/output content (carefully!)
• Error logging: Exception tracking
• Performance logging: Latency, tokens
• Debug logging: Detailed execution flow
• Structured logging: JSON format
• Log levels: Debug, info, warn, error

4. Chat Memory Tracking

• Conversation logging: Message history
• Usage tracking: How many messages
• Memory statistics: Size, tokens
• Performance impact: Memory overhead

5. Vector Store Observability

• Query metrics: Search latency
• Result quality: Retrieved document count
• Storage metrics: Size, document count
• Similarity distribution: Score analysis
• Retrieval performance: Top-K latency

6. Tool/Function Execution

• Tool call metrics: Count per tool
• Execution time: Per tool latency
• Success rate: Tool call success
• Error tracking: Tool errors
• Tool usage patterns: Analytics

7. Cost Analysis

• Per-model costs: Track by model
• Per-request costs: Granular tracking
• Batch costs: Bulk operation costs
• Cost anomalies: Alert on spikes
• Budget tracking: Spending trends

Code Patterns

Basic Metrics with Micrometer

@Service
public class MetricsService {
    private final MeterRegistry meterRegistry;
    private final Counter requestCounter;
    private final Timer requestTimer;

    @Autowired
    public MetricsService(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
        this.requestCounter = Counter.builder("ai.request.count")
            .description("Total AI requests")
            .register(meterRegistry);
        this.requestTimer = Timer.builder("ai.request.duration")
            .description("AI request duration")
            .register(meterRegistry);
    }

    public String trackChat(String message) {
        return requestTimer.recordCallable(() -> {
            requestCounter.increment();
            return performChat(message);
        });
    }
}

Token Usage Tracking

@Service
public class TokenTrackingService {
    private final MeterRegistry meterRegistry;
    private final ChatClient chatClient;

    @Autowired
    public TokenTrackingService(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
    }

    public String chatWithTracking(String message) {
        ChatResponse response = chatClient.prompt()
            .user(message)
            .call()
            .chatResponse();

        UsageInfo usage = response.getMetadata().getUsageInformation();

        meterRegistry.counter("ai.tokens.input",
            "model", response.getMetadata().getModel())
            .increment(usage.getPromptTokens());

        meterRegistry.counter("ai.tokens.output",
            "model", response.getMetadata().getModel())
            .increment(usage.getGenerationTokens());

        return response.getResult().getOutput().getContent();
    }
}

Cost Calculation

@Service
public class CostCalculationService {
    private final MeterRegistry meterRegistry;

    // Cost per 1000 tokens (example rates)
    private static final Map<String, Double> INPUT_COST = Map.of(
        "gpt-4o", 0.005,
        "gpt-4", 0.03,
        "gpt-3.5-turbo", 0.0005
    );

    private static final Map<String, Double> OUTPUT_COST = Map.of(
        "gpt-4o", 0.015,
        "gpt-4", 0.06,
        "gpt-3.5-turbo", 0.0015
    );

    public double calculateRequestCost(String model, int inputTokens, int outputTokens) {
        double inputCost = (inputTokens / 1000.0) * INPUT_COST.getOrDefault(model, 0.0);
        double outputCost = (outputTokens / 1000.0) * OUTPUT_COST.getOrDefault(model, 0.0);
        double total = inputCost + outputCost;

        meterRegistry.counter("ai.cost.total",
            "model", model)
            .increment(total);

        return total;
    }
}

OpenTelemetry Tracing

@Configuration
public class TracingConfig {
    @Bean
    public OpenTelemetry openTelemetry() {
        return GlobalOpenTelemetry.get();
    }

    @Bean
    public Tracer tracer(OpenTelemetry openTelemetry) {
        return openTelemetry.getTracer("spring-ai");
    }
}

@Service
public class TracedChatService {
    private final Tracer tracer;
    private final ChatClient chatClient;

    @Autowired
    public TracedChatService(Tracer tracer) {
        this.tracer = tracer;
    }

    public String chat(String message) {
        Span span = tracer.spanBuilder("ai.chat.request")
            .setAttribute("message.length", message.length())
            .startSpan();

        try (Scope scope = span.makeCurrent()) {
            String response = chatClient.prompt()
                .user(message)
                .call()
                .content();

            span.setAttribute("response.length", response.length());
            return response;
        } finally {
            span.end();
        }
    }
}

Structured Logging

@Service
public class StructuredLoggingService {
    private static final Logger logger = LoggerFactory.getLogger(
        StructuredLoggingService.class
    );

    public String chatWithLogging(String userId, String message) {
        MDC.put("user_id", userId);
        MDC.put("message_id", UUID.randomUUID().toString());

        long startTime = System.currentTimeMillis();

        try {
            String response = performChat(message);

            long duration = System.currentTimeMillis() - startTime;

            logger.info("Chat request completed",
                "duration_ms", duration,
                "status", "success",
                "message_length", message.length(),
                "response_length", response.length()
            );

            return response;
        } catch (Exception e) {
            logger.error("Chat request failed",
                "error", e.getMessage(),
                "error_type", e.getClass().getSimpleName()
            );
            throw e;
        } finally {
            MDC.clear();
        }
    }
}

RAG Metrics

@Service
public class RagMetricsService {
    private final MeterRegistry meterRegistry;
    private final VectorStore vectorStore;

    public List<Document> searchWithMetrics(String query, int k) {
        Timer.Sample sample = Timer.start();

        List<Document> results = vectorStore.similaritySearch(query, k);

        sample.stop(Timer.builder("rag.search.latency")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(meterRegistry));

        meterRegistry.gauge("rag.results.count",
            results.size());

        // Similarity distribution
        results.forEach(doc -> {
            double similarity = extractSimilarity(doc);
            meterRegistry.recordDouble("rag.similarity.score",
                similarity);
        });

        return results;
    }
}

Tool Execution Monitoring

@Service
public class ToolMonitoringService {
    private final MeterRegistry meterRegistry;

    public <T> T executeToolWithMonitoring(
            String toolName,
            Callable<T> toolFn) {

        Timer.Sample sample = Timer.start();

        try {
            T result = toolFn.call();

            sample.stop(Timer.builder("tool.execution.time")
                .tag("tool", toolName)
                .tag("status", "success")
                .register(meterRegistry));

            meterRegistry.counter("tool.execution.success",
                "tool", toolName).increment();

            return result;
        } catch (Exception e) {
            sample.stop(Timer.builder("tool.execution.time")
                .tag("tool", toolName)
                .tag("status", "error")
                .register(meterRegistry));

            meterRegistry.counter("tool.execution.error",
                "tool", toolName,
                "error_type", e.getClass().getSimpleName())
                .increment();

            throw new RuntimeException(e);
        }
    }
}

Conversation Analytics

@Service
public class ConversationAnalyticsService {
    private final MeterRegistry meterRegistry;
    private final ChatMemoryStore memoryStore;

    public void analyzeConversation(String conversationId) {
        List<Message> messages = memoryStore.get(conversationId);

        // Track message count
        meterRegistry.gauge("conversation.message.count",
            messages.size());

        // Track token count
        long totalTokens = messages.stream()
            .mapToLong(msg -> countTokens(msg.getContent()))
            .sum();

        meterRegistry.gauge("conversation.token.count",
            totalTokens);

        // Track turn count
        int turns = messages.size() / 2;
        meterRegistry.gauge("conversation.turn.count",
            turns);
    }
}

Configuration

Micrometer Setup

@Configuration
public class MetricsConfiguration {
    @Bean
    public MeterRegistryCustomizer meterRegistryCustomizer() {
        return registry -> {
            registry.config().commonTags(
                "application", "spring-ai",
                "environment", "production"
            );
        };
    }
}

OpenTelemetry Exporter

otel:
  exporter:
    otlp:
      endpoint: http://localhost:4317
  resource:
    attributes:
      service.name: spring-ai-app
      service.version: 1.0.0

Logging Configuration

# Spring AI logging
logging.level.org.springframework.ai=DEBUG
logging.level.org.springframework.ai.chat=INFO

# Structured logging with SLF4J
logging.pattern.console=%d{ISO8601} - %msg%n
logging.pattern.file=%d{ISO8601} [%thread] %-5level %logger{36} - %msg%n

Dashboard Examples

Grafana Dashboard Queries

# Request rate
rate(ai_request_count[5m])

# Average latency
histogram_quantile(0.95, ai_request_duration_seconds)

# Token usage
increase(ai_tokens_total[1h])

# Error rate
rate(ai_request_errors_total[5m])

Best Practices

• Monitor token usage for cost control
• Track model performance variations
• Set up alerts for anomalies
• Sample logging (avoid full content)
• Implement retention policies
• Monitor RAG quality metrics
• Track tool success rates
• Use structured logging
• Export traces for analysis
• Secure sensitive data

Related Skills

• chat-models/SKILL.md - Chat monitoring
• rag-retrieval/SKILL.md - RAG metrics
• tools-agents/SKILL.md - Tool monitoring

References

• Micrometer: https://micrometer.io
• OpenTelemetry: https://opentelemetry.io
• Spring documentation: /pages/api/observability.adoc