# Observability Guidelines > Observability guidelines for distributed systems using OpenTelemetry, tracing, metrics, and structured logging Use when this capability is needed. - Skill: `tomevault-io/observability-guidelines` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/observability-guidelines` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/observability-guidelines/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/observability-guidelines --- # Observability Guidelines Apply these observability principles to ensure comprehensive visibility into distributed systems and microservices. ## Core Observability Principles - Guide the development of idiomatic, maintainable, and high-performance code with built-in observability - Enforce modular design and separation of concerns through Clean Architecture - Promote test-driven development and robust observability from the start ## OpenTelemetry Integration - Use OpenTelemetry for distributed tracing, metrics, and structured logging - Start and propagate tracing spans across all service boundaries - Use otel.Tracer for creating spans and otel.Meter for collecting metrics - Export data to OpenTelemetry Collector, Jaeger, or Prometheus - Configure appropriate sampling rates for production environments ## Distributed Tracing - Trace all incoming requests and propagate context through internal calls - Use middleware to instrument HTTP and gRPC endpoints automatically - Include trace context in all downstream service calls - Create child spans for significant operations within a service - Add relevant attributes to spans for debugging and analysis ## Metrics Collection Monitor these key metrics across all services: - **Request latency**: Track p50, p90, p95, and p99 percentiles - **Throughput**: Measure requests per second by endpoint - **Error rate**: Track 4xx and 5xx responses separately - **Resource usage**: Monitor CPU, memory, disk, and network utilization - **Custom business metrics**: Track domain-specific KPIs ## Structured Logging - Include unique request IDs and trace context in all logs for correlation - Use structured logging formats (JSON) for machine parseability - Include relevant context: timestamp, service name, trace ID, span ID - Log at appropriate levels: DEBUG, INFO, WARN, ERROR - Avoid logging sensitive information (PII, credentials) ## Architecture Patterns - Apply Clean Architecture with handlers, services, repositories, and domain models - Use domain-driven design principles for clear boundaries - Prioritize interface-driven development with explicit dependency injection - Prefer composition over inheritance; favor small, purpose-specific interfaces ## Correlation and Context - Propagate context through the entire request lifecycle - Use correlation IDs for request tracking across services - Include service version and deployment information in telemetry - Tag traces with relevant business context for filtering - Enable trace-to-log and log-to-trace correlation ## Alerting and Dashboards - Create dashboards for service health and business metrics - Set up alerts based on SLOs and error budgets - Use anomaly detection for proactive issue identification - Document runbooks for common alert scenarios - Review and tune alerts regularly to reduce noise ## Instrumentation Best Practices - Instrument at service boundaries (entry/exit points) - Add custom spans for database operations and external calls - Include relevant attributes (user ID, request type, etc.) - Avoid over-instrumentation that creates noise - Use semantic conventions for consistent attribute naming ## Production Considerations - Configure appropriate sampling rates to balance visibility and cost - Use head-based sampling for consistent trace capture - Implement tail-based sampling for capturing errors - Set retention policies based on debugging needs - Monitor observability infrastructure health --- > Converted and distributed by [TomeVault](https://tomevault.io/claim/mindrally) — claim your Tome and manage your conversions.