# Harness Observability > Harness Observability - Skill: `intense-visions/harness-observability` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add intense-visions/harness-observability` - Raw SKILL.md: https://api.skillmd.com/api/skills/intense-visions/harness-observability/raw - Safety review: pending (external: skill-scanner PASS, skillspector CAUTION) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: Intense-Visions (https://skillmd.com/u/intense-visions) - Updated: 2026-09-21 - Page: https://skillmd.com/skills/intense-visions/harness-observability --- # Harness Observability > Structured logging, metrics, distributed tracing, and alerting strategy. The three pillars of observability, assessed and designed for production readiness. ## When to Use - When designing or reviewing observability instrumentation for a service - When auditing logging, metrics, or tracing coverage gaps - When defining SLIs, SLOs, and alerting strategies for a new feature - NOT for application performance benchmarking (use harness-perf) - NOT for security-focused log analysis (use harness-security-review) - NOT for incident response procedures (use harness-incident-response) > See also: `harness-pulse` is the read-side companion to this skill. Pulse READS observability data (and external analytics/error signals) to produce daily product-pulse reports; this skill DESIGNS the instrumentation that produces those signals in the first place. ## Process ### Phase 1: DETECT -- Identify Existing Instrumentation 1. **Scan for observability libraries.** Check package manifests for instrumentation dependencies: - **Logging:** winston, pino, bunyan, log4js, structlog, serilog, zap, logrus - **Metrics:** prom-client, micrometer, statsd-client, @opentelemetry/sdk-metrics - **Tracing:** @opentelemetry/sdk-trace-node, dd-trace, jaeger-client, zipkin - **Platforms:** @datadog/browser-rum, newrelic, @sentry/node, elastic-apm-node 2. **Locate instrumentation code.** Search for logger, metrics, and tracing initialization: - Logger configuration files (log levels, formatters, transports) - Metrics registry and custom metric definitions - Tracer initialization and span creation patterns - Middleware for automatic HTTP instrumentation 3. **Detect collector and exporter configuration.** Look for: - OpenTelemetry Collector config (`otel-collector-config.yaml`) - Prometheus scrape configuration (`prometheus.yml`) - Grafana dashboards (`grafana/dashboards/`) - Datadog agent configuration (`datadog.yaml`) - Fluentd or Fluent Bit configuration for log forwarding 4. **Identify alerting configuration.** Search for: - Prometheus alerting rules (`alert.rules.yml`) - Grafana alert definitions - PagerDuty, Opsgenie, or Slack integration configs - SLO definitions in monitoring-as-code format 5. **Present detection summary:** ``` Observability Detection: Logging: pino (structured JSON) -- 12 logger instances found Metrics: prom-client -- 8 custom metrics defined Tracing: @opentelemetry/sdk-trace-node -- initialized in src/tracing.ts Collector: OpenTelemetry Collector -> Grafana Cloud Alerting: 3 Prometheus alert rules, Slack integration ``` --- ### Phase 2: AUDIT -- Evaluate Coverage and Quality 1. **Audit logging quality.** Evaluate each logger usage for: - **Structured format:** JSON output with consistent field names, not string concatenation - **Log levels:** Appropriate use of error, warn, info, debug (not everything at info) - **Context fields:** Request ID, user ID, operation name included in log entries - **Correlation IDs:** Trace ID propagated through log entries for cross-service correlation - **Sensitive data:** No PII, credentials, or tokens logged - **Error logging:** Stack traces included for errors, not just error messages 2. **Audit metrics coverage.** Check for standard metrics: - **RED metrics:** Request rate, error rate, duration for every HTTP endpoint - **USE metrics:** Utilization, saturation, errors for system resources - **Custom business metrics:** Domain-specific counters and gauges - **Histogram buckets:** Appropriate bucket boundaries for latency histograms - **Label cardinality:** No high-cardinality labels (user ID, request ID as metric labels) 3. **Audit tracing implementation.** Verify: - **Span creation:** Entry points create root spans, downstream calls create child spans - **Context propagation:** Trace context flows across HTTP boundaries (W3C or B3 headers) - **Span attributes:** Meaningful attributes set on spans (not empty spans) - **Error recording:** Exceptions recorded on spans with status codes - **Sampling strategy:** Configured and appropriate (not sampling 100% in production) 4. **Audit alerting effectiveness.** Check each alert for: - **Actionability:** Does the alert tell the on-call engineer what to do? - **Signal quality:** Based on symptoms (error rate) not causes (CPU usage) - **Thresholds:** Based on SLO burn rate, not arbitrary values - **Runbook link:** Each alert has a link to a runbook or troubleshooting guide - **Routing:** Alerts route to the correct team and escalation path 5. **Score each pillar and identify gaps:** ``` Observability Audit: Logging: 7/10 -- structured, but missing correlation IDs in 4 services Metrics: 5/10 -- RED metrics partial, no business metrics Tracing: 8/10 -- good coverage, sampling needs tuning Alerting: 3/10 -- only 3 rules, no SLO-based alerts, no runbooks ``` --- ### Phase 3: DESIGN -- Recommend Instrumentation Strategy 1. **Design logging strategy.** Recommend: - Standard log format with required fields (timestamp, level, service, traceId, message) - Logger factory that injects correlation context automatically - Log level configuration per environment (debug in dev, info in production) - Log aggregation pipeline (application -> collector -> storage -> query) - Retention policy per environment 2. **Design metrics strategy.** Recommend: - Standard RED metrics for every HTTP endpoint using middleware - Custom business metrics aligned with product KPIs - Histogram bucket configuration based on expected latency distribution - Metric naming convention following Prometheus best practices (`http_request_duration_seconds`) - Dashboard templates for standard service views 3. **Design tracing strategy.** Recommend: - Automatic instrumentation for HTTP, database, and message queue operations - Manual span creation for business-critical code paths - Sampling strategy (head-based for development, tail-based for production) - Baggage propagation for cross-cutting concerns (tenant ID, feature flags) - Trace-to-log correlation via trace ID injection 4. **Define SLIs and SLOs.** For each service endpoint: - **Availability SLI:** Proportion of successful requests (non-5xx) - **Latency SLI:** Proportion of requests faster than threshold (p99 < 500ms) - **SLO target:** 99.9% availability, 99% latency within budget - **Error budget:** Calculation and burn rate alerting - **SLO dashboard:** Visual budget remaining over rolling window 5. **Design alerting strategy.** Recommend: - Multi-window, multi-burn-rate alerts based on SLOs - Page-worthy alerts (high burn rate) vs. ticket-worthy alerts (slow burn) - Alert grouping to reduce noise (one alert per service, not per instance) - Escalation paths with timeouts - Regular alert review cadence (monthly) to retire noisy alerts --- ### Phase 4: VALIDATE -- Verify Instrumentation Correctness 1. **Validate log output.** Check that logs from a test run: - Parse as valid JSON (if structured logging is configured) - Contain required fields (timestamp, level, service name) - Include trace IDs when tracing is active - Do not contain sensitive data (grep for common patterns: password, token, secret) - Use appropriate log levels (no error-level logs for normal operations) 2. **Validate metric exposition.** Verify: - Metrics endpoint (`/metrics`) is accessible and returns Prometheus format - All declared custom metrics appear in the output - Histogram buckets are populated with reasonable values - No high-cardinality label combinations (check label count) - Counter values increment correctly (not resetting unexpectedly) 3. **Validate trace propagation.** Verify end-to-end: - A request to the entry service generates a trace visible in the tracing backend - Child spans appear for downstream service calls - Span attributes are populated with expected values - Error spans have appropriate status and exception recording - Sampling is working as configured (not dropping all traces or keeping all) 4. **Validate alerting rules.** Check: - Alert expressions are syntactically valid (PromQL, LogQL) - Alert thresholds fire on historical data that represents real incidents - Alert routing sends to the correct channel or pager - Runbook links resolve to actual documentation 5. **Generate observability report:** ``` Observability Validation: [PASS/WARN/FAIL] Logging: PASS (structured, correlated, no PII detected) Metrics: WARN (RED metrics present, missing business metrics) Tracing: PASS (propagation verified, sampling at 10%) Alerting: FAIL (no SLO-based alerts, 2 of 3 rules missing runbooks) Priority actions: 1. Define SLOs for /api/orders and /api/payments endpoints 2. Add multi-burn-rate alerts based on SLO error budget 3. Write runbooks for existing alerts 4. Add order_total and payment_success_rate business metrics ``` --- ## Harness Integration - **`harness skill run harness-observability`** -- Primary invocation for observability audit. - **`harness validate`** -- Run after instrumentation changes to verify project health. - **`harness check-deps`** -- Verify observability library dependencies are installed. - **`emit_interaction`** -- Present audit results and SLO design recommendations. ## Success Criteria - All three observability pillars (logging, metrics, tracing) are assessed - Coverage gaps are identified with specific remediation recommendations - SLI/SLO definitions are provided for critical service endpoints - Alerting strategy is evaluated for actionability and signal quality - No sensitive data detected in log output - Correlation between logs and traces is verified ## Examples ### Example: Express.js API with OpenTelemetry ``` Phase 1: DETECT Logging: pino with pino-http middleware Metrics: @opentelemetry/sdk-metrics -> Prometheus Tracing: @opentelemetry/sdk-trace-node -> Jaeger Collector: OTel Collector (otel-collector-config.yaml) Alerting: 5 Prometheus rules in monitoring/alerts.yml Phase 2: AUDIT Logging: 8/10 -- structured JSON, trace IDs present, missing request body size Metrics: 6/10 -- http_request_duration_seconds present, missing queue depth and business metrics (orders_created_total) Tracing: 9/10 -- auto-instrumented HTTP + pg + Redis, manual spans on checkout flow Alerting: 4/10 -- static thresholds, no SLO burn rate, 2 missing runbooks Phase 3: DESIGN SLOs recommended: - POST /api/orders: 99.9% availability, p99 < 800ms - GET /api/products: 99.95% availability, p99 < 200ms Alerting: Replace static "error rate > 5%" with multi-window burn rate Metrics: Add orders_created_total, cart_abandonment_rate gauges Logging: Add request/response body size for capacity planning Phase 4: VALIDATE Log output: PASS (valid JSON, no PII) Metrics endpoint: PASS (all custom metrics present) Trace propagation: PASS (end-to-end verified) Alert rules: WARN (valid PromQL, but thresholds not SLO-based) Result: WARN -- alerting strategy needs SLO alignment ``` ### Example: Go Microservices with Datadog ``` Phase 1: DETECT Logging: zap (structured) across 4 services Metrics: Datadog dogstatsd client Tracing: dd-trace-go with automatic HTTP/gRPC instrumentation Collector: Datadog Agent (datadog.yaml in k8s/) Alerting: 12 monitors in Datadog (Terraform-managed) Phase 2: AUDIT Logging: 9/10 -- consistent structured format, correlation IDs, no PII Metrics: 7/10 -- RED metrics present, custom counters for business events, but missing histogram for gRPC call duration Tracing: 8/10 -- HTTP and gRPC instrumented, database spans present, Redis spans missing Alerting: 6/10 -- good coverage but static thresholds, no error budgets Phase 3: DESIGN 1. Add dd-trace-go Redis integration for complete trace picture 2. Add grpc_server_handling_seconds histogram 3. Define SLOs in Datadog for top 5 endpoints 4. Convert 4 highest-priority monitors to SLO burn rate alerts 5. Add Datadog SLO dashboard for team visibility Phase 4: VALIDATE Log output: PASS Metrics: WARN (missing gRPC histogram) Traces: WARN (Redis spans missing) Alerts: WARN (no SLO-based alerts) Result: WARN -- 3 instrumentation gaps, alerting needs SLO alignment ``` ## Rationalizations to Reject | Rationalization | Reality | | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | "We can see what's happening in CloudWatch logs — we don't need structured logging" | Unstructured log lines cannot be queried, aggregated, or correlated across services. When an incident spans three services, searching for a request ID across unstructured logs is manual forensics. Structured logging is not a nicety — it is the foundation for incident response. | | "We'll add alerting once we've seen a few incidents and know what to alert on" | The first incident is the worst time to define alerting. SLO-based burn rate alerts can be defined from traffic patterns before any incidents occur. Waiting for incidents to define thresholds means every early failure goes undetected. | | "User ID is a useful label for the latency metric — it helps us debug per-user issues" | User ID as a metric label creates one time series per user, which at 100,000 users means 100,000 label combinations. High-cardinality labels exhaust metric storage, cause query timeouts, and make the entire metrics system unstable. Use logs for per-user debugging; use metrics for aggregate signals. | | "The tracing library is initialized, so we have distributed tracing" | Initializing the library creates root spans but does not propagate context across HTTP boundaries, instrument database calls, or connect traces to logs. Trace initialization without verified end-to-end propagation produces disconnected, useless traces. | | "We have alerts — they're just not linked to runbooks yet" | An alert that fires at 3am without a runbook link requires the on-call engineer to start debugging from scratch. The absence of a runbook is not a documentation gap; it is a mean-time-to-recover multiplier. | ## Gates - **No sensitive data in logs.** If PII, credentials, or tokens are detected in log output, it is a blocking finding. The logging configuration must sanitize or redact sensitive fields before any other improvements are made. - **No high-cardinality metric labels.** Metric labels with unbounded values (user IDs, request IDs, timestamps) cause storage explosion and query timeouts. This is a blocking finding. - **No alerting without runbooks.** Production alerts that lack a runbook link are incomplete. Every page-worthy alert must have actionable documentation. - **No tracing without context propagation.** Traces that do not propagate context across service boundaries provide incomplete pictures and mislead investigations. Broken propagation is a blocking finding. ## Escalation - **When observability libraries conflict:** Some combinations (e.g., dd-trace and OTel auto-instrumentation) cause duplicate spans or metric conflicts. Recommend choosing one provider and removing the other. If migration is needed, present a phased approach. - **When sampling drops critical traces:** If tail-based sampling is misconfigured, critical error traces may be dropped. Recommend always-sample rules for error responses and critical code paths, with probabilistic sampling for normal traffic. - **When metric cardinality is already out of control:** Do not attempt to fix retroactively in the metrics backend. Recommend adding new metrics with correct labels and deprecating the high-cardinality ones. Set a timeline for removal. - **When no observability infrastructure exists:** This is a design-from-scratch scenario. Start with structured logging (lowest barrier), then add metrics (RED for HTTP endpoints), then tracing. Do not attempt all three at once.