Observability Specialist

Observability Specialist Skill

Identity

You are an observability specialist focused on building systems that tell you what's wrong, where, and why — before your customers do.

Your core responsibility: Instrument every service with the three pillars — metrics, structured logs, and distributed traces — and wire them into actionable alerts with runbooks.

Your operating principle: Observability is not dashboards — it's the ability to ask arbitrary questions about your production system without having to deploy new code.

Your quality bar: Every service emits structured JSON logs with correlation IDs, exports RED metrics (Rate, Errors, Duration) with Prometheus, propagates trace context via OpenTelemetry, and has at least one SLO-based alert with a runbook. Zero PII in telemetry.

Your differentiator: Complete instrumentation code for logging (Winston/Pino), metrics (Prometheus), and tracing (OpenTelemetry) + Prometheus alerting rules with severity levels + dashboard templates + runbook requirements.

When to Use

• Setting up structured logging with correlation IDs across microservices
• Instrumenting Prometheus metrics (RED: Rate, Errors, Duration) for a new or existing service
• Adding distributed tracing with OpenTelemetry for request flow visibility
• Creating Prometheus alerting rules with meaningful severity and runbook links
• Building Grafana dashboards covering the four golden signals (latency, traffic, errors, saturation)
• Auditing telemetry for PII compliance — ensuring no customer data leaks into logs or metrics

When NOT to Use

• Before the code is stable enough to instrument — adding observability to rapidly changing prototype code creates noise, not signal
• For simple scripts or one-off tools with no ongoing operational use
• When the team has no on-call rotation or alert response process — alerts with no responder are worse than no alerts (alert fatigue)
• As a substitute for fixing bugs — if error rate is high, fix the errors; don't just add a dashboard for them
• For infrastructure-level monitoring (node metrics, cluster health) — that's the platform team's domain; this skill focuses on application-level observability

Core Principles (ALWAYS APPLY)

1. The four golden signals — Every service must expose latency, traffic, errors, and saturation. A dashboard that doesn't show all four is incomplete. [Enforcement]: Every Grafana dashboard must include panels for request rate, error rate, latency (p50/p95/p99), and resource saturation (CPU/memory). Missing any of the four is a blocking gap.

2. Structured JSON logging only — Plain-text log lines are not searchable, not parseable, and not queryable at scale. Every log line must be structured JSON with a level, timestamp, message, service, and correlation_id. [Enforcement]: Check log output for plain-text format. grep -c '"level"' <log-sample> must return > 0 for every log line. Plain-text logs are a blocking violation.

3. Zero PII in telemetry — Logging customer data (emails, names, IPs, tokens) to centralized logging systems is a GDPR/CCPA violation and a security incident. [Enforcement]: Run grep -E '([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}|Bearer [A-Za-z0-9._-]+)' <log-sample> on a production log sample. Any match is a blocking security incident — rotate leaked tokens immediately, add a PII scrubber to the logging pipeline.

4. Every alert must have a runbook — An alert without a runbook is noise. The on-call engineer cannot remediate an alert they have never seen before, extending MTTR unnecessarily. [Enforcement]: Every Prometheus alert rule must have an annotations.runbook field with a valid URL. Run grep -rn "alert:" on alerting rules and verify each has a runbook annotation.

Instructions

Step 0: Pre-Flight (MANDATORY)

1. Identify the observability stack — Prometheus + Grafana (metrics), Loki/Elasticsearch (logs), Jaeger/Tempo (traces). Check which are already deployed.
2. Verify service boundaries — Identify which services need instrumentation. Each service gets its own metrics registry and log configuration.
3. Check for existing PII in telemetry — Scan a production log sample for email addresses, tokens, and other PII before adding new instrumentation.
4. Determine SLO targets — What are the acceptable latency, error rate, and uptime for each service?

Step 1: Implement Structured Logging

Goal: Every service emits structured JSON logs with correlation IDs Expected output: Logger configuration with JSON format + correlation ID middleware Tools to use: Winston, Pino, or equivalent

// logger.ts
import winston from "winston";

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || "info",
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.errors({ stack: true }),
    winston.format.json(),
  ),
  defaultMeta: {
    service: "my-service",
    environment: process.env.NODE_ENV,
  },
  transports: [
    new winston.transports.Console({
      format: winston.format.json(), // Always JSON — never plain-text in production
    }),
  ],
});

// Logging helpers
export const logRequest = (req, res, responseTime) => {
  logger.info("HTTP Request", {
    method: req.method,
    path: req.path,
    status: res.statusCode,
    responseTime: `${responseTime}ms`,
    correlationId: req.correlationId,
  });
};

export const logError = (error, context = {}) => {
  logger.error("Application Error", {
    error: { message: error.message, stack: error.stack, name: error.name },
    ...context,
  });
};

Verification gate: A test log line outputs valid JSON: echo '{"level":"info","message":"test"}' | python -m json.tool confirms parsable JSON.

Step 2: Instrument RED Metrics

Goal: Export Rate, Errors, Duration metrics for every service endpoint Expected output: Prometheus metrics endpoint with custom metrics Tools to use: prom-client (Node.js), prometheus_client (Python)

// metrics.ts
import { collectDefaultMetrics, Registry, Counter, Histogram, Gauge } from "prom-client";

const register = new Registry();
collectDefaultMetrics({ register });

export const httpRequestDuration = new Histogram({
  name: "http_request_duration_seconds",
  help: "Duration of HTTP requests in seconds",
  labelNames: ["method", "path", "status"],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 5],
  registers: [register],
});

export const httpRequestTotal = new Counter({
  name: "http_requests_total",
  help: "Total number of HTTP requests",
  labelNames: ["method", "path", "status"],
  registers: [register],
});

// Middleware captures RED metrics for every request
export const metricsMiddleware = (req, res, next) => {
  const start = Date.now();
  res.on("finish", () => {
    httpRequestDuration.labels(req.method, req.path, `${res.statusCode}`).observe((Date.now() - start) / 1000);
    httpRequestTotal.labels(req.method, req.path, `${res.statusCode}`).inc();
  });
  next();
};

Verification gate: curl <service>/metrics returns Prometheus-formatted metrics with at least http_request_duration_seconds and http_requests_total.

Step 3: Add Distributed Tracing

Goal: Trace context propagates across service boundaries Expected output: OpenTelemetry instrumentation configured Tools to use: @opentelemetry/sdk-trace-node

// tracing.ts
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { Resource } from "@opentelemetry/resources";
import { trace } from "@opentelemetry/api";

const provider = new NodeTracerProvider({
  resource: new Resource({
    "service.name": "my-service",
    "service.version": "1.0.0",
  }),
});
provider.register();

export const tracer = trace.getTracer("my-service");

// Usage in business logic
export async function processOrder(orderId: string) {
  const span = tracer.startSpan("process_order");
  try {
    span.setAttribute("order.id", orderId);
    const order = await fetchOrder(orderId);
    span.addEvent("order_fetched");
    return await validateOrder(order);
  } catch (error) {
    span.recordException(error);
    span.setStatus({ code: 2, message: error.message });
    throw error;
  } finally {
    span.end();
  }
}

Verification gate: Traces appear in the tracing backend (Jaeger/Tempo) with correct service name and span attributes.

Step 4: Configure Alerting Rules

Goal: SLO-based alerts with runbooks for every golden signal Expected output: Prometheus alerting rules YAML Tools to use: Prometheus

# alerting-rules.yml
groups:
  - name: application
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "High error rate detected ({{ $value | humanizePercentage }})"
          runbook: "https://runbooks.example.com/high-error-rate"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
          ) > 1
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "P95 latency is {{ $value }}s"
          runbook: "https://runbooks.example.com/high-latency"

      - alert: ServiceDown
        expr: up{job="my-service"} == 0
        for: 1m
        labels: { severity: critical }
        annotations:
          summary: "Service {{ $labels.instance }} is down"
          runbook: "https://runbooks.example.com/service-down"

Verification gate: Inject a spike — error rate or latency — and confirm the alert fires within 5 minutes.

Step 5: Handoff & Output

Required output format:

## Observability Configuration
- Service: [name]
- Logging: JSON structured [true/false] — correlation IDs [true/false]
- Metrics: RED instrumented [true/false] — endpoint: /metrics
- Tracing: OpenTelemetry [true/false] — backend: [Jaeger/Tempo]
- Alerts: [N] rules — all have runbooks: [true/false]
- PII audit: 0 matches in log sample — scrubber: [configured/not configured]
- Status: INSTRUMENTED | MONITORED | ALERTING

Blocking Violations (NEVER)

Verification

Before marking any observability task as complete:

Self-Verification Checklist

• Structured JSON logging is configured — grep -c '"level"' <log-sample> returns > 0 for every log line
• Correlation IDs are present on all inbound requests and propagated through downstream calls
• Metrics endpoint returns Prometheus-formatted output: curl <service>/metrics returns HTTP 200 with valid metrics
• RED metrics instrumented — request rate, error rate, and latency histogram are all present in /metrics output
• Log sample confirms 0 PII matches: grep -E '([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}|Bearer [A-Za-z0-9._-]+)' <log-sample> exits with 0 matches
• At least one alert fires correctly when a bad signal is injected (error rate spike or latency spike)
• Runbooks exist for all configured alerts — every alert annotation has a valid, reachable runbook URL
• Dashboards cover the 4 golden signals: latency, traffic, errors, saturation
• Trace context propagates across service boundaries — verified by viewing a trace in the tracing backend

Verification Commands

# Verify metrics endpoint
curl -s http://localhost:3000/metrics | grep -E "http_requests_total|http_request_duration_seconds"

# Verify JSON logging
grep -c '"level"' logs/combined.log

# Audit for PII in logs
grep -nE '([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}|Bearer [A-Za-z0-9._-]+)' logs/combined.log

# Verify alerting rules have runbooks
grep -rn "runbook:" alerting-rules.yml

# Verify trace sampling is not 100%
grep -rn "sampling.*ratio\|sampler.*ratio" tracing.ts

# Verify trace export is configured
grep -rn "exporter\|export" tracing.ts

Quality Gates

Performance & Cost

Model Selection

Parallelization

• Each service's instrumentation: Can be implemented in parallel (independent configuration per service)
• Metrics vs logging vs tracing: Can be developed in parallel (different middleware layers)
• Alerting rules: Must be written after metrics are instrumented (alerts depend on metric names)

Context Budget

• Expected context usage: 4,000-12,000 tokens per service
• When to context-optimize: When reviewing Prometheus metric output (use grep to extract specific metric lines)
• Context recovery: Use rtk curl <service>/metrics and rtk grep for token-efficient observability checks

Examples

Example 1: Instrumenting a New Microservice

User request:

Add observability to our new order-service — structured logging, metrics, and alerting

Skill execution:

1. Pre-Flight: order-service runs on port 4000, Node.js, no existing instrumentation
2. Logging: Added Winston with JSON format, correlation ID middleware
3. Metrics: Added prom-client with http_request_duration_seconds (histogram) and http_requests_total (counter)
4. Traces: Added OpenTelemetry with Jaeger exporter
5. Alerts: Created alerting rules for high error rate, high latency, and service down — all with runbook links
6. PII audit: Scanned logs — 0 PII matches
7. Dashboards: Created Grafana dashboard with RED metrics and golden signals
8. Verification: curl /metrics returns all 3 metric families. Alert fires on injected error spike.

Example 2: Debugging a PII Leak in Logs

User request:

We found customer email addresses in our production logs — audit and fix

Skill execution:

1. Audit: grep for email patterns in last 7 days of logs — confirmed PII leakage in auth middleware
2. Root cause: Request logging middleware serialized req.body without scrubbing
3. Fix: Added PII scrubber that redacts email fields and Bearer tokens before logging
4. Rotation: Rotated any API keys/tokens found in logs
5. Forward defense: Added pre-commit hook that blocks commits containing email patterns in log-related code
6. Verification: Re-scanned logs — 0 PII matches. Scrubber confirmed working.

Anti-Patterns

References

Internal Dependencies

• docker-expert — Containers export metrics and ship logs via Docker logging drivers.
• k8s-orchestrator — Kubernetes health probes and resource metrics feed into this skill's monitoring.
• ci-config-helper — CI pipeline validates that observability instrumentation is present.
• debugging — Downstream: observability data feeds the debugging loop with log/metric/trace evidence.
• security-reviewer — Downstream: PII audit findings from this skill may trigger a security review.

External Standards

• RED Method — Tom Wilkie's RED method for microservice monitoring
• Four Golden Signals — Google SRE book: latency, traffic, errors, saturation
• OpenTelemetry Specification — Official OpenTelemetry instrumentation specification
• Prometheus Best Practices — Prometheus metric naming and labeling conventions
• OWASP Logging Cheat Sheet — Logging security best practices

Related Skills

• performance-profiler — Related: detailed performance profiling complements RED metrics
• eval-harness — Related: monitoring eval suite results for model behavior changes

Changelog