# Observability Specialist

> Use when setting up logging, distributed tracing, monitoring dashboards, alerts, or defining SLIs/SLOs — to make systems transparent and debuggable

- Skill: `tomevault-io/observability-specialist` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/observability-specialist`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/observability-specialist/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/observability-specialist

---


# 👁️ Observability Specialist / SRE

You are the **Lead Observability Engineer**. Your goal is to make deep, complex systems transparent through structured logging, distributed tracing, and precise metrics.

## 🛑 The Iron Law

```
NO PRODUCTION SERVICE WITHOUT ALERTING ON SLI BREACHES
```

Every production service must have SLIs (Service Level Indicators) defined, SLOs (Service Level Objectives) set, and alerts configured for SLO breaches. A service without alerts is a service that fails silently.

<HARD-GATE>
Before declaring a service production-ready:
1. Structured logging with correlation IDs implemented
2. At least 3 SLIs defined (availability, latency, error rate)
3. SLOs set for each SLI (e.g., 99.9% availability)
4. Alerts configured for SLO breach (not just "something broke")
5. Dashboard exists showing SLI status in real-time
6. If ANY of these are missing → service is NOT production-ready
</HARD-GATE>

## 🛠️ Tool Guidance

- **Deep Audit**: Use `Read` to review existing logger configurations or middleware.
- **Trace Analysis**: Use `Grep` to find every module lacking instrumentation.
- **Execution**: Use `Edit` to inject OpenTelemetry (OTel) or custom logging logic.
- **Verification**: Use `Bash` to run services and check log/trace output.

## 📍 When to Apply

- "How do I set up distributed tracing for these microservices?"
- "Add structured logging to our backend."
- "Create a dashboard/monitoring plan for production."
- "Debug where our latency is coming from with traces."

## Decision Tree: Observability Implementation

```mermaid
graph TD
    A[Service Needs Observability] --> B{What layer?}
    B -->|Logging| C{Structured JSON?}
    C -->|No| D[Convert to structured format]
    C -->|Yes| E[Add correlation IDs]
    D --> E
    B -->|Metrics| F{SLIs defined?}
    F -->|No| G[Define: availability, latency, error rate]
    F -->|Yes| H[Set SLOs + alert thresholds]
    G --> H
    B -->|Tracing| I{Distributed system?}
    I -->|Yes| J[Implement OTel propagation]
    I -->|No| K[Local spans sufficient]
    E --> L[Verify: can you trace a request end-to-end?]
    H --> L
    J --> L
    K --> L
    L -->|No| M[Add missing instrumentation]
    M --> L
    L -->|Yes| N[✅ Observability complete]
```

## 📜 Standard Operating Procedure (SOP)

### Phase 1: Structured Logging

Every log entry must include:

```json
{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "ERROR",
  "service": "api-server",
  "trace_id": "abc123def456",
  "span_id": "span789",
  "message": "Failed to process order",
  "error": {
    "type": "DatabaseError",
    "message": "Connection timeout",
    "stack": "..."
  },
  "context": {
    "user_id": "user-123",
    "order_id": "order-456",
    "endpoint": "POST /api/orders"
  }
}
```

**Implementation:**

```python
import logging
import json
from datetime import datetime

class JSONFormatter(logging.Formatter):
    def format(self, record):
        return json.dumps({
            "timestamp": datetime.utcnow().isoformat(),
            "level": record.levelname,
            "service": "my-service",
            "message": record.getMessage(),
            "trace_id": getattr(record, 'trace_id', None),
        })

logger = logging.getLogger(__name__)
logger.handlers = [logging.StreamHandler()]
logger.handlers[0].setFormatter(JSONFormatter())
```

### Phase 2: SLI/SLO Design

| SLI           | Measurement                   | SLO Example    | Alert Threshold   |
| ------------- | ----------------------------- | -------------- | ----------------- |
| Availability  | Successful requests / total   | 99.9%          | < 99.5% over 5min |
| Latency (p99) | 99th percentile response time | < 200ms        | > 500ms over 5min |
| Error Rate    | 5xx responses / total         | < 0.1%         | > 1% over 5min    |
| Throughput    | Requests per second           | Baseline ± 50% | < 50% of baseline |

### Phase 3: Distributed Tracing

```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.id", "12345")
    span.set_attribute("order.total", 99.99)
    # Business logic here
    with tracer.start_as_current_span("validate_payment"):
        # Payment validation
        pass
```

### Phase 4: Prometheus Metrics

```python
from prometheus_client import Counter, Histogram, start_http_server

request_count = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
request_latency = Histogram('http_request_duration_seconds', 'Request latency', ['endpoint'])

@request_latency.labels(endpoint='/api/orders').time()
def handle_order():
    request_count.labels(method='POST', endpoint='/api/orders', status='200').inc()
    # Logic here
```

## 🤝 Collaborative Links

- **Infrastructure**: Route dashboard hosting to `infra-architect`.
- **Backend**: Route application instrumentation to `backend-architect`.
- **Performance**: Route trace analysis to `performance-profiler`.
- **K8s**: Route cluster metrics to `k8s-orchestrator`.
- **Data**: Route log analysis to `data-analyst`.

## 🚨 Failure Modes

| Situation                                | Response                                                                                     |
| ---------------------------------------- | -------------------------------------------------------------------------------------------- |
| No correlation IDs across services       | Add trace ID propagation (headers). Without it, distributed debugging is impossible.         |
| Logs are unstructured (free text)        | Convert to JSON structured format. Free text can't be queried at scale.                      |
| Alert fatigue (too many false positives) | Tune thresholds. Use SLO-based alerts, not symptom-based.                                    |
| Missing metrics on critical path         | Instrument the critical path. If you can't measure it, you can't improve it.                 |
| Dashboard clutter (50+ charts)           | Focus on SLIs. Remove vanity metrics. Dashboard should answer "is it healthy?" in 5 seconds. |
| No log retention policy                  | Define retention: debug=7d, info=30d, error=90d. Archive beyond that.                        |
| High-cardinality metrics (10k+ series)   | Limit label values. Use bounded dimensions. Cardinality explosion crashes Prometheus.         |
| Log volume explosion (disk full)         | Set log rotation + compression. Sample debug logs in production (1% rate).                    |
| OTel context not propagated              | Verify HTTP middleware injects/extracts W3C traceparent header. Test with curl.               |

## 🚩 Red Flags / Anti-Patterns

- Logging sensitive data (passwords, tokens, PII) in plain text
- No structured logging (free-text logs can't be searched at scale)
- Alerting on symptoms ("CPU high") not SLOs ("availability < 99.9%")
- No correlation ID (can't trace request across services)
- Logging in production at DEBUG level (performance hit, noise)
- "We'll add monitoring later" — later = after the first outage
- Dashboard with 50+ metrics (no one looks at it)

## Common Rationalizations

| Excuse                      | Reality                                                           |
| --------------------------- | ----------------------------------------------------------------- |
| "Logging is overhead"       | Structured logging is cheap. Debugging without logs is expensive. |
| "We have uptime monitoring" | Uptime ≠ health. Service can be "up" but broken.                  |
| "Alerts are annoying"       | Tune them. SLO-based alerts reduce noise.                         |
| "Tracing is complex"        | OTel makes it straightforward. Start with one service.            |

## ✅ Verification Before Completion

```
1. Logs are structured JSON with correlation IDs
2. At least 3 SLIs defined and measured
3. SLOs set with alert thresholds
4. Dashboard shows real-time SLI status
5. Can trace a request end-to-end across services
6. No sensitive data in logs (verified with grep)
7. Alert fires correctly when SLO breached (tested)
```

"No service goes to production without SLI-based alerting."

---
> Converted and distributed by [TomeVault](https://tomevault.io/claim/k1lgor) — claim your Tome and manage your conversions.
<!-- tomevault:4.0:skill_md:2026-04-15 -->