UitbreidenOS Observability Designer Skill
When to activate
- Designing an observability strategy from scratch for a service or platform
- Moving from log-only monitoring to full three-pillar observability
- Deciding between observability tools (Datadog, Grafana, Honeycomb, New Relic)
- Instrumenting a service with OpenTelemetry
- Setting up structured logging at scale
- Defining what to alert on (and what not to)
When NOT to use
- SLO design — use the slo-architect skill first, then come here to implement
- Runbook writing — use the runbook-generator skill
- Security monitoring (SIEM) — different tooling and threat model
- Business analytics / product metrics — use the analytics-tracking skill
Instructions
Three-pillar observability design
Design an observability strategy for [system/service].
System: [describe — microservices / monolith / serverless / data pipeline]
Current state: [no observability / logs only / basic metrics / partial]
Team: [solo / small SRE team / large platform team]
Budget: [$0 (open source) / $X/month managed service]
Primary pain: [unknown failures / slow debugging / alert fatigue / no visibility into dependencies]
Three pillars:
1. LOGS — What happened (events)
Purpose: debugging, audit trail, root cause analysis
Best for: "tell me exactly what occurred and when"
Structured logging standard:
Every log line = valid JSON with:
- timestamp (ISO 8601)
- level (DEBUG/INFO/WARN/ERROR)
- service (name of the emitting service)
- trace_id (link to trace for correlation)
- message (human-readable)
- [context fields]: user_id, request_id, duration_ms, error_code
Log levels — when to use each:
DEBUG: development only — verbose, never in production by default
INFO: business events that always matter (order placed, user registered)
WARN: unexpected but recoverable (retry #2, slow query > 500ms)
ERROR: needs human attention (payment failed, 3rd-party API down)
FATAL: process must stop (unrecoverable state)
2. METRICS — How much / how fast (aggregations)
Purpose: alerting, dashboards, trending, capacity planning
Best for: "show me the current state at a glance"
Metric types:
Counter: cumulative, always increasing (requests_total, errors_total)
Gauge: current snapshot (active_connections, queue_depth, memory_bytes)
Histogram: distribution over time (request_duration_seconds — gives p50/p95/p99)
Key metrics for every service (USE method):
Utilisation: CPU %, memory %, thread pool saturation
Saturation: queue depth, pending requests, throttle rate
Errors: error_rate, 5xx_count, timeout_rate
Key metrics for request-serving services (RED method):
Rate: requests_per_second
Errors: error_rate (%)
Duration: p50, p95, p99 latency
3. TRACES — Why it happened (causality)
Purpose: end-to-end latency attribution, distributed dependency visibility
Best for: "show me the exact path of this slow request across all services"
Trace anatomy:
- Trace = one end-to-end request (single trace_id)
- Span = one unit of work (service, function, DB query) within the trace
- Context propagation = passing trace_id through all services (headers)
What to instrument:
- Inbound HTTP requests (auto-instrument with OTEL SDK)
- Outbound HTTP calls (auto-instrument)
- Database queries (auto-instrument for most ORMs)
- Message queue publish/consume (manual span)
- External API calls (manual span with provider, endpoint, status)
Design the three-pillar strategy for my system with specific tool recommendations.
Tooling selection
Choose observability tooling for [team/budget].
Scale: [X services / X requests per second / X GB logs per day]
Team: [X engineers]
Budget: $[X/month]
Cloud provider: [AWS / GCP / Azure / multi-cloud / on-prem]
Existing stack: [what you already use]
Tool comparison:
MANAGED (pay-per-use, least ops overhead):
Datadog: Best all-in-one. Logs + Metrics + Traces + APM + Dashboards + Alerts.
Cost: $15-23/host/month + $0.10/GB logs ingested. Expensive at scale.
Best for: < 100 hosts, teams without dedicated ops
Honeycomb: Best for traces and high-cardinality data. Query any field, not just pre-aggregated.
Cost: $130/month starter. Pricing by events.
Best for: distributed systems with complex debugging needs
New Relic: Full-stack observability, generous free tier (100GB/month free).
Cost: $0 for < 100GB/month. $0.30/GB after.
Best for: teams wanting full stack on a budget
Grafana Cloud: Open source stack (Loki + Prometheus + Tempo) as managed service.
Cost: Free tier generous. Paid from $8/month.
Best for: teams comfortable with Prometheus/Grafana, wanting managed ops
SELF-HOSTED (more ops, lower cost at scale):
Prometheus + Grafana: Metrics + dashboards. Industry standard, free.
Add Loki (logs) + Tempo (traces) for full three-pillar stack.
Cost: infrastructure only (~$50-200/month on a small cluster)
Best for: > $5K/month observability budget, teams with ops capacity
OpenTelemetry: Vendor-neutral instrumentation standard. Collect once, export anywhere.
Use with: any backend above. Future-proofs your instrumentation.
Recommendation: always instrument with OTEL — swap backends without code changes.
Recommendation for my constraints: [tool stack + rationale + estimated monthly cost]
Structured logging implementation
Implement structured logging for [service].
Language/framework: [Node.js/Express / Python/FastAPI / Go / Java/Spring]
Current logging: [console.log / print() / basic logger / none]
Destination: [CloudWatch / Datadog / Elasticsearch / stdout (for k8s)]
Node.js (using Pino — fastest structured logger):
import pino from 'pino';
const logger = pino({
level: process.env.LOG_LEVEL || 'info',
...(process.env.NODE_ENV === 'development' && {
transport: { target: 'pino-pretty' } // Human-readable in dev
}),
});
// Request logger middleware (Express):
import { randomUUID } from 'crypto';
app.use((req, res, next) => {
req.id = randomUUID();
req.log = logger.child({ req_id: req.id, path: req.path, method: req.method });
req.log.info('request_received');
next();
});
// Logging patterns:
req.log.info({ user_id: user.id, action: 'payment_initiated', amount_cents: 4999 }, 'Payment started');
req.log.warn({ retry_attempt: 3, service: 'stripe' }, 'Stripe slow response, retrying');
req.log.error({ error: err.message, stack: err.stack, order_id: order.id }, 'Payment failed');
Python (using structlog):
import structlog
log = structlog.get_logger()
# Bind context for the request:
log = log.bind(request_id=request_id, user_id=user_id)
log.info("payment_initiated", amount=49.99, currency="USD")
log.error("payment_failed", error=str(e), stripe_error_code=e.code)
Go (using zerolog):
log := zerolog.New(os.Stdout).With().Timestamp().Str("service", "payment").Logger()
log.Info().Str("user_id", userID).Int("amount_cents", 4999).Msg("payment_initiated")
log.Error().Err(err).Str("order_id", orderID).Msg("payment_failed")
Generate implementation for my specific language and framework.
Alerting philosophy
Design an alerting strategy for [system].
Current state: [no alerts / too many alerts / alert fatigue]
Team size: [X people on-call]
On-call rotation: [24/7 / business hours / single person]
Incident history: [common failure patterns]
Alerting philosophy (Google SRE):
ALERT ON SYMPTOMS, NOT CAUSES:
❌ Alert: "CPU > 80%" — cause (might not affect users)
✅ Alert: "Error rate > 1%" — symptom (users are affected)
❌ Alert: "Disk > 85%" — cause (logs rotate, it self-heals)
✅ Alert: "Disk will fill in < 4 hours" — predictive, actionable
ALERT TIERS:
Page (wake someone up — severity: critical/P1):
- User-visible outage or impairment
- SLO burn rate fast-burn threshold exceeded
- Error rate > X% for > 5 minutes
- Service completely unreachable
Rule: if you wouldn't wake someone up for it, don't page
Ticket (fix next business day — severity: warning/P2):
- SLO slow-burn threshold exceeded
- Degraded performance below threshold
- Dependency showing instability
Rule: ticket fires are resolved in 1-2 business days; not page-worthy
Informational (log, don't act — Slack/dashboard):
- Deployments, restarts, scaling events
- Approaching capacity (not yet at threshold)
Rule: no alert → no action required
Alert design principles:
- Every alert must have a runbook link
- Alert name = what's happening + who's affected
- Every alert must be actionable — if no action is possible, it shouldn't alert
- Alert on rate of change, not static thresholds
- No alert with < 5-minute window — everything < 5 min is noise
Anti-patterns to eliminate:
□ Alerts that fire multiple times before anyone looks at them
□ Alerts with no clear owner
□ Alerts that are always acknowledged without investigation
□ "Dashboard-only" metrics that don't have corresponding alerts
Design the alerting strategy for my system with specific threshold recommendations.
Example
User: We have 8 microservices, only basic CloudWatch logs, and we get paged 3x a week on average. Most pages are for things that auto-resolved. How do we fix this?
Claude's observability plan:
Three pages/week with mostly auto-resolving issues = alert fatigue from cause-based alerting, not symptom-based. Fix the alerting philosophy first, then add the observability you're missing.
Phase 1 (this week) — Fix alerting, no new tooling:
Change every CloudWatch alarm from threshold-based (CPU > 80%) to symptom-based:
- Replace
CPU > 80%with5xx error rate > 0.5% for 10 minutes - Replace
Memory > 85%withp99 latency > 2x baseline for 5 minutes - For auto-resolving issues: increase window from 5 min → 15 min — most will stop paging
Result: typically reduces page volume by 50-70% immediately.
Phase 2 (this month) — Add traces: Your 8 microservices need distributed tracing or you'll never find the slow one. Cheapest path: instrument all 8 with OpenTelemetry auto-instrumentation (1-2 lines of code per service), export to AWS X-Ray (free for first 100K traces/month). This gives you: the slow span, the slow service, the slow database query — without manual log correlation.
Phase 3 (next quarter) — Structured logs:
Replace console.log("Payment failed: " + err) with structured JSON across all services. Add trace_id to every log line. Now you can: CloudWatch Insights query → find the exact log lines for a specific trace.
Tool recommendation for 8 services at startup scale: Grafana Cloud free tier covers you: Prometheus metrics + Loki logs + Tempo traces. Zero cost until you scale beyond their generous free limits. When you hit $500/month Grafana cost, re-evaluate.