Tpl Situacao Monitoramento Observabilidade

SITUATION: Monitoring & Observability Setup

Skill gerado a partir do pack templates-claude-code. Arquivo de origem: situacao/14-monitoramento-observabilidade.md. Use como baseline e adapte ao projeto antes de mudancas grandes.

Conteudo do template

CONTEXT

Use this configuration when setting up or improving observability for a live application — adding metrics, structured logging, distributed tracing, alerting, SLO definitions, or runbook creation. Observability is not an add-on — an unmonitored production service is a liability you can't reason about. This configuration ensures you can answer: "Is the service healthy right now? How do I know? What do I do when it's not?"

OBJECTIVES

• Instrument all three pillars of observability: metrics, logs, and traces
• Define SLOs/SLIs before incidents reveal you needed them
• Alert on symptoms (what users experience) not just causes (CPU usage)
• Create actionable runbooks linked from every alert
• Enable a developer to debug a production incident using only the observability stack

APPROACH RULES

1. Three pillars: metrics, logs, traces. Metrics tell you something is wrong. Logs tell you what happened. Traces tell you where in the system it happened. You need all three for full observability.

2. Alert on symptoms, not causes. "Error rate > 1%" is a symptom — users are affected. "CPU > 80%" is a cause — but doesn't necessarily mean users are impacted. Start with symptom-based alerts.

3. Every alert must have a runbook. If an alert fires, the on-call engineer must know what to do. An alert without a runbook is just noise that trains people to ignore alerts. Runbook link must be in the alert body.

4. SLO > SLA. Define Service Level Objectives (SLOs) and track your error budget. If you're burning through error budget, slow down feature development and fix reliability. If you have budget headroom, safe to move fast.

5. Structured logging everywhere. Never console.log("user logged in"). Always logger.info({ event: "user_login", userId: user.id, ip: req.ip }). Structured logs are queryable. Strings are not.

6. Correlation IDs across services. Every request gets a unique trace/correlation ID at the entry point. Every log line, every downstream call includes that ID. Without this, distributed debugging is a nightmare.

7. Monitor what users do, not just what systems do. Business metrics (conversion rate, checkout completions, active users) belong on the same dashboard as technical metrics. When technical metrics are fine but business metrics drop, you have a bug.

ROUTING TABLE

Three Pillars Implementation Guide

Metrics (What is happening)

// Using prom-client (Node.js)
import { Counter, Histogram, register } from 'prom-client'

const httpRequestDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5]
})

const httpRequestTotal = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status_code']
})

// Expose at /metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType)
  res.send(await register.metrics())
})

Logs (What happened)

// Using pino (Node.js)
import pino from 'pino'

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  base: { service: 'api', version: process.env.APP_VERSION },
  timestamp: pino.stdTimeFunctions.isoTime
})

// ✅ Good: structured, queryable
logger.info({ event: 'payment_processed', userId, amount, currency }, 'Payment processed')
logger.error({ err: error, userId, requestId }, 'Payment failed')

// ❌ Bad: string only, not queryable
console.log(`Payment failed for user ${userId}: ${error.message}`)

Traces (Where it happened)

// Using OpenTelemetry
import { trace, context, propagation } from '@opentelemetry/api'

const tracer = trace.getTracer('my-service')

async function processOrder(orderId: string) {
  const span = tracer.startSpan('processOrder', {
    attributes: { 'order.id': orderId }
  })
  
  try {
    await span.setAttribute('order.status', 'processing')
    const result = await doWork(orderId)
    span.setStatus({ code: SpanStatusCode.OK })
    return result
  } catch (err) {
    span.recordException(err)
    span.setStatus({ code: SpanStatusCode.ERROR })
    throw err
  } finally {
    span.end()
  }
}

SLO Definition Template

service: checkout-api
slos:
  - name: Checkout Success Rate
    description: Percentage of checkout requests that succeed
    sli: (sum of successful checkout requests) / (sum of all checkout requests)
    target: 99.5%
    measurement_window: 30 days
    error_budget: 0.5% → 216 minutes/month allowable downtime

  - name: Checkout Latency
    description: P95 latency for checkout endpoint
    sli: 95th percentile response time
    target: < 1000ms
    measurement_window: 30 days

alerts:
  - name: Checkout Error Rate Burn
    condition: Error budget burn rate > 5× for 1 hour
    severity: page (wake someone up)
    runbook: docs/runbooks/checkout-errors.md

DO NOT

• DO NOT set up monitoring after the first incident — it's too late then
• DO NOT log PII (email, name, phone, card numbers) — mask or hash before logging
• DO NOT create alerts that can't be acted on — each alert needs a clear response procedure
• DO NOT use average latency as your primary metric — use P95 and P99
• DO NOT monitor only infrastructure (CPU, memory) and ignore application-level health
• DO NOT alert with CRITICAL severity on things that are informational
• DO NOT delete old logs without understanding retention requirements (GDPR, compliance)

OUTPUT FORMAT

For each monitoring setup, deliver:

Observability Coverage Matrix:

| Signal | Tool | Coverage | Retention | Alert? |
|--------|------|----------|-----------|--------|
| App logs | Loki | INFO+ in prod. DEBUG in staging | 30 days | On ERROR+ |
| Infrastructure metrics | Prometheus | CPU, memory, disk, network | 90 days | On threshold |
| App metrics | Prometheus | Request rate, error rate, latency histograms | 90 days | On SLO burn |
| Traces | Jaeger/Tempo | Sample rate: 10% prod, 100% staging | 7 days | On trace error |
| Uptime | Healthchecks.io | External probe every 60s | 365 days | On unavailable |

**Runbook Inventory:**
| Alert | Runbook | Owner | Last Verified |
|-------|---------|-------|---------------|
| High error rate | docs/runbooks/high-error-rate.md | Platform | 2024-01-01 |

QUALITY GATES

• P95 and P99 latency tracked for all user-facing endpoints
• Error rate (5xx rate) tracked and alerting at > 1%
• Every log entry is structured JSON with at minimum: timestamp, level, service, message
• Correlation/trace IDs present in all logs for every request
• At least one SLO defined with error budget tracked on dashboard
• Every production alert has a linked runbook
• Runbook tested: follow it cold and confirm it resolves the described scenario
• PII not present in any log line (verified by audit sample)
• On-call rotation documented and first responder always identified
• Post-mortem template ready before first incident (not after)