Monitoring

Monitoring

You are wiring up the outside view of a service that already shipped: is it alive, is it fast, and when it breaks, does exactly one human get exactly one actionable page. This skill emits a concrete setup — a checker config, a health-endpoint contract, symptom-based alert rules, and an on-call rotation. Not telemetry instrumentation (that is ../observability/SKILL.md), not the release-gating healthcheck (that is ../deployment/SKILL.md).

The one rule

A page is justified only when there is real user impact AND a human action the system can't take itself. Everything below descends from this. Internalize the three tiers:

• Page (wake someone): users are hurting now and a human must intervene. Checkout returns 5xx. Site unreachable. Error budget burning fast.
• Ticket (look during business hours): degraded but not bleeding. Slow-burn budget use, cert expiring in 14 days.
• Dashboard-only (don't notify): CPU at 80%, a single retry, a transient blip the system already healed.

If an alert doesn't map to an immediate human action, it is not a page — it's noise, and noise trains people to ignore the one page that matters.

The 4-layer stack

Don't skip layers and don't collapse them — each answers a different question.

Layer 1 catches "the whole thing is gone." Layer 2 catches "a dependency died" before users do. Layer 3 catches "we're degrading faster than we can afford." Layer 4 makes sure a human shows up.

Pick a tool

Decide by budget, team size, and self-host appetite. Pricing as of 2026-06.

Default: Uptime Kuma if you already run a VPS (1 GB box is comfortable; needs ~400 MB RAM), UptimeRobot free tier if you don't want infra. Kuma 2.1 added Globalping worldwide probe locations (so you test from regions, not just your one box) and built-in domain-expiry monitors. Concrete docker-compose and notification wiring live in references/tool-setup.md.

Health endpoints done right

Split liveness from readiness — they trigger different machine actions.

• /livez (liveness): is the process healthy? If it fails, the orchestrator restarts the container. Keep it dumb: just "am I running and not deadlocked." Never check the database here.
• /readyz (readiness): are dependencies reachable so I can serve traffic? If it fails, the orchestrator pulls this instance from the load-balancer rotation but does not kill it.

The cheap-probe rule: a probe runs constantly, so it must be <100ms and must not cascade-check every downstream. Why: if /livez pings the DB and the DB is briefly slow, liveness fails, the container restarts, the restart hammers the recovering DB — a restart loop that turns a 30-second blip into an outage.

# Bad — one /health that cascades and returns 500 on any hiccup.
# A slow Redis takes the whole service down and triggers restart loops.
@app.get("/health")
def health():
    db.execute("SELECT 1")          # blocks
    redis.ping()                    # blocks
    requests.get(PAYMENTS_URL)      # blocks on a third party!
    return {"status": "ok"}         # 500 if ANY of these throws

# Good — split, cheap, correct status codes, small JSON.
@app.get("/livez")                  # liveness: process only. Restart if this fails.
def livez():
    return {"status": "alive"}      # 200, ~1ms, touches nothing downstream

@app.get("/readyz")                 # readiness: deps with short timeouts. Pull from LB if this fails.
def readyz():
    checks = {"db": ping(db, timeout=0.2), "cache": ping(redis, timeout=0.1)}
    ok = all(checks.values())
    return JSONResponse(
        {"status": "ready" if ok else "degraded", "checks": checks},
        status_code=200 if ok else 503,   # 503 so the LB pulls this instance
    )

Do not put a third-party API call in readiness — a payment provider's outage shouldn't pull all your instances from rotation and take you fully down. Degrade that path in code (../error-handling/SKILL.md), don't fail-closed on it here. Go and FastAPI handler examples are in references/tool-setup.md.

What to actually monitor

The golden checklist. Monitor the symptom users feel, not just that one URL returns 200.

• Availability — the critical endpoint(s) reachable, from multiple regions.
• Latency p95/p99 — averages hide the tail; alert on p95/p99 against a budget, not the mean.
• Error rate — the 5xx-to-total ratio, because raw 5xx count says nothing without traffic volume.
• SSL cert expiry — alert at 14 days; an expired cert is a full outage that no app metric catches.
• Domain expiry — alert at 30 days; a lapsed domain is the most embarrassing avoidable outage.
• The critical user journey — a synthetic check that does login → core action → result, NOT just GET /. The homepage can be 200 while checkout is broken.

The homepage being up tells you almost nothing. Probe the path that makes you money.

Alerts that don't cry wolf

Alert on symptoms, not causes. Page on "checkout error rate >2% for 5 min" (user impact), not on "CPU >80%" (a cause that may be harmless and self-resolving). High CPU with happy users is a dashboard line, not a 3am page.

Use multi-window, multi-burn-rate for SLO alerts (Google SRE workbook). Burn rate = how fast you're spending the monthly error budget. Require a long and a short window to both fire — the long window says "this is real," the short window says "this is still happening," and together they kill false positives from a single spike.

Routing: critical → page, warning → ticket, info → dashboard/review. Dedupe and group related alerts into one incident (10 hosts failing the same check = one page, not ten). Set maintenance windows so planned deploys don't page anyone. The full burn-rate math and a copy-paste Prometheus-style rule with a runbook_url annotation are in references/burn-rate-and-oncall.md.

On-call basics

• Rotation: weekly, with a primary and a secondary. One person can't be the single point of failure for the system that catches single points of failure.
• Escalation policy: page primary → if no ack in 5–10 min, page secondary → then the manager. No-ack must always hop; an unacked page is a dropped page.
• Runbook per alert: every alert links to a runbook with five fields — Symptom, Impact, First 3 checks, Mitigation, How to escalate. The person paged at 3am should not have to think from scratch. Template in references/burn-rate-and-oncall.md.
• Status page + comms: a public status page (Kuma and Better Stack include one) so customers self-serve "is it you or me," cutting inbound during an incident.
• Do NOT start a new on-call on Opsgenie — Atlassian is retiring it (EOL 2027-04-05; new sales ended 2025-06-04). Grafana OnCall OSS was also deprecated (folded into Grafana Cloud IRM). For a new setup use PagerDuty, incident.io, Better Stack, or Grafana Cloud IRM.

Verify it works

An untested alert is not an alert. Before you call monitoring "done," prove the wire end-to-end:

1. Trigger a real synthetic failure (stop the service, or point a monitor at a forced-503 route).
2. Confirm the page actually lands on a phone — not just "the rule exists in the UI."
3. Let the ack timeout lapse and confirm escalation hops to the secondary.
4. Restore, and confirm the resolve/all-clear notification fires too.

scripts/verify.sh enforces the structural half of this on your config: liveness split from readiness, at least one symptom/burn-rate alert with two windows, a runbook reference on every alert, and a banlist for Opsgenie-as-new-setup and homepage-only monitors. Run it in CI so config drift can't silently re-introduce a noisy or untested setup.

Anti-patterns

References

• references/tool-setup.md — Uptime Kuma docker-compose (rootless image, volume, port), HTTP + push-heartbeat + SSL/domain-expiry monitors, a synthetic multi-step journey, notification wiring (ntfy / Slack webhook / PagerDuty integration key), a UptimeRobot/Better Stack monitor JSON shape, and Go + FastAPI health-endpoint handlers.
• references/burn-rate-and-oncall.md — full multi-window multi-burn-rate math, a copy-paste Prometheus-style alert rule with runbook_url, severity mapping, a sample escalation-policy YAML, and a fill-in runbook template.

Related: ../observability/SKILL.md (what the app emits), ../deployment/SKILL.md (release-gating healthcheck + rollback), ../error-handling/SKILL.md (degrade in code), ../domains-dns/SKILL.md (provision certs/DNS), ../scaling/SKILL.md (survive the load monitoring detected).