Amey-Thakur Infrastructure monitoring
Monitoring exists to answer two questions fast: is the user affected, and where is the problem. Instrument for those, alert only on the first, and every page carries a link to what to do next: anything else is dashboards nobody reads and alarms everybody mutes.
Method
- Instrument the golden signals per service. Latency (distributions, not averages: p50/p95/p99), traffic (request rate), errors (rate and type), saturation (how full the constrained resource is): these four catch most user-facing problems and localize them (see observability, distributed-tracing for the request-level layer). Complement with the USE method for resources (Utilization, Saturation, Errors per CPU/disk/network: the systems-profiling view at fleet scale).
- Separate metrics, logs, and traces by job. Metrics for trends and alerting (cheap, aggregatable); logs for detail and forensics (structured: see structured-logging, log-analysis); traces for request flow across services (see distributed-tracing): correlated by shared IDs so an alert links to the traces links to the logs. Three pillars, one correlation key.
- Alert only on symptoms users feel. Page on: SLO breach (error rate, latency past target), not on every CPU spike or single-host blip (see alerting-design's discipline). Cause-based alerts (disk 80% full) are tickets or auto-remediation, not pages; a pager that fires on non-impacting conditions trains responders to ignore it, and the real one arrives muted.
- Set thresholds from SLOs and error budgets. Define the service level objective (99.9% of requests under 300ms), derive the alerting from budget burn rate (fast burn pages, slow burn tickets: see error-budget-policy), and review against actual variance and seasonality (weekends are not incidents). Static thresholds copied from a blog rot into noise.
- Link every alert to a runbook. The page includes: what broke, the dashboard, and the first diagnostic steps (see runbook-writing); a responder woken at 3am should not also have to remember the architecture. This single practice cuts mean-time-to-resolve more than any dashboard redesign.
- Dashboard for the question, not the metric dump. One overview per service answering "is it healthy" (the golden signals), drill-downs for diagnosis; deployment and config-change markers overlaid so "what changed" is visible (most incidents are a recent change: see rollback-strategy); reviewed in operations forums with actions attached (see dashboard-design, product-metrics' decide-don't-decorate rule).
Boundaries
- Monitoring is knowns-focused; unknown-unknowns need observability's ad-hoc query power (high-cardinality traces and logs you can slice by any dimension after the fact: see observability).
- Alert coverage has diminishing returns and a fatigue cost; fewer, higher-quality pages beat exhaustive ones (see alerting-design). Prune alerts that never actioned.
- Business-metric monitoring (conversion, revenue: see product-metrics) is a related but separate system with different owners; infra-green while business-red is a real and dangerous state to design for.