# Monitoring And Alerting Principles

> Apply health check and metrics patterns when implementing readiness/liveness probes, instrumenting RED/USE metrics, integrating error tracking, or adding production observability code. Covers Prometheus metrics, health endpoints, and alert threshold design.

- Skill: `irahardianto/monitoring-and-alerting-principles-3` (Agent Skill)
- Install (CLI): `npx skillmds@latest add irahardianto/monitoring-and-alerting-principles-3`
- Raw SKILL.md: https://api.skillmd.com/api/skills/irahardianto/monitoring-and-alerting-principles-3/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: irahardianto (https://skillmd.com/u/irahardianto)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/irahardianto/monitoring-and-alerting-principles-3

---


## Monitoring and Alerting Principles

> **Scope:** This rule covers what the agent **implements in code**. Organizational 
> concerns (SLO/SLI definitions, on-call rotation, alert escalation) are out of scope —
> those are team/org decisions, not code generation concerns.

### Health Checks

**Every service must expose health check endpoints:**

- **`/health` (Liveness)** — "Is the process alive?"
  - Returns 200 if the process is running
  - No dependency checks (database, cache, etc.)
  - Used by orchestrators to decide whether to restart

- **`/ready` (Readiness)** — "Can the service accept traffic?"
  - Checks all critical dependencies (database, cache, message queue)
  - Returns 503 if any dependency is unavailable
  - Used by load balancers to route traffic

**Rules:**
- Health checks must be fast (< 1 second)
- Health checks must not have side effects
- Separate liveness from readiness — they serve different purposes

### Metrics Instrumentation

**Instrument code using the RED method for services:**
- **Rate** — requests per second
- **Errors** — error count/rate
- **Duration** — request latency (use histograms, not averages)

**Instrument code using the USE method for resources:**
- **Utilization** — how much of the resource is used
- **Saturation** — how much work is queued
- **Errors** — error count for the resource

**Rules:**
- Use counters for things that only go up (requests, errors)
- Use gauges for things that go up and down (connections, queue depth)
- Use histograms for distributions (latency, response size)
- Label metrics consistently (service, method, status_code)
- Don't create high-cardinality labels (no user IDs as labels)

### Error Tracking Integration

- **Capture unhandled exceptions** with full stack traces
- **Include context** — user ID, request ID, correlation ID
- **Group errors** by root cause, not by instance
- **Set severity levels** based on user impact

### Graceful Degradation

- **Circuit breakers** for external dependencies — stop calling failing services
- **Fallbacks** for non-critical features — serve cached data, show reduced UI
- **Timeouts** on all external calls — never wait indefinitely
- **Retry with backoff** for transient failures — exponential backoff with jitter

### Implementation Notes

This rule is tool-agnostic. Whether the project uses Datadog, LGTM stack, Sentry,
New Relic, or CloudWatch — the code patterns (health checks, metrics, error tracking)
are the same. The specific client library belongs in project-level configuration.

### Monitoring Checklist

- [ ] Health check endpoints implemented (/health and /ready)?
- [ ] Liveness probe has no dependency checks?
- [ ] Readiness probe checks all critical dependencies?
- [ ] Key operations instrumented with RED/USE metrics?
- [ ] No high-cardinality metric labels?
- [ ] Unhandled exceptions captured with context?
- [ ] Circuit breakers on external dependencies?
- [ ] Timeouts on all external calls?

### Related Principles
- Logging and Observability Mandate @.claude/rules/logging-and-observability-mandate.md
- Logging and Observability Principles @.claude/skills/logging-and-observability-principles/SKILL.md
- Error Handling Principles @.claude/rules/error-handling-principles.md
- Resource and Memory Management Principles @.claude/skills/resources-and-memory-management/SKILL.md
- Concurrency and Threading Principles @.claude/skills/concurrency-and-threading-principles/SKILL.md