Monitoring and Alerting Principles
Covers what agent implements in code. Org concerns (SLO defs, on-call, escalation) out of scope.
Health Checks
/health (Liveness): 200 if alive. No dependency checks. Orchestrator uses to restart./ready (Readiness): checks all deps (DB, cache, MQ). 503 if unavailable. LB uses to route.
Rules: fast (<1s), no side effects, separate liveness from readiness.
Metrics — RED/USE
RED (services): Rate (req/s), Errors (count/rate), Duration (histogram, not avg).
USE (resources): Utilization, Saturation (queued), Errors.
Rules: counters (monotonic), gauges (up/down), histograms (distributions). Consistent labels (service, method, status_code). No high-cardinality labels (no user IDs).
Error Tracking
- Capture unhandled exceptions + stack traces
- Include context (userId, requestId, correlationId)
- Group by root cause, not instance
- Severity by user impact
Graceful Degradation
- Circuit breakers for external deps
- Fallbacks for non-critical (cached data, reduced UI)
- Timeouts on all external calls
- Retry with exponential backoff + jitter
Tool-agnostic: Datadog, LGTM, Sentry, New Relic, CloudWatch — same code patterns.
Checklist
Related
- Logging Mandate GEMINI.md § Logging and Observability Mandate
- Logging Principles @.gemini/skills/logging-and-observability-principles/SKILL.md
- Error Handling GEMINI.md § Error Handling Principles
- Resources @.gemini/skills/resources-and-memory-management/SKILL.md
- Concurrency @.gemini/skills/concurrency-and-threading-principles/SKILL.md
1---2name: monitoring-and-alerting-principles3description: Health checks, metrics instrumentation (RED/USE), error tracking, graceful degradation. Prometheus, probes, alert thresholds.4---56## Monitoring and Alerting Principles78> Covers what agent implements in code. Org concerns (SLO defs, on-call, escalation) out of scope.910### Health Checks1112- **`/health` (Liveness):** 200 if alive. No dependency checks. Orchestrator uses to restart.13- **`/ready` (Readiness):** checks all deps (DB, cache, MQ). 503 if unavailable. LB uses to route.1415Rules: fast (<1s), no side effects, separate liveness from readiness.1617### Metrics — RED/USE1819**RED (services):** Rate (req/s), Errors (count/rate), Duration (histogram, not avg).2021**USE (resources):** Utilization, Saturation (queued), Errors.2223Rules: counters (monotonic), gauges (up/down), histograms (distributions). Consistent labels (service, method, status_code). No high-cardinality labels (no user IDs).2425### Error Tracking26- Capture unhandled exceptions + stack traces27- Include context (userId, requestId, correlationId)28- Group by root cause, not instance29- Severity by user impact3031### Graceful Degradation32- Circuit breakers for external deps33- Fallbacks for non-critical (cached data, reduced UI)34- Timeouts on all external calls35- Retry with exponential backoff + jitter3637Tool-agnostic: Datadog, LGTM, Sentry, New Relic, CloudWatch — same code patterns.3839### Checklist40- [ ] /health and /ready endpoints41- [ ] Liveness: no dep checks42- [ ] Readiness: checks all critical deps43- [ ] RED/USE metrics on key ops44- [ ] No high-cardinality labels45- [ ] Unhandled exceptions captured + context46- [ ] Circuit breakers on external deps47- [ ] Timeouts on all external calls4849### Related50- Logging Mandate GEMINI.md § Logging and Observability Mandate51- Logging Principles @.gemini/skills/logging-and-observability-principles/SKILL.md52- Error Handling GEMINI.md § Error Handling Principles53- Resources @.gemini/skills/resources-and-memory-management/SKILL.md54- Concurrency @.gemini/skills/concurrency-and-threading-principles/SKILL.md
irahardianto