kimtth Goal: be able to answer new questions about production without redeploying.
Use for:
- instrumenting services for debugging and alerting
- diagnosing latency, errors, and saturation
- designing what to measure and emit
Workflow:
- Cover the three pillars: logs, metrics, and traces.
- Emit structured logs with correlation/trace IDs.
- Track the golden signals: latency, traffic, errors, saturation.
- Propagate trace context across service boundaries.
- Alert on symptoms users feel, not noisy internals.
- Verify you can trace one request end to end.
What to capture:
- request-level traces with timing per span
- RED/USE metrics for services and resources
- structured, queryable logs with context
- meaningful, low-noise alerts on SLOs
Rules:
- correlate logs, metrics, and traces with shared IDs
- alert on user-facing symptoms, not every blip
- instrument boundaries and dependencies first
- never log secrets or PII in telemetry