Observability Methods
Reference for the four industry-canonical observability methodologies used by production-analysis and incident-response workflows in this plugin. Pick one named method per problem class to give analysis a vocabulary handle and a defined signal coverage. Cross-reference SLI metrics, error-budget burn, and active alerts against the chosen method's signals.
Four Golden Signals (Google SRE)
Source: SRE Book Ch. 6 — Monitoring Distributed Systems.
User-facing service monitoring. Tracks the four signals a user-visible service must expose:
- Latency — p50 / p95 / p99. Success and failed requests must be tracked separately (failed-fast errors otherwise hide tail latency).
- Traffic — demand on the service, typically requests per second (RPS) or transactions per second.
- Errors — rate of failed requests, broken down by error type (5xx, 4xx, timeouts, business-logic errors).
- Saturation — how "full" the service is — resource headroom before degradation (CPU, memory, queue depth, connection pool, thread pool).
Apply when: the service is user-facing, has a defined SLO, or owns an SLI.
RED Method (Tom Wilkie — microservices)
Source: RED Method.
Request-driven microservices. Three signals per service:
- Rate — requests per second the service is handling.
- Errors — number/percentage of failed requests.
- Duration — distribution of request latency, especially the p99 tail.
Apply when: diagnosing a request-response microservice, especially with many small services where per-service uniformity matters more than resource depth. Strong fit for slow-API and 5xx-spike investigations.
USE Method (Brendan Gregg — resources)
Source: USE Method.
Resource-driven. For every resource (CPU, memory, disk, network, queue), check three signals:
- Utilization — percentage of time the resource was busy.
- Saturation — degree of extra work queued / waiting (run queue, swap, retries).
- Errors — error events for that resource (OOMKilled, disk I/O errors, NIC drops).
Apply when: resource exhaustion is suspected — OOMKilled pods, CPU throttling, disk pressure, node-level pressure, container crashloops driven by limits.
Distributed Tracing
For latency or cross-service failures, a single trace through 5–7 services is the canonical root-cause path:
- Storage: Tempo / Jaeger / Zipkin.
- Instrumentation: OpenTelemetry SDK (vendor-neutral) or vendor agent (Datadog APM, New Relic, Honeycomb Beelines).
- Search: by
service.name, trace.id, or slow-trace heatmap. Honeycomb BubbleUp narrows attributes that correlate with slow traces.
- Span attributes that matter: HTTP route, DB query (statement + duration), external API call (host + latency), retry count, error/exception flag.
Apply when: a request crosses ≥2 services, latency is high but no single service is obviously saturated, error blame is unclear, or aggregate metrics show the symptom but not the cause.
Method-to-Problem Mapping
| Problem |
Method |
Why |
| Slow API |
RED |
Duration p99 tail is the surfaced signal |
| 5xx spike |
Golden Signals |
Errors + Saturation cover cause and capacity together |
| OOMKilled / crashloop |
USE |
Memory/CPU saturation + resource errors |
| Customer-reported latency |
RED + Distributed Tracing |
RED localizes the slow service, tracing finds the slow span |
| Node / disk pressure |
USE |
Resource-axis Utilization + Saturation |
| Cross-service failure (no single hotspot) |
Distributed Tracing |
Single trace reveals the failing hop |
| New service with SLO |
Golden Signals |
Establish baseline for all four user-facing signals |
When this applies
| Workflow |
Apply this knowledge |
/analyze-prod (snapshot phase) |
Pick a named method per problem class before running queries; cross-reference SLI/SLO and alerts against the method's signals |
/analyze-local (Docker logs) |
Apply USE on container resource limits when local services crashloop or are OOMKilled |
/env-analyze (multi-scope) |
Use Golden Signals or RED to frame service-level findings; USE for node/resource layer |
/infra-change (post-apply verify) |
Use Golden Signals to confirm SLO is not regressed by the change |
/bugfix (production-context bugs) |
Use RED + tracing to localize the failing service / span before code-level investigation |
Integration
- Used by:
/analyze-prod, /analyze-local, /env-analyze, /infra-change, /bugfix
- Companion knowledge:
@telemetry-stacks (vendor-specific queries that surface these signals), @cloud-platforms (managed-service metric sources), @deployment-procedures (post-deploy health verification)
- External references:
- Google SRE Book Ch. 6 — Monitoring Distributed Systems (Four Golden Signals)
- Tom Wilkie — RED Method
- Brendan Gregg — USE Method
- OpenTelemetry specification (tracing semantic conventions)
1---2name: observability-methods3description: Use this skill when picking a diagnostic vocabulary for a latency, error-rate, saturation, or crashloop investigation, when authoring or reviewing an analysis workflow that needs the method reference, or when correlating SLI metrics and active alerts against a method's signal set — a knowledge skill of industry-canonical observability methodologies (Four Golden Signals, RED, USE, Distributed Tracing) with method-to-problem mapping that explains which signals each method surfaces and when each method applies. Loaded by `/analyze-prod`, `/analyze-local`, `/env-analyze`, `/infra-change`, and `/bugfix` workflows when production-context diagnosis needs a named methodology.4---56# Observability Methods78Reference for the four industry-canonical observability methodologies used by production-analysis and incident-response workflows in this plugin. Pick one named method per problem class to give analysis a vocabulary handle and a defined signal coverage. Cross-reference SLI metrics, error-budget burn, and active alerts against the chosen method's signals.910## Four Golden Signals (Google SRE)1112Source: [SRE Book Ch. 6 — Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/).1314User-facing service monitoring. Tracks the four signals a user-visible service must expose:1516- **Latency** — p50 / p95 / p99. Success and failed requests must be tracked **separately** (failed-fast errors otherwise hide tail latency).17- **Traffic** — demand on the service, typically requests per second (RPS) or transactions per second.18- **Errors** — rate of failed requests, broken down by error type (5xx, 4xx, timeouts, business-logic errors).19- **Saturation** — how "full" the service is — resource headroom before degradation (CPU, memory, queue depth, connection pool, thread pool).2021Apply when: the service is user-facing, has a defined SLO, or owns an SLI.2223## RED Method (Tom Wilkie — microservices)2425Source: [RED Method](https://thenewstack.io/monitoring-microservices-red-method/).2627Request-driven microservices. Three signals per service:2829- **Rate** — requests per second the service is handling.30- **Errors** — number/percentage of failed requests.31- **Duration** — distribution of request latency, especially the p99 tail.3233Apply when: diagnosing a request-response microservice, especially with many small services where per-service uniformity matters more than resource depth. Strong fit for slow-API and 5xx-spike investigations.3435## USE Method (Brendan Gregg — resources)3637Source: [USE Method](https://brendangregg.com/usemethod.html).3839Resource-driven. For every resource (CPU, memory, disk, network, queue), check three signals:4041- **Utilization** — percentage of time the resource was busy.42- **Saturation** — degree of extra work queued / waiting (run queue, swap, retries).43- **Errors** — error events for that resource (OOMKilled, disk I/O errors, NIC drops).4445Apply when: resource exhaustion is suspected — OOMKilled pods, CPU throttling, disk pressure, node-level pressure, container crashloops driven by limits.4647## Distributed Tracing4849For latency or cross-service failures, a single trace through 5–7 services is the canonical root-cause path:5051- **Storage**: Tempo / Jaeger / Zipkin.52- **Instrumentation**: OpenTelemetry SDK (vendor-neutral) or vendor agent (Datadog APM, New Relic, Honeycomb Beelines).53- **Search**: by `service.name`, `trace.id`, or slow-trace heatmap. Honeycomb BubbleUp narrows attributes that correlate with slow traces.54- **Span attributes that matter**: HTTP route, DB query (statement + duration), external API call (host + latency), retry count, error/exception flag.5556Apply when: a request crosses ≥2 services, latency is high but no single service is obviously saturated, error blame is unclear, or aggregate metrics show the symptom but not the cause.5758## Method-to-Problem Mapping5960| Problem | Method | Why |61|---|---|---|62| Slow API | RED | Duration p99 tail is the surfaced signal |63| 5xx spike | Golden Signals | Errors + Saturation cover cause and capacity together |64| OOMKilled / crashloop | USE | Memory/CPU saturation + resource errors |65| Customer-reported latency | RED + Distributed Tracing | RED localizes the slow service, tracing finds the slow span |66| Node / disk pressure | USE | Resource-axis Utilization + Saturation |67| Cross-service failure (no single hotspot) | Distributed Tracing | Single trace reveals the failing hop |68| New service with SLO | Golden Signals | Establish baseline for all four user-facing signals |6970## When this applies7172| Workflow | Apply this knowledge |73|---|---|74| `/analyze-prod` (snapshot phase) | Pick a named method per problem class before running queries; cross-reference SLI/SLO and alerts against the method's signals |75| `/analyze-local` (Docker logs) | Apply USE on container resource limits when local services crashloop or are OOMKilled |76| `/env-analyze` (multi-scope) | Use Golden Signals or RED to frame service-level findings; USE for node/resource layer |77| `/infra-change` (post-apply verify) | Use Golden Signals to confirm SLO is not regressed by the change |78| `/bugfix` (production-context bugs) | Use RED + tracing to localize the failing service / span before code-level investigation |7980## Integration8182- **Used by**: `/analyze-prod`, `/analyze-local`, `/env-analyze`, `/infra-change`, `/bugfix`83- **Companion knowledge**: `@telemetry-stacks` (vendor-specific queries that surface these signals), `@cloud-platforms` (managed-service metric sources), `@deployment-procedures` (post-deploy health verification)84- **External references**:85 - Google SRE Book Ch. 6 — Monitoring Distributed Systems (Four Golden Signals)86 - Tom Wilkie — RED Method87 - Brendan Gregg — USE Method88 - OpenTelemetry specification (tracing semantic conventions)
avav25