K8S Observability And Operations Readiness

Kubernetes Observability and Operations Readiness

When to use

Invoke when making a Kubernetes workload or namespace observable and operable for production — scrape wiring, log shipping, tracing, SLO alerts, and runbook inputs — or when auditing an inherited workload that is a black box in incidents.

Do not use for: base manifest authoring (use k8s-workload-packaging-and-manifest); NetworkPolicy/RBAC (use k8s-network-and-identity-policy); autoscaler tuning that consumes these metrics (use k8s-scaling-and-resilience-topology); image hardening and admission (use k8s-supply-chain-and-image-hardening); choosing the observability backend/substrate (decided upstream in infrastructure-platform.md / architecture/operations); cluster provisioning and control-plane audit enablement (out of Family G — owned by the cloud platform stack and Terraform; this skill wires workload/namespace-level collection).

Inputs

Required:

• A workload manifest set from k8s-workload-packaging-and-manifest (the Service/pod this skill makes observable; it exposes a metrics port and stdout logs).
• Approved architecture/operations decisions on the observability substrate, alerting destinations, and runbook hooks, or explicit confirmation they are intentionally deferred.

Optional:

• Approved architecture/reliability SLOs and error-budget policy (drive alert thresholds).
• The workload tier from architecture-schema (drives alert severity, retention, sampling).
• The observability stack in use (Prometheus/Grafana/Loki/Tempo or cloud-native) — consumed, not chosen here.
• The trace propagation format (W3C tracecontext / B3) and sampling target.
• Known operational failure modes for this workload (informs runbook inputs).

Operating rules

• Never generate vanity telemetry. A metric nobody alerts on, a log nobody can query, and a trace with no propagation are not observability — they are cost.
• Consume architecture/operations and architecture/reliability; do not invent decisions. The observability substrate, alert destinations, SLOs, and runbook hooks are decisions made upstream. If a needed decision is missing, pause and raise an ADR candidate rather than guessing.
• The observability backend is consumed, not chosen. ServiceMonitor/log-shipper/collector targets the substrate named upstream. Selecting Prometheus-vs-Datadog here is out of scope — a missing substrate is an ADR candidate, not a default.
• Every workload exposes the three signals. Metrics via a scraped endpoint (ServiceMonitor/PodMonitor, not just bare annotations where the stack supports CRDs), logs to stdout shipped structured, traces emitted with context propagation. A workload missing one signal is not operations-ready.
• Alerts are tied to SLOs and are actionable. Each alert rule maps to an SLO or a known failure mode, has a severity from the tier, and names the runbook. A threshold with no runbook and no SLO is rejected.
• Signal coverage includes the platform, not just the app. kube-state-metrics (object state), cAdvisor (container resource), and node-pressure signals are wired so eviction storms and node pressure are visible — not only app latency.
• Logs are structured and PII-safe by default. Shipped logs are structured (JSON), correlate to traces (trace/span IDs), and do not carry secrets or PII; redaction is in the pipeline, not assumed downstream.
• Audit-log collection is scoped to what this layer owns. Workload/namespace-relevant audit events (RBAC denials, policy-deny from the network archetype, admission rejections) are collected; enabling the API-server audit policy is a control-plane concern handed off, not done here.
• Runbook inputs are concrete, not prose. For pod-eviction storms, ImagePullBackOff sprees, and node-pressure incidents: the signal that fires, the query to confirm, the first diagnostic step, and the escalation — structured for the operations runbook, not a paragraph.
• This skill owns observability + operations readiness wiring. Base manifests, network/identity, autoscaler tuning, image hardening, and the substrate decision are named handoffs, not implemented here.
• A workload whose alerts have not been test-fired and whose runbook inputs have not been dry-run is not operations-ready.

Output contract

The generated observability and operations wiring MUST conform to:

• observability-standards — metrics scraped via ServiceMonitor/PodMonitor, structured stdout logs shipped, traces with context propagation, SLO-tied alerts, trace/log correlation.
• deployment-standards — observability wiring shipped as reproducible manifests, not click-ops dashboards; env-agnostic.
• naming-conventions — ServiceMonitor/alert/dashboard object names kebab-case, suffixed by kind when ambiguous.
• architecture-schema — tier classification drives alert severity, retention, and sampling rate.

Upstream contract: architecture/operations is the source of truth for the observability substrate, alert destinations, and runbook hooks; architecture/reliability is the source of truth for SLOs and error budgets. If a needed decision is missing, pause and raise an ADR candidate. Base manifests, network/identity, autoscaler tuning, image hardening, and substrate selection are named handoffs.

Progressive references

• Read references/k8s-observability-operations-playbook.md when wiring any owned signal or checking the anti-pattern list.
• Read references/k8s-observability-operations-quality-rubric.md before declaring the workload operations-ready.
• Use assets/k8s-observability-operations.template.md as the ServiceMonitor/log/trace/alert/runbook pattern reference.

Process

1. Gather context: load architecture/operations (substrate, alert destinations, runbook hooks) and architecture/reliability (SLOs, error budgets). Resolve the workload tier from architecture-schema. Confirm the workload exposes a metrics port and stdout logs. If a needed decision is missing, raise an ADR candidate before proceeding.
2. Wire metrics: a ServiceMonitor/PodMonitor (or the substrate's equivalent) scraping the workload endpoint; confirm kube-state-metrics and cAdvisor coverage so object state and container resource are visible.
3. Wire log shipping: the upstream-named shipper (Fluent Bit / Vector / cloud-native) collecting stdout, parsing to structured JSON, injecting trace/span IDs, redacting PII/secrets in-pipeline, to the named log backend.
4. Wire tracing: the OpenTelemetry collector (or substrate equivalent), context propagation format, and the tier-correct sampling rate; confirm trace↔log correlation IDs line up.
5. Author SLO alert rules: each tied to an SLO or a known failure mode, severity from the tier, destination from the upstream alert config, and a named runbook — no orphan thresholds.
6. Wire platform-signal coverage: node-pressure, eviction, and ImagePullBackOff signals so cluster-level incidents are visible, not only app metrics.
7. Scope audit-log collection: collect workload/namespace RBAC-denial, network-policy-deny, and admission-rejection events; hand off API-server audit-policy enablement to the control-plane owner.
8. Produce runbook inputs: for pod-eviction storms, ImagePullBackOff sprees, and node-pressure — the firing signal, the confirming query, the first diagnostic step, the escalation — structured for the operations runbook.
9. Verify: test-fire each alert (synthetic breach) and confirm it routes to the destination; dry-run each runbook input against the live signals; document any check that cannot run in the environment.
10. Emit the wiring under k8s/observability/ plus operations-readiness.md (signal inventory, alert→SLO→runbook map, sampling/retention) and the named handoff list. Validate against observability-, deployment-standards, naming-conventions, and architecture-schema. Revise until all pass or the gap is documented.

Outputs

Required:

• ServiceMonitor/PodMonitor scrape wiring with kube-state-metrics and cAdvisor coverage.
• Log-shipping configuration (structured, trace-correlated, PII/secret-redacted) to the named backend.
• OpenTelemetry collector wiring with context propagation and tier-correct sampling.
• SLO alert rules, each tied to an SLO/failure mode, severity, destination, and a named runbook.
• Platform-signal coverage (node-pressure, eviction, ImagePullBackOff).
• Scoped audit-log collection (with control-plane audit-policy handoff marked).
• Concrete runbook inputs for the three named incident classes.
• operations-readiness.md (signal inventory, alert→SLO→runbook map, sampling/retention) and the named handoff list.

Output rules:

• Functional, reproducible wiring as manifests — not click-ops, not placeholder.
• No orphan alerts (every alert maps to SLO/failure mode + runbook); no PII/secrets in shipped logs.
• The substrate is consumed, not chosen; substrate selection is an upstream handoff.
• Base manifests, network/identity, autoscaler tuning, and image hardening are handoffs, not implemented here.

Quality checks

• The workload exposes all three signals: scraped metrics, structured stdout logs, propagated traces.
• Metrics are wired via ServiceMonitor/PodMonitor (or substrate equivalent); kube-state-metrics and cAdvisor coverage is present.
• Shipped logs are structured, carry trace/span correlation IDs, and are PII/secret-redacted in-pipeline.
• Traces use the upstream propagation format with tier-correct sampling; trace↔log IDs correlate.
• Every alert rule maps to an SLO or known failure mode, has tier-correct severity, a destination, and a named runbook — no orphans.
• Node-pressure, eviction, and ImagePullBackOff signals are covered (platform, not only app).
• Audit-log collection is scoped to workload/namespace events; API-server audit-policy enablement is handed off.
• Concrete runbook inputs exist for pod-eviction storms, ImagePullBackOff sprees, and node-pressure (signal → query → first step → escalation).
• Each alert was test-fired to its destination and each runbook input dry-run, or the gap is documented.
• The observability substrate is the one named upstream (or an ADR candidate is raised); base manifest/network/autoscaler/hardening are named handoffs.

References

• Upstream: architecture/operations, architecture/reliability.
• Builds on: k8s-workload-packaging-and-manifest (exposes the metrics port and stdout logs this skill wires).
• Holistic review pass: k8s-deploy-manifest-review (omnibus).
• Related Family G archetype skills: k8s-network-and-identity-policy (produces policy-deny audit events collected here), k8s-scaling-and-resilience-topology (consumes the metrics wired here), k8s-supply-chain-and-image-hardening.
• Standards: observability-standards, deployment-standards, naming-conventions, architecture-schema.