DevOps Observability
Agent Workflow (MANDATORY)
Before ANY monitoring config or change, spawn in parallel, one spawn_agent call each:
- explore-codebase - find existing
prometheus.yml, rule files, Alertmanager/Loki/Alloy/Collector configs, Helm values, dashboards, ServiceMonitors - research-expert - verify option names for the deployed version at prometheus.io, grafana.com/docs, opentelemetry.io (the OpenTelemetry Collector renames components between 0.x releases)
- mcp__context7__query-docs - Prometheus, Grafana, Loki, OpenTelemetry docs
Then discover read-only (running versions, promtool check, amtool check-config, alloy validate). After writing files, run sniper. Live changes follow the agent's Operations Safety Protocol and observability-safe-ops.md.
Versions (checked 2026-11-10)
Prometheus 3.16.0 is latest; LTS 3.13 is the conservative choice. Alertmanager 0.35.0 has breaking flag/config changes (event recorder outputs need a name). Promtail reached end of life on 2026-03-02: collect logs with Grafana Alloy. The OpenTelemetry Collector renamed otlp to otlp_grpc, otlphttp to otlp_http, and added a queue_batch processor meant to replace batch; read its changelog before every 0.x bump.
Pinned versions:
prometheus: "3.16.0 (2026-11-04); LTS 3.13 (supported to 2027-07-31)"
alertmanager: "0.35.0 (2026-11-09)"
grafana: "13.3.0"
loki: "3.7.9"
grafana-alloy: "1.21.3 (2026-10-28)"
opentelemetry-collector: "0.165.0 (core 1.71.0)"
prometheus-operator: "0.95.1 (2026-11-05); kube-prometheus-stack chart 93.1.1 (main)"
exporters: "node_exporter 1.12.1, blackbox_exporter 0.29.1, cAdvisor 0.60.6, kube-state-metrics 2.20.0"
Overview
| Need | Decision in one line |
|---|---|
| What to instrument | USE for resources, RED for services, golden signals at the edge; plus a blackbox probe of every user-facing URL |
| Metrics store | Prometheus (local TSDB, retention by time AND size flags); long-term via remote write to Mimir/Thanos/VictoriaMetrics only when needed |
| Alerts | Symptom-based, for: + keep_firing_for, severity page/ticket, every alert has runbook_url; routing and inhibition in Alertmanager |
| Kubernetes | kube-prometheus-stack; ServiceMonitor/PodMonitor/PrometheusRule; check the release label selectors and PVCs |
| Logs | Loki + Alloy; few static labels, rest in structured metadata; retention by the compactor |
| Traces | OpenTelemetry SDK -> Collector/Alloy (OTLP 4317/4318) -> Tempo; tail-sample errors and slow traces |
| Dashboards/alerts as code | Grafana provisioning files in Git; metric alerts live in Prometheus, not Grafana |
Critical Rules
- Version first - read the running versions and the config of the target before writing an option; renames and removals are frequent (OpenTelemetry Collector, Alertmanager, Prometheus 3.x)
- Validate offline before any reload -
promtool check config|rules,amtool check-config,alloy validate,otelcol validate; reloading production is a proposal in ACTION / EFFECT / PROOF / WAIT-FOR-GO format (observability-safe-ops.md) - Cardinality is the budget - no unbounded label values (user IDs, trace IDs, pod names as Loki labels); use
sample_limit/label_limit,metric_relabel_configs, recording rules - Destructive verbs are proposed, not run - admin-API series deletion, shrinking retention, Loki delete requests, removing PVCs/buckets, mass silences,
helm uninstall - Persistent storage or it is not monitoring - emptyDir/no volume loses metrics and silences on restart
- Never print secrets - receivers use
*_fileoptions; GrafanasecureJsonData; no tokens in configs or logs - Monitor the monitoring - Watchdog alert to an external dead man's switch, scrape and notification-failure alerts
- Never invent a key or metric name - cite the page for the version; unknown = "not verified"
Architecture
monitoring/
├── prometheus.yml # global, scrape_configs, alerting, (remote_write)
├── rules/ # alerts and recording rules (promtool-checked)
├── alertmanager.yml # route tree, inhibit_rules, receivers (secrets as files)
├── blackbox.yml # probe modules
├── alloy/*.alloy # logs (Loki), host metrics, OTLP gateway
├── otel-collector.yaml # OpenTelemetry Collector gateway (if used instead of Alloy)
├── loki-config.yaml # storage, schema, compactor retention
└── grafana/
├── provisioning/{datasources,dashboards,alerting}/
└── dashboards/*.json
Reference Guide
Concepts
| Topic | Reference | When to Consult |
|---|---|---|
| Signals and method | signals-and-method.md | Deciding what to instrument first |
| Prometheus config | prometheus-config.md | Scrape, SD, relabeling, rules, retention, remote write, cardinality |
| PromQL | promql-essentials.md | Queries, histograms, native histograms, pitfalls |
| Alertmanager | alertmanager.md | Routing, grouping, inhibition, silences, receivers |
| Exporters | exporters.md | node, blackbox, cAdvisor, kube-state-metrics |
| Operator / K8s | prometheus-operator-k8s.md | CRDs, kube-prometheus-stack, missing targets |
| Long-term storage | long-term-storage.md | Thanos, Mimir, VictoriaMetrics pointers |
| Grafana | grafana-as-code.md | Provisioning datasources, dashboards, alerting |
| Loki and Alloy | logs-loki-alloy.md | Collection, labels, LogQL, retention, Promtail migration |
| Other log stacks | logs-other-stacks.md | OpenSearch/Elastic, Vector, Fluent Bit |
| Tracing | tracing-opentelemetry.md | Collector pipelines, OTLP, sampling, Tempo/Jaeger |
| Alert design | alert-design.md | Writing and pruning alerts |
| Safe ops | observability-safe-ops.md | Any reload, restart, deletion, silence |
| Troubleshooting | troubleshooting-playbook.md | Missing data, silent alerts, slow or full stores |
Templates
| Template | When to Use |
|---|---|
| prometheus.yml | Server config with SD, relabeling, blackbox, limits |
| file-sd-nodes.yml | File-based target list |
| blackbox.yml | Probe modules |
| rules-alerts.yml | Symptom alerts with runbook links, Watchdog |
| rules-recording.yml | Recording rules, level:metric:operations |
| alertmanager.yml | Route tree, inhibition, time intervals, receivers |
| servicemonitor.yaml | ServiceMonitor + PodMonitor |
| prometheusrule.yaml | PrometheusRule |
| kube-prometheus-stack-values.yaml | Helm values: retention, PVC, selectors |
| grafana-datasources.yaml | Prometheus, Loki, Tempo, Alertmanager datasources |
| grafana-dashboards-provider.yaml | Dashboard file provider |
| grafana-alerting-routing.yaml | Contact point + notification policy |
| loki-config.yaml | Single-binary Loki with compactor retention |
| alloy-host.alloy | Host metrics + log files |
| alloy-k8s-logs.alloy | Pod logs on Kubernetes |
| alloy-otlp.alloy | OTLP gateway in Alloy |
| otel-collector.yaml | Collector gateway with tail sampling |
| docker-compose-monitoring.yaml | Single-host stack |
Quick Reference
promtool check config prometheus.yml && promtool check rules rules-*.yml
amtool check-config alertmanager.yml
amtool config routes test --config.file=alertmanager.yml --tree severity=page team=sre
alloy validate /etc/alloy/config.alloy
otelcol-contrib validate --config=otel-collector.yaml
curl -s http://prometheus:9090/-/ready ; curl -s http://loki:3100/ready
Best Practices
DO
- Pin image and chart versions; read the release notes of each jump (Alertmanager 0.35, OpenTelemetry 0.x, kube-prometheus-stack majors)
- Alert on user-visible symptoms; link a runbook; group and inhibit in Alertmanager
- Keep dashboards, rules, datasources and Alloy/Collector configs in Git; deliver via GitOps (devops-gitops-packaging)
- Bind monitoring ports to localhost or an authenticated proxy; use TLS between sites
DON'T
- Enable
--web.enable-admin-apior--web.enable-lifecycleon an exposed port - Use
iratein alerts or label metrics/logs with unbounded values - Run Promtail or a second host-metrics collector next to Alloy/node_exporter
- Duplicate the same alert in Prometheus and Grafana
Related skills
- devops-sre-ops
- devops-kubernetes
- devops-k8s-cluster
- devops-gitops-packaging
- devops-docker
- devops-linux
- devops-networking
- devops-storage-backup
- devops-virtualization