Deploying Monitoring Stacks
Overview
Deploy production monitoring stacks (Prometheus + Grafana, Datadog, or Victoria Metrics) with metric collection, custom dashboards, and alerting rules. Configure exporters, scrape targets, recording rules, and notification channels for comprehensive infrastructure and application observability.
Prerequisites
- Target infrastructure identified: Kubernetes cluster, Docker hosts, or bare-metal servers
- Metric endpoints accessible from the monitoring platform (application
/metrics, node exporters)
- Storage backend capacity planned for time-series data (Prometheus TSDB, Thanos, or Cortex for long-term)
- Alert notification channels defined: Slack webhook, PagerDuty integration key, or email SMTP
- Helm 3+ for Kubernetes deployments using kube-prometheus-stack or similar charts
Instructions
- Select the monitoring platform: Prometheus + Grafana for open-source self-hosted, Datadog for managed SaaS, Victoria Metrics for high-cardinality workloads
- Deploy the monitoring stack:
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack or Docker Compose for non-Kubernetes
- Install exporters on monitored systems: node-exporter for host metrics, kube-state-metrics for Kubernetes object states, application-specific exporters
- Configure scrape targets in
prometheus.yml: define job names, scrape intervals, and relabeling rules for service discovery
- Create recording rules for frequently queried aggregations to reduce dashboard query load
- Define alerting rules with meaningful thresholds: high CPU (>80% for 5m), high memory (>90%), error rate (>1%), latency P99 (>500ms)
- Configure Alertmanager with routing, grouping, and notification channels (Slack, PagerDuty, email)
- Build Grafana dashboards: RED metrics (Rate, Errors, Duration) for services, USE metrics (Utilization, Saturation, Errors) for resources
- Set up data retention: configure TSDB retention period (15-30 days local), set up Thanos/Cortex for long-term storage if needed
- Test the full pipeline: trigger a test alert and verify notification delivery
Output
- Helm values file or Docker Compose for the monitoring stack
- Prometheus configuration with scrape targets, recording rules, and alerting rules
- Alertmanager configuration with routing tree and notification receivers
- Grafana dashboard JSON files for infrastructure and application metrics
- Exporter deployment manifests (node-exporter DaemonSet, application ServiceMonitor)
Error Handling
| Error |
Cause |
Solution |
No data points in dashboard |
Scrape target not reachable or metric name wrong |
Check Targets page in Prometheus UI; verify service discovery and metric name |
Too many time series (high cardinality) |
Labels with unbounded values (user IDs, request IDs) |
Remove high-cardinality labels with metric_relabel_configs; use recording rules for aggregation |
Alert condition met but no notification |
Alertmanager routing or receiver misconfigured |
Verify Alertmanager config with amtool check-config; test receiver with amtool silence |
Prometheus OOMKilled |
Insufficient memory for series count |
Increase memory limits; reduce scrape targets or retention; add WAL compression |
Grafana datasource connection failed |
Wrong Prometheus URL or network policy blocking access |
Verify datasource URL in Grafana; check Kubernetes service name and port; review network policies |
Examples
- "Deploy kube-prometheus-stack on Kubernetes with alerts for node CPU > 80%, pod restart count > 5, and API error rate > 1%, sending to Slack."
- "Set up Prometheus + Grafana on Docker Compose for monitoring 10 application servers with node-exporter and custom application metrics."
- "Create Grafana dashboards for the four golden signals (latency, traffic, errors, saturation) for a microservices application."
Resources
Source: jeremylongshore/claude-code-plugins-plus-skills → skills/.curated/deploying-monitoring-stacks/SKILL.md
Also appears in: jeremylongshore/claude-code-plugins-plus-skills/plugins/devops/monitoring-stack-deployer/skills/deploying-monitoring-stacks/SKILL.md
1---2name: deploying-monitoring-stacks3description: 'Monitor use when deploying monitoring stacks including Prometheus, Grafana, and Datadog. Trigger with phrases like "deploy monitoring stack", "setup prometheus", "configure grafana", or "install datadog agent". Generates production-ready configurations with metric collection, visualization dashboards, and alerting rules. '4---56# Deploying Monitoring Stacks78## Overview910Deploy production monitoring stacks (Prometheus + Grafana, Datadog, or Victoria Metrics) with metric collection, custom dashboards, and alerting rules. Configure exporters, scrape targets, recording rules, and notification channels for comprehensive infrastructure and application observability.1112## Prerequisites1314- Target infrastructure identified: Kubernetes cluster, Docker hosts, or bare-metal servers15- Metric endpoints accessible from the monitoring platform (application `/metrics`, node exporters)16- Storage backend capacity planned for time-series data (Prometheus TSDB, Thanos, or Cortex for long-term)17- Alert notification channels defined: Slack webhook, PagerDuty integration key, or email SMTP18- Helm 3+ for Kubernetes deployments using kube-prometheus-stack or similar charts1920## Instructions21221. Select the monitoring platform: Prometheus + Grafana for open-source self-hosted, Datadog for managed SaaS, Victoria Metrics for high-cardinality workloads232. Deploy the monitoring stack: `helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack` or Docker Compose for non-Kubernetes243. Install exporters on monitored systems: node-exporter for host metrics, kube-state-metrics for Kubernetes object states, application-specific exporters254. Configure scrape targets in `prometheus.yml`: define job names, scrape intervals, and relabeling rules for service discovery265. Create recording rules for frequently queried aggregations to reduce dashboard query load276. Define alerting rules with meaningful thresholds: high CPU (>80% for 5m), high memory (>90%), error rate (>1%), latency P99 (>500ms)287. Configure Alertmanager with routing, grouping, and notification channels (Slack, PagerDuty, email)298. Build Grafana dashboards: RED metrics (Rate, Errors, Duration) for services, USE metrics (Utilization, Saturation, Errors) for resources309. Set up data retention: configure TSDB retention period (15-30 days local), set up Thanos/Cortex for long-term storage if needed3110. Test the full pipeline: trigger a test alert and verify notification delivery3233## Output3435- Helm values file or Docker Compose for the monitoring stack36- Prometheus configuration with scrape targets, recording rules, and alerting rules37- Alertmanager configuration with routing tree and notification receivers38- Grafana dashboard JSON files for infrastructure and application metrics39- Exporter deployment manifests (node-exporter DaemonSet, application ServiceMonitor)4041## Error Handling4243| Error | Cause | Solution |44|-------|-------|---------|45| `No data points in dashboard` | Scrape target not reachable or metric name wrong | Check `Targets` page in Prometheus UI; verify service discovery and metric name |46| `Too many time series (high cardinality)` | Labels with unbounded values (user IDs, request IDs) | Remove high-cardinality labels with `metric_relabel_configs`; use recording rules for aggregation |47| `Alert condition met but no notification` | Alertmanager routing or receiver misconfigured | Verify Alertmanager config with `amtool check-config`; test receiver with `amtool silence` |48| `Prometheus OOMKilled` | Insufficient memory for series count | Increase memory limits; reduce scrape targets or retention; add WAL compression |49| `Grafana datasource connection failed` | Wrong Prometheus URL or network policy blocking access | Verify datasource URL in Grafana; check Kubernetes service name and port; review network policies |5051## Examples5253- "Deploy kube-prometheus-stack on Kubernetes with alerts for node CPU > 80%, pod restart count > 5, and API error rate > 1%, sending to Slack."54- "Set up Prometheus + Grafana on Docker Compose for monitoring 10 application servers with node-exporter and custom application metrics."55- "Create Grafana dashboards for the four golden signals (latency, traffic, errors, saturation) for a microservices application."5657## Resources5859- Prometheus documentation: https://prometheus.io/docs/60- Grafana documentation: https://grafana.com/docs/grafana/latest/61- kube-prometheus-stack: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack62- Alerting best practices: https://prometheus.io/docs/practices/alerting/63- Datadog documentation: https://docs.datadoghq.com/6465---6667**Source:** [`jeremylongshore/claude-code-plugins-plus-skills`](https://github.com/jeremylongshore/claude-code-plugins-plus-skills) → `skills/.curated/deploying-monitoring-stacks/SKILL.md`6869**Also appears in:** `jeremylongshore/claude-code-plugins-plus-skills/plugins/devops/monitoring-stack-deployer/skills/deploying-monitoring-stacks/SKILL.md`
thedixitjain