# Prometheus > "Prometheus in Cloud-Native Engineering - The Prometheus monitoring system" and time series database. - Skill: `paulpas/prometheus` (Agent Skill) - Install (CLI): `npx skillmds@latest add paulpas/prometheus` - Raw SKILL.md: https://api.skillmd.com/api/skills/paulpas/prometheus/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - License: MIT - Author: paulpas (https://skillmd.com/u/paulpas) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/paulpas/prometheus --- related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-fluentd # Prometheus in Cloud-Native Engineering **Category:** alerting **Status:** Active **Stars:** 63,688 **Last Updated:** 2026-04-22 **Primary Language:** Go **Documentation:** [https://prometheus.io/docs/](https://prometheus.io/docs/) --- ## Purpose and Use Cases Prometheus is a core component of the cloud-native ecosystem, serving as a monitoring and alerting toolkit designed for reliability and scalability in dynamic cloud-native environments. ### What Problem Does It Solve? Complex monitoring requirements in dynamic cloud environments with frequent scale changes, microservices, and ephemeral containers. It provides reliable metrics collection, storage, and alerting. ### When to Use This Project Use Prometheus when you need to monitor microservices, require metrics-based alerting, want to integrate with Kubernetes via kube-state-metrics and cAdvisor, or need a scalable, pull-based monitoring system. Consider alternatives if you need native distributed tracing or complex log correlation. ### Key Use Cases - Kubernetes cluster monitoring with kube-state-metrics - Application performance monitoring - Infrastructure metrics collection - Alerting for service degradation - Integration with Grafana for visualization --- ## Architecture Design Patterns ### Core Components - **Prometheus Server**: Main server that scrapes and stores metrics - **PromQL**: Query language for metrics analysis - **Alertmanager**: Alert management and notification routing - **Client Libraries**: Instrumentation libraries for applications - **Exporter**: Components that expose metrics from third-party systems - **Service Discovery**: Automatic discovery of targets for scraping ### Component Interactions 1. **Prometheus → Targets**: Scrapes metrics via HTTP 2. **Service Discovery → Prometheus**: Provides target list 3. **Prometheus → Alertmanager**: Sends alerts 4. **Client → Prometheus API**: Queries and retrieves data 5. **TSDB**: Storage for time-series data 6. **Rules → Alertmanager**: Evaluates alerting rules ### Data Flow Patterns 1. **Scrape Cycle**: Service discovery → Target list → HTTP scrape → Storage → Queryable 2. **Alerting**: Rule evaluation → Triggered alerts → Alertmanager → Notifications 3. **Storage**: Timestamped samples → Chunk creation →Compaction → Block creation 4. **Query Path**: PromQL parsing → AST evaluation → Result generation → API response ### Design Principles - **Pull-based Scraping**: Servers expose metrics, Prometheus polls - **Federation**: Hierarchical data collection - **Time-Series Database**: Optimized for metrics storage - **PromQL**: Powerful query language for metrics - **Multi-dimensional**: Labels for metric identification - **Pushgateway**: Short-lived job support --- ## Integration Approaches ### Integration with Other CNCF Projects - **Kubernetes**: kube-state-metrics, node-exporter, cAdvisor integration - **Grafana**: Visualization and dashboards - **Alertmanager**: Alert management and routing - **OpenTelemetry**: Metrics collection instrumentation - **etcd**: Metrics for cluster monitoring - **CoreDNS**: DNS metrics collection - **Helm**: Prometheus operator for deployment ### API Patterns - **HTTP API**: Query and management endpoints - **Pull Model**: Server scrapes targets - **Metrics Endpoint**: Standard format for metrics exposure - **PromQL API**: Query endpoint with PromQL - **Alertmanager API**: Alert management endpoints - **Service Discovery API**: Dynamic target discovery ### Configuration Patterns - **YAML Configuration**: Server configuration - **Service Discovery**: Dynamic target configuration - **Rule Files**: Alerting and recording rules - **ConfigMap**: Kubernetes configuration - **Environment Variables**: Override configuration ### Extension Mechanisms - **Exporters**: Expose third-party metrics - **Client Libraries**: Instrument applications - **Remote Write**: Send data to storage - **Alertmanager Templates**: Custom alert formats - **Service Discovery**: Custom discovery mechanisms --- ## Common Pitfalls and How to Avoid Them ### Misconfigurations - **Scrape Intervals**: Too frequent or infrequent - **Retention**: Not setting appropriate retention time - **Resource Limits**: Insufficient memory for metrics - **Service Discovery**: Not discovering all targets - **Alert Rules**: Missing or incorrect alert thresholds - **Federation**: Incorrect federation configuration - **Time Series Cardinality**: Too many unique labels ### Performance Issues - **Memory Usage**: High memory for time series data - **Scrape Latency**: Slow target responses - **Query Performance**: Complex PromQL queries - **Storage I/O**: High disk usage - **Series Cardinality**: Too many time series - **Retention Size**: Excessive storage requirements - **Rule Evaluation**: Slow rule processing ### Operational Challenges - **Retention Policy**: Balancing storage and data duration - **High Availability**: Redundancy configuration - **Scaling**: Vertical and horizontal scaling - **Data Migration**: Moving between installations - **Backup Strategy**: Metrics data backup - **Alert Fatigue**: Managing alert noise - **Label Management**: Consistent labeling strategy ### Security Pitfalls - **API Access**: Unrestricted query API - **Target Authentication**: Missing authentication for scraping - **Secrets**: Storing sensitive data in metrics - **External Labels**: Information disclosure - **Federation**: Unrestricted federation endpoints --- ## Coding Practices ### Idiomatic Configuration - **YAML Configuration**: Clear structure - **Environment Variables**: Dynamic configuration - **Service Discovery**: Auto-discover targets - **Rule Files**: Modular alerting rules - **ConfigMap**: Kubernetes integration ### API Usage Patterns - **HTTP API**: Query and management - **PromQL**: Metrics querying - **Alertmanager API**: Alert management - **Service Discovery API**: Target management - **Pushgateway API**: Short-lived jobs ### Observability Best Practices - **Metrics Collection**: Comprehensive metrics - **Alerting Rules**: Clear alert thresholds - **Recording Rules**: Pre-computed metrics - **Service Discovery**: Dynamic target discovery - **High Availability**: Redundant deployments - **Federation**: Hierarchical collection ### Testing Strategies - **Unit Tests**: Rule and expression tests - **Integration Tests**: Service integration - **E2E Tests**: Full monitoring stack - **Benchmark Tests**: Performance measurement - **Compatibility Tests**: Version compatibility ### Development Workflow - **Development**: Prometheus server for testing - **Testing**: promtool for rule validation - **Debugging**: Metrics endpoint, query API - **Deployment**: Operator or Helm - **CI/CD**: Integration with pipeline tools - **Tools**: promtool, Grafana, prombench --- ## Fundamentals ### Essential Concepts - **Time-Series**: Metrics stored with timestamps - **Metrics Types**: Counter, Gauge, Histogram, Summary - **Labels**: Key-value pairs for metric identification - **Metrics Name**: Unique identifier for a metric - **Instance**: Single scraped target - **Job**: Group of instances with the same scraping config - **Alert**: Triggered when threshold exceeded - **Recording Rule**: Pre-computed metric expression - **TSDB**: Time-series database for storage - **Pull Model**: Metrics are scraped, not pushed ### Terminology Glossary - **Scrape**: Pull metrics from target - **Target**: Service being monitored - **Metric**: Measurable value - **Label**: Key-value pair for metric identification - **Timestamp**: Time associated with sample - **Sample**: Single data point - **Series**: All samples of a metric - **Alert**: Triggered threshold - **Rule**: Alerting or recording rule - **TSDB**: Time-series database - **Pushgateway**: Gateway for short-lived jobs ### Data Models and Types - **Metric Samples**: Time-series data points - **Labels**: Key-value pairs for metrics - **Alert**: Alerting rule instance - **Rule Group**: Group of rules - **Series**: Time-series identifier - **Histogram Buckets**: Histogram data - **Summary Quantiles**: Summary data - **Timestamp**: Time in milliseconds - **Value**: Numeric metric value ### Lifecycle Management - **Scrape Lifecycle**: Discover → Scrape → Store - **Alert Lifecycle**: Evaluate → Trigger → Send - **Rule Lifecycle**: Create → Evaluate → Update - **Storage Lifecycle**: Append → Compaction → Block - **Retention Lifecycle**: Expiration → Deletion - **Service Discovery Lifecycle**: Discover → Update → Remove ### State Management - **TSDB**: Time-series data storage - **Series Metadata**: Metric labels - **Rule State**: Alert and rule evaluation - **Target State**: Scraped targets - **Storage Blocks**: Compacted data blocks - **Checkpoint**: WAL checkpoint for recovery --- ## Scaling and Deployment Patterns ### Horizontal Scaling - **Sharding**: Horizontal sharding of metrics - **Federation**: Hierarchical federation - **Rule Sharding**: Distributing alerting rules - **Storage Sharding**: Time-based storage separation ### High Availability - **Two-Prometheus HA**: Mutual scraping - **Remote Write**: Redundant storage endpoints - **Federation HA**: Redundant federation endpoints - **Storage HA**: Clustered storage solutions ### Production Deployments - **Architecture**: HA pair with remote storage - **Storage Configuration**: Long-term storage integration - **Alerting Setup**: Multiple alert notification channels - **Service Discovery**: Dynamic target discovery - **Security**: Authentication and authorization - **Resource Limits**: Appropriate memory and CPU limits - **Backup Strategy**: TSDB snapshot and remote storage ### Upgrade Strategies - **Version Skew**: Compatible versions - **Storage Compatibility**: TSDB compatibility - **Configuration Changes**: Configuration drift detection - **Rolling Update**: Zero-downtime upgrade - **Data Migration**: Storage migration if needed ### Resource Management - **Memory Configuration**: TSDB memory usage - **Storage Quota**: Disk space management - **Retention Period**: Data retention configuration - **Scrape Interval**: Resource vs. granularity trade-off --- ## Additional Resources - **Official Documentation:** [https://prometheus.io/docs/](https://prometheus.io/docs/) - **GitHub Repository:** [github.com/prometheus/prometheus](https://github.com/prometheus/prometheus) - **CNCF Project Page:** [cncf.io/projects/prometheus/](https://www.cncf.io/projects/prometheus/) - **Community:** Check the GitHub repository for community channels - **Versioning:** Refer to project's release notes for version-specific features --- ## Troubleshooting ### Common Issues 1. **Deployment Failures** - Check pod logs for errors - Verify configuration values - Ensure network connectivity 2. **Performance Issues** - Monitor resource usage - Adjust resource limits - Check for bottlenecks 3. **Configuration Errors** - Validate YAML syntax - Check required fields - Verify environment-specific settings 4. **Integration Problems** - Verify API compatibility - Check dependency versions - Review integration documentation ### Getting Help - Check official documentation - Search GitHub issues - Join community channels - Review logs and metrics *Content generated automatically. Verify against official documentation before production use.* ## Tutorial This tutorial will guide you through installing, configuring, and using Prometheus for monitoring Kubernetes applications. ### Prerequisites Before beginning, ensure you have: - A running Kubernetes cluster (minikube, kind, EKS, GKE, AKS) - `kubectl` configured with cluster-admin permissions - `helm` CLI (for Helm installation method) - Basic understanding of monitoring concepts (metrics, alerts, dashboards) Verify your setup: ```bash # Check cluster connectivity kubectl cluster-info # Verify kubectl configuration kubectl get nodes # Verify access to kube-system namespace kubectl get pods -n kube-system ``` --- ### 1. Installation Prometheus can be installed using various methods including kubectl, Helm, or using the Prometheus Operator. #### Method 1: Using kubectl with kube-prometheus-stack (Recommended) ```bash # Create monitoring namespace kubectl create namespace monitoring # Download and apply Prometheus Operator manifests kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/bundle.yaml # Verify installation kubectl get pods -n monitoring # Or use kube-prometheus-stack for a complete installation kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/kube-prometheus-crd.yaml # Wait for CRDs to be created kubectl wait --for=condition=Established --all=CustomResourceDefinition --timeout=30s # Apply kube-prometheus-stack kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/ ``` #### Method 2: Using Helm (Recommended for most users) ```bash # Add the Prometheus Community Helm repository helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo add prometheus-stack https://prometheus-community.github.io/helm-charts helm repo update # Install kube-prometheus-stack (includes Prometheus, Grafana, Alertmanager) helm install prometheus prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace \ --wait # Verify installation kubectl get pods -n monitoring helm list -n monitoring ``` #### Method 3: Using Helm with Custom Values ```bash # Create custom values file cat > prometheus-values.yaml << EOF prometheus: prometheusSpec: retention: 30d replicas: 2 resources: requests: memory: 400Mi cpu: 100m limits: memory: 2Gi cpu: 1000m storageSpec: volumeClaimTemplate: spec: storageClassName: standard accessModes: ["ReadWriteOnce"] resources: requests: storage: 50Gi alertmanager: alertmanagerSpec: replicas: 3 resources: requests: memory: 100Mi cpu: 50m grafana: enabled: true adminPassword: admin EOF # Install with custom configuration helm install prometheus prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace \ -f prometheus-values.yaml \ --wait ``` #### Method 4: Using kubectl with vanilla Prometheus ```bash # Create namespace kubectl create namespace monitoring # Deploy Prometheus without Operator kubectl apply -n monitoring -f - < prometheus-scrape-config.yaml << EOF apiVersion: v1 kind: ConfigMap metadata: name: prometheus-additional-configs namespace: monitoring data: additional-scrape-configs.yaml: | - job_name: 'myapp' static_configs: - targets: ['myapp-service:8080'] metrics_path: /metrics EOF kubectl apply -f prometheus-scrape-config.yaml ``` #### Configuring Alerting Rules ```yaml # Create alerting rules cat > prometheus-alert-rules.yaml << EOF apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: application-alerts namespace: monitoring labels: prometheus: kube-prometheus role: alert-rules spec: groups: - name: application.rules rules: - alert: HighErrorRate expr: | sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05 for: 5m labels: severity: critical annotations: summary: "High error rate detected" description: "Error rate is {{ $value | humanizePercentage }}" - alert: ServiceDown expr: up == 0 for: 2m labels: severity: critical annotations: summary: "Service {{ $labels.instance }} is down" description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes." EOF kubectl apply -f prometheus-alert-rules.yaml ``` #### Configuring Service Monitors ```yaml # ServiceMonitor to automatically discover and scrape services cat > service-monitor.yaml << EOF apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: myapp-monitor namespace: production labels: release: prometheus spec: selector: matchLabels: app: myapp namespaceSelector: matchNames: - production endpoints: - port: metrics interval: 15s path: /metrics scheme: http EOF kubectl apply -f service-monitor.yaml ``` --- related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-fluentd ### 3. Usage Examples #### Querying Metrics with PromQL ```bash # View all metrics curl http://localhost:9090/api/v1/label/__name__/values # Query CPU usage curl -s 'http://localhost:9090/api/v1/query?query=node_cpu_seconds_total' | jq . # Query memory usage curl -s 'http://localhost:9090/api/v1/query?query=node_memory_MemAvailable_bytes' | jq . # Query pod CPU usage curl -s 'http://localhost:9090/api/v1/query?query=sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))' | jq . ``` #### Using PromQL in Prometheus UI ```promql # CPU usage percentage 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) # Memory usage percentage (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 # HTTP error rate sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100 # Request latency percentiles histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)) # Up time percentage avg(rate(up[5m])) by (instance) * 100 ``` #### Creating Grafana Dashboards ```bash # Access Grafana kubectl -n monitoring port-forward svc/prometheus-grafana 3000:80 # Or get admin password kubectl -n monitoring get secret prometheus-grafana \ -o jsonpath="{.data.admin-password}" | base64 -d; echo # Import dashboard via API curl -X POST http://localhost:3000/api/dashboards/db \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d @dashboard.json ``` --- ### 4. Common Operations #### Monitoring Prometheus Health ```bash # Check Prometheus status kubectl get pods -n monitoring -l app=kube-prometheus-prometheus # View Prometheus logs kubectl logs -n monitoring prometheus-kube-prometheus-prometheus-0 -c prometheus # Check Prometheus configuration kubectl -n monitoring get configmap prometheus-kube-prometheus-prometheus -o yaml # View Prometheus targets curl http://localhost:9090/api/v1/targets | jq . # View active alerts curl http://localhost:9090/api/v1/alerts | jq . # View rule groups curl http://localhost:9090/api/v1/rules | jq . ``` #### Managing Retention and Storage ```bash # Scale storage (for StatefulSet deployments) kubectl -n monitoring scale statefulset prometheus-kube-prometheus-prometheus --replicas=3 # Check storage usage kubectl -n monitoring exec prometheus-kube-prometheus-prometheus-0 -- df -h /prometheus # Extend retention period kubectl -n monitoring patch prometheus prometheus-kube-prometheus-prometheus \ --type='merge' -p '{"spec":{"retention":"90d"}}' ``` #### Backing Up and Restoring ```bash # Create backup of Prometheus data kubectl -n monitoring exec prometheus-kube-prometheus-prometheus-0 \ -- mkdir -p /backup kubectl -n monitoring exec prometheus-kube-prometheus-prometheus-0 \ -- cp -r /prometheus/* /backup/ # Export backup kubectl -n monitoring cp monitoring/prometheus-kube-prometheus-prometheus-0:/backup \ ./prometheus-backup --container=prometheus ``` #### Upgrading Prometheus ```bash # Using Helm helm upgrade prometheus prometheus-community/kube-prometheus-stack \ --namespace monitoring \ -f prometheus-values.yaml # Using kubectl (Operator) kubectl -n monitoring apply -f manifests/ # Rollback if needed helm rollback prometheus -n monitoring ``` --- ### 5. Best Practices #### Configuration Best Practices 1. **Set Appropriate Retention**: Balance between storage costs and historical data needs 2. **Configure Scrape Intervals**: Shorter intervals for critical metrics, longer for infrequent metrics 3. **Use Relabeling**: Configure relabeling to reduce metric cardinality 4. **Configure Alertmanagers**: Set up HA Alertmanager configuration 5. **Set Resource Limits**: Configure CPU and memory limits for Prometheus 6. **Enable Remote Write**: Configure remote write for long-term storage 7. **Use Recording Rules**: Pre-compute expensive queries 8. **Configure Federation**: Set up federation for multi-cluster monitoring #### Query Best Practices 1. **Use Aggregation**: Always aggregate metrics when possible 2. **Set Appropriate Time Ranges**: Use appropriate duration windows for queries 3. **Avoid High Cardinality**: Filter metrics by labels to reduce cardinality 4. **Use Rate vs Counter**: Use rate() for counters, instant for gauges 5. **Test Queries**: Test queries in Prometheus UI before dashboards 6. **Use Vector Matching**: Understand instant vs range vectors 7. **Limit Results**: Use limit() for large result sets 8. **Cache Results**: Use recording rules for complex queries #### Security Best Practices 1. **Enable Authentication**: Configure basic auth or OAuth for Prometheus UI 2. **Use Network Policies**: Restrict access to Prometheus and Alertmanager 3. **Configure RBAC**: Grant minimal required permissions 4. **Encrypt Traffic**: Use TLS for all communications 5. **Scrape interval**: Configure appropriate scrape intervals to avoid overwhelming targets 6. **Secret Management**: Store sensitive configuration in secrets 7. **Audit Logging**: Enable audit logging for security analysis 8. **Image Verification**: Use verified Prometheus images from trusted sources #### Alerting Best Practices 1. **Set Appropriate Thresholds**: Balance between false positives and missed alerts 2. **Use For Duration**: Include for clause to prevent flapping alerts 3. **Group Related Alerts**: Use alert groups for related metrics 4. **Set Severity Levels**: Use warning and critical severity levels 5. **Downtime Notification**: Configure silence rules for maintenance 6. **Escalation Policies**: Set up alert escalation paths 7. **Alert Clustering**: Use Alertmanager for alert clustering 8. **Test Alerts**: Regularly test alerting rules *Content generated automatically. Verify against official documentation before production use.* ## Examples ### Prometheus Configuration with Kubernetes Service Discovery ```yaml # prometheus.yaml global: scrape_interval: 15s evaluation_interval: 15s external_labels: cluster: production environment: prod alerting: alertmanagers: - static_configs: - targets: - alertmanager:9093 rule_files: - /etc/prometheus/rules/*.yml scrape_configs: # Kubernetes node metrics - job_name: 'kubernetes-nodes' kubernetes_sd_configs: - role: node scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) # Kubernetes pods - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ # Kubernetes services - job_name: 'kubernetes-services' kubernetes_sd_configs: - role: service relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true ``` ### Alerting Rules for Application Health ```yaml # rules/alerting.rules.yml groups: - name: application-alerts rules: - alert: HighErrorRate expr: | sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05 for: 5m labels: severity: critical annotations: summary: "High error rate detected" description: "Error rate is {{ $value | humanizePercentage }} for the last 5 minutes." - alert: ServiceDown expr: up{job="kubernetes-pods"} == 0 for: 2m labels: severity: critical annotations: summary: "Service is down" description: "Service {{ $labels.pod }} in namespace {{ $labels.namespace }} has been down for more than 2 minutes." - alert: HighLatency expr: | histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service) ) > 1 for: 10m labels: severity: warning annotations: summary: "High latency detected" description: "99th percentile latency is above 1 second for service {{ $labels.service }}." - name: infrastructure-alerts rules: - alert: HighMemoryUsage expr: | (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes > 0.8 for: 5m labels: severity: warning annotations: summary: "High memory usage on node" description: "Node {{ $labels.instance }} is using {{ $value | humanizePercentage }} of memory." - alert: HighCPUUsage expr: | 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80 for: 5m labels: severity: warning annotations: summary: "High CPU usage on node" description: "Node {{ $labels.instance }} is using {{ $value | humanizePercentage }} of CPU." ``` ### Service Monitor for Custom Application ```yaml # ServiceMonitor for Prometheus Operator apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: myapp-monitor namespace: monitoring labels: release: prometheus spec: selector: matchLabels: app: myapp namespaceSelector: matchNames: - production endpoints: - port: http interval: 15s path: /metrics scheme: http scrapeTimeout: 10s honorLabels: true metricRelabelings: - sourceLabels: [__name__] regex: 'go_.*' action: drop ``` --- ## When to Use Use this skill when: - **Integrating a CNCF project into Kubernetes infrastructure** — You need to configure, deploy, or troubleshoot a cloud-native tool within a cluster - **Designing cloud-native architecture** — You are selecting and integrating CNCF tools to solve specific infrastructure challenges - **Resolving operational issues** — A CNCF component is misbehaving, underperforming, or needs configuration changes --- ## Core Workflow 1. **Assess Requirements** — Understand the use case, scale, integration needs, and existing infrastructure. **Checkpoint:** Document requirements, constraints, and success criteria. 2. **Design Architecture** — Plan component interactions, data flow, and deployment strategy using cloud-native best practices. **Checkpoint:** Verify the architecture addresses all requirements and follows CNCF conventions. 3. **Implement & Configure** — Create manifests, configurations, and deployment scripts. Include resource limits, health checks, and observability hooks. **Checkpoint:** Validate all YAML against schema and test in a staging environment. 4. **Deploy & Monitor** — Apply manifests to the cluster, verify component health, and confirm observability is working. **Checkpoint:** Confirm all pods/services are running, probes passing, and metrics/alerts configured. --- ## Constraints ### MUST DO - Include at least one complete working YAML manifest example - Note when content is auto-generated vs. manually verified - Reference relevant CNCF project documentation ### MUST NOT DO - Deploy manifests without testing in a staging environment first - Use deprecated API versions (e.g., apps/v1beta1) - Omit resource limits and requests in Kubernetes manifests --- ## Live References > **These links are resolved at load time** — content is fetched and inlined to provide the latest documentation. - [Prometheus Documentation](https://prometheus.io/docs/introduction/overview/) - [PromQL Reference](https://prometheus.io/docs/prometheus/latest/querying/basics/) - [Prometheus Configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) - [Alerting Rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) - [Service Discovery](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) - [Prometheus HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/)