# Observability > Implements Linux system observability with metrics, logs, and performance profiling for proactive infrastructure management across cloud and on-prem environments. - Skill: `paulpas/observability` (Agent Skill) - Install (CLI): `npx skillmds@latest add paulpas/observability` - Raw SKILL.md: https://api.skillmd.com/api/skills/paulpas/observability/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - License: MIT - Author: paulpas (https://skillmd.com/u/paulpas) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/paulpas/observability --- # Linux System Observability Infrastructure engineer implementing comprehensive Linux system observability with metrics collection, log aggregation, performance profiling, and capacity planning for proactive infrastructure management across cloud and on-prem environments. ## TL;DR Checklist - [ ] Deploy metrics collection for CPU, memory, disk I/O, and network throughput - [ ] Configure log collection from systemd journal and application logs - [ ] Set up performance profiling with perf and eBPF for latency investigation - [ ] Define alerting thresholds for resource utilization and error rates - [ ] Implement capacity planning with trend analysis and growth projections - [ ] Validate observability stack with synthetic load and failure injection tests - [ ] Document baseline metrics and alerting configuration for each workload type - [ ] Test alerting delivery and runbook execution for each alert type --- ## When to Use Use this skill when: - **Setting up monitoring from scratch** — You're deploying a new system and need to establish observability baselines - **Investigating performance issues** — A system is experiencing latency, throughput, or reliability problems and needs profiling - **Capacity planning** — You need to project resource needs based on historical growth trends - **Setting up alerting** — You need threshold-based alerts for proactive incident detection - **Post-incident review** — You need to analyze what happened during an incident using collected metrics and logs - **Cloud vs on-prem observability differences** — You need to adapt observability strategy for different deployment environments --- ## When NOT to Use Avoid this skill for: - **Application-level monitoring** — Use application-specific APM tools (OpenTelemetry, Datadog APM) for application metrics - **Network packet analysis** — Use tcpdump or Wireshark for packet-level investigation - **Security forensics** — Use `linux-security` skill for forensic investigation of security incidents - **Real-time debugging** — Use `perf record` and eBPF for low-level kernel debugging, not production monitoring Use `resource-management` when investigating specific resource contention issues (CPU throttling, memory pressure). Use `networking` when network performance is the suspected bottleneck. --- ## Core Workflow ### 1. Deploy Metrics Collection Set up system metrics collection for CPU, memory, disk, and network. ```bash # Install node_exporter (Prometheus metrics) sudo useradd --no-create-home --shell /bin/false node_exporter wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz tar xzf node_exporter-*.tar.gz sudo cp node_exporter-*/node_exporter /usr/local/bin/ rm -rf node_exporter-* # Create systemd service cat > /etc/systemd/system/node_exporter.service << 'NODE' [Unit] Description=Node Exporter After=network.target [Service] Type=simple User=node_exporter Group=node_exporter ExecStart=/usr/local/bin/node_exporter \ --collector.textfile.directory=/var/lib/node_exporter/textfile_collector \ --web.listen-address=:9100 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target NODE sudo systemctl daemon-reload sudo systemctl enable --now node_exporter # Verify metrics endpoint curl -s http://localhost:9100/metrics | head -20 # Collect additional system metrics # CPU stats cat /proc/stat | head -5 cat /proc/cpuinfo | grep "model name" | head -1 # Memory stats free -h cat /proc/meminfo | head -10 # Disk I/O cat /proc/diskstats iostat -x 1 5 # Network stats ss -s cat /proc/net/netstat | head -5 ``` **Checkpoint:** node_exporter is running and exposing metrics on port 9100. All system metrics are collectable. ### 2. Configure Log Collection Set up centralized log collection from systemd journal and application logs. ```bash # Configure systemd journal forwarding cat > /etc/systemd/journald.conf << 'JOURNAL' [Journal] Storage=persistent SystemMaxUse=500M SystemMaxFileSize=50M MaxRetentionSec=30day ForwardToSyslog=yes Compress=yes JournalFileFlushInterval=30s JOURNAL sudo systemctl restart systemd-journald # Configure rsyslog for log forwarding (centralized logging) sudo apt install -y rsyslog # or dnf install rsyslog cat > /etc/rsyslog.d/50-forward.conf << 'RSYSLOG' # Forward all logs to centralized log server *.* @@log-server.example.com:514 # Forward critical logs with higher priority mail.* /var/log/mail.log auth,authpriv.* /var/log/auth.log kern.* /var/log/kern.log # Rate limit to prevent log floods $SystemLogRateLimitInterval 5 $SystemLogRateLimitBurst 1000 RSYSLOG sudo systemctl restart rsyslog # Configure logrotate for application logs cat > /etc/logrotate.d/myapp << 'LOGROTATE' /var/log/myapp/*.log { daily missingok rotate 30 compress delaycompress notifempty create 0640 myapp myapp sharedscripts postrotate systemctl reload myapp > /dev/null 2>&1 || true endscript } LOGROTATE ``` **Checkpoint:** Journal storage is persistent and sized appropriately. Logs are forwarded to central server. Log rotation prevents disk exhaustion. ### 3. Set Up Performance Profiling Configure performance profiling with perf and eBPF for latency investigation. ```bash # Install perf and BPF tools sudo apt install -y linux-tools-generic linux-tools-$(uname -r) bpfcc-tools # Debian/Ubuntu sudo dnf install -y kernel-tools bpf-tools # RHEL/CentOS # Check eBPF availability sudo bpftrace -l | head -10 # List available eBPF probes # Trace system calls for a specific process sudo perf record -g -p $(pgrep myapp) -F 99 -a -- sleep 30 sudo perf report --stdio # Profile specific syscalls sudo bpftrace -e ' tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); } ' # Measure disk I/O latency sudo bpftrace -e ' tracepoint:block:block_rq_issue { @start[args->dev] = nsecs; } tracepoint:block:block_rq_complete { @latency[args->dev] = hist(nsecs - @start[args->dev]); delete(@start[args->dev]); } ' # Profile kernel functions sudo perf record -g -e cpu-clock task-clock page-faults context-switches \ -p $(pgrep myapp) -- sleep 60 sudo perf report --stdio # Trace network latency with bcc sudo trace-cmd record -e tcp:tcp_sendmsg -e tcp:tcp_rcv_space_adjust \ -p $(pgrep myapp) -- sleep 30 sudo trace-cmd report ``` **Checkpoint:** Performance profiling tools are installed and functional. Baseline profiles are captured for comparison. ### 4. Define Alerting Thresholds Set up threshold-based alerts for resource utilization and error rates. **Bash — Generate Prometheus alerting rules by workload type** ```bash #!/bin/bash # Generate Prometheus alerting rules for a workload type # Usage: ./generate-alerts.sh [output-dir] # Workload types: database, web-server, batch-job, monitoring set -euo pipefail WORKLOAD="${1:-database}" OUTPUT_DIR="${2:-/etc/prometheus/rules}" mkdir -p "$OUTPUT_DIR" # Define alert thresholds per workload type case "$WORKLOAD" in database) CPU_WARN=70; CPU_CRIT=90 MEM_WARN=75; MEM_CRIT=90 DISK_WARN=75; DISK_CRIT=85 NET_ERR_RATE=0.01 ;; web-server) CPU_WARN=60; CPU_CRIT=80 MEM_WARN=70; MEM_CRIT=85 DISK_WARN=80; DISK_CRIT=90 NET_ERR_RATE=0.05 ;; batch-job) CPU_WARN=80; CPU_CRIT=95 MEM_WARN=80; MEM_CRIT=95 DISK_WARN=85; DISK_CRIT=95 NET_ERR_RATE=0.10 ;; monitoring) CPU_WARN=50; CPU_CRIT=70 MEM_WARN=60; MEM_CRIT=80 DISK_WARN=70; DISK_CRIT=80 NET_ERR_RATE=0.01 ;; *) echo "Unknown workload: $WORKLOAD" echo "Available: database, web-server, batch-job, monitoring" exit 1 ;; esac OUTPUT_FILE="${OUTPUT_DIR}/${WORKLOAD}-alerts.yml" cat > "$OUTPUT_FILE" << EOF groups: - name: ${WORKLOAD}-alerts rules: - alert: HighCPUUsage expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > ${CPU_CRIT} for: 5m labels: severity: critical workload: ${WORKLOAD} annotations: summary: "CPU usage above ${CPU_CRIT}% on \${labels.instance}" description: "CPU has been above ${CPU_CRIT}% for more than 5 minutes." - alert: HighCPUUsageWarning expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > ${CPU_WARN} for: 10m labels: severity: warning workload: ${WORKLOAD} annotations: summary: "CPU usage above ${CPU_WARN}% on \${labels.instance}" - alert: HighMemoryUsage expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > ${MEM_CRIT} for: 5m labels: severity: critical workload: ${WORKLOAD} annotations: summary: "Memory usage above ${MEM_CRIT}% on \${labels.instance}" - alert: HighMemoryUsageWarning expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > ${MEM_WARN} for: 10m labels: severity: warning workload: ${WORKLOAD} annotations: summary: "Memory usage above ${MEM_WARN}% on \${labels.instance}" - alert: DiskSpaceCritical expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100 > ${DISK_CRIT} for: 5m labels: severity: critical workload: ${WORKLOAD} annotations: summary: "Disk usage above ${DISK_CRIT}% on \${labels.instance}" - alert: DiskSpaceWarning expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100 > ${DISK_WARN} for: 10m labels: severity: warning workload: ${WORKLOAD} annotations: summary: "Disk usage above ${DISK_WARN}% on \${labels.instance}" - alert: HighNetworkErrorRate expr: rate(node_network_receive_errs_total[5m]) / rate(node_network_receive_bytes_total[5m]) > ${NET_ERR_RATE} for: 5m labels: severity: critical workload: ${WORKLOAD} annotations: summary: "Network error rate above $(echo "$NET_ERR_RATE * 100" | bc)%% on \${labels.instance}" EOF echo "Generated Prometheus alerting rules: ${OUTPUT_FILE}" echo "Thresholds: CPU ${CPU_WARN}/${CPU_CRIT}%, Memory ${MEM_WARN}/${MEM_CRIT}%, Disk ${DISK_WARN}/${DISK_CRIT}%" ``` **Checkpoint:** Alert thresholds are defined for each workload type. Rules are deployed to the monitoring stack and tested with synthetic load. ### 5. Implement Capacity Planning Set up capacity planning with trend analysis and growth projections. ```bash # Collect historical metrics for capacity analysis # Create metrics collection script cat > /usr/local/bin/capacity-collect << 'CAPACITY' #!/bin/bash TIMESTAMP=$(date +%s) HOSTNAME=$(hostname) OUTPUT="/var/lib/capacity-history/metrics-$(hostname).csv" mkdir -p /var/lib/capacity-history # CPU usage (percentage) CPU_IDLE=$(top -bn1 | grep "Cpu(s)" | awk '{print $8}' | cut -d. -f1) CPU_USAGE=$((100 - CPU_IDLE)) # Memory usage (percentage) MEM_TOTAL=$(grep MemTotal /proc/meminfo | awk '{print $2}') MEM_AVAIL=$(grep MemAvailable /proc/meminfo | awk '{print $2}') MEM_USAGE=$(echo "scale=1; (1 - $MEM_AVAIL / $MEM_TOTAL) * 100" | bc) # Disk usage (percentage) DISK_USAGE=$(df / --output=pcent | tail -1 | tr -d ' %') # Load average (1min, 5min, 15min) LOADAVG=$(cat /proc/loadavg | awk '{print $1, $2, $3}') # Network throughput (bytes received/sent in last interval) NET_RX=$(cat /proc/net/dev | grep -E "eth[0-9]|ens[0-9]|enp[0-9]" | tail -1 | awk '{print $2}') NET_TX=$(cat /proc/net/dev | grep -E "eth[0-9]|ens[0-9]|enp[0-9]" | tail -1 | awk '{print $10}') echo "${TIMESTAMP},${HOSTNAME},${CPU_USAGE},${MEM_USAGE},${DISK_USAGE},${LOADAVG},${NET_RX},${NET_TX}" >> "$OUTPUT" CAPACITY sudo chmod +x /usr/local/bin/capacity-collect # Run every 5 minutes sudo bash -c 'echo "*/5 * * * * /usr/local/bin/capacity-collect" > /etc/cron.d/capacity-collect' # Capacity analysis script cat > /usr/local/bin/capacity-analyze << 'ANALYZE' #!/bin/bash # Analyze capacity trends and project when thresholds will be hit METRICS_FILE="/var/lib/capacity-history/metrics-$(hostname).csv" if [ ! -f "$METRICS_FILE" ]; then echo "No metrics data found. Run capacity-collect first." exit 1 fi # Calculate current averages and trends echo "=== Capacity Analysis: $(hostname) ===" echo "Date: $(date)" echo "" # Disk trend (simple linear projection) DISK_CURRENT=$(tail -1 "$METRICS_FILE" | cut -d',' -f5) DISK_HISTORY=$(awk -F',' '{print $5}' "$METRICS_FILE" | tail -288) # 24 hours at 5min intervals echo "Disk Usage:" echo " Current: ${DISK_CURRENT}%" echo " 24h avg: $(echo "$DISK_HISTORY" | awk '{sum+=$1} END {printf "%.1f", sum/NR}')%" echo " 24h min: $(echo "$DISK_HISTORY" | sort -n | head -1)%" echo " 24h max: $(echo "$DISK_HISTORY" | sort -n | tail -1)%" # Project days until disk is full (simple linear extrapolation) # This is a simplified model; use proper time-series for production echo "" echo "Estimated days until disk full: $(echo "$DISK_CURRENT" | awk '{if ($1 >= 100) print "ALREADY FULL"; else printf "%d", (100-$1)/($1-$(echo "$DISK_HISTORY" | sort -n | head -1))}') days" ANALYZE sudo chmod +x /usr/local/bin/capacity-analyze ``` **Checkpoint:** Historical metrics are being collected. Capacity trends are analyzable. Growth projections provide actionable planning data. ### 6. Cloud vs On-Prem Observability ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ Cloud vs On-Prem Observability Strategy │ ├─────────────────────┬───────────────────────────────────────────────────┤ │ Aspect │ Cloud │ On-Prem │ ├─────────────────────┼───────────────────────────────────────────────────┤ │ Infrastructure │ Cloud provider metrics (CloudWatch, Stackdriver, │ │ Metrics │ Azure Monitor) — auto-collected, no agents │ │ │ │ needed for compute, storage, network │ │ ├─────────────────────┼───────────────────────────────────────────────────┤ │ Instance Metrics │ node_exporter + Prometheus for application- │ node_exporter + │ │ │ level metrics; CloudWatch agent for │ Prometheus │ │ │ enhanced monitoring │ (self-hosted) │ ├─────────────────────┼───────────────────────────────────────────────────┤ │ Logging │ CloudTrail + VPC Flow Logs + │ rsyslog + │ │ │ structured logging to S3/GCS/ │ journald + │ │ │ Blob Storage + CloudWatch Logs │ centralized │ │ │ │ log server │ ├─────────────────────┼───────────────────────────────────────────────────┤ │ Tracing │ X-Ray, Cloud Trace, Application │ Jaeger + │ │ │ Insights (auto-instrumentation) │ OpenTelemetry │ ├─────────────────────┼───────────────────────────────────────────────────┤ │ Cost │ Pay per metric ingested; │ Fixed cost; │ │ │ auto-scaling observability is free │ self-managed │ ├─────────────────────┼───────────────────────────────────────────────────┤ │ Access │ IAM-based; no physical access needed │ Full hardware │ │ │ │ access; TPM/ │ │ │ │ IPMI for hardware│ └─────────────────────┴───────────────────────────────────────────────────┘ ``` --- ## Implementation Patterns ### Pattern 1: System Health Dashboard Script (BAD vs. GOOD) **BAD — Manual health checking without automation** ```bash # ❌ BAD: Ad-hoc manual health checks echo "CPU: $(top -bn1 | grep 'Cpu(s)')" free -h df -h ss -tuln # Problems: # - Not repeatable or automatable # - No historical data for comparison # - No alerting — must manually review each time # - Inconsistent format makes comparison impossible # - No baseline comparison ``` **GOOD — Automated health check with structured output** ```bash #!/bin/bash # Automated system health check with JSON output # Usage: ./health-check.sh [--json] [--output-dir /var/lib/health-checks/] set -euo pipefail JSON_OUTPUT=false OUTPUT_DIR="/var/lib/health-checks" HOSTNAME=$(hostname) TIMESTAMP=$(date -Iseconds) while [[ $# -gt 0 ]]; do case "$1" in --json) JSON_OUTPUT=true; shift ;; --output-dir) OUTPUT_DIR="$2"; shift 2 ;; *) shift ;; esac done mkdir -p "$OUTPUT_DIR" # Check CPU usage get_cpu_usage() { local idle idle=$(top -bn1 | grep "Cpu(s)" | awk '{print $8}' | cut -d. -f1) echo $((100 - ${idle:-0})) } # Check memory usage get_memory_usage() { local total avail total=$(awk '/^MemTotal:/{print $2}' /proc/meminfo) avail=$(awk '/^MemAvailable:/{print $2}' /proc/meminfo) if [[ $total -gt 0 ]]; then echo "scale=1; (1 - $avail / $total) * 100" | bc | cut -d. -f1 else echo 0 fi } # Check disk usage get_disk_usage() { df / --output=pcent | tail -1 | tr -d ' %' } # Check load ratio get_load_ratio() { local load nproc load_ratio load=$(awk '{print $1}' /proc/loadavg) nproc=$(nproc) load_ratio=$(echo "scale=2; $load / $nproc" | bc) echo "$load_ratio" } # Determine status from threshold get_status() { local value warn crit value=$(echo "$1" | cut -d. -f1) warn=${2:-70} crit=${3:-90} if [[ $value -ge $crit ]]; then echo "critical" elif [[ $value -ge $warn ]]; then echo "warning" else echo "ok"; fi } CPU_USAGE=$(get_cpu_usage) MEM_USAGE=$(get_memory_usage) DISK_USAGE=$(get_disk_usage) LOAD_RATIO=$(get_load_ratio) CPU_STATUS=$(get_status "$CPU_USAGE" 70 90) MEM_STATUS=$(get_status "$MEM_USAGE" 75 90) DISK_STATUS=$(get_status "$DISK_USAGE" 75 90) LOAD_STATUS=$(get_status "${LOAD_RATIO%.*}" 1 2) # Determine overall status OVERALL="healthy" if [[ "$CPU_STATUS" == "critical" || "$MEM_STATUS" == "critical" || "$DISK_STATUS" == "critical" || "$LOAD_STATUS" == "critical" ]]; then OVERALL="critical" elif [[ "$CPU_STATUS" == "warning" || "$MEM_STATUS" == "warning" || "$DISK_STATUS" == "warning" || "$LOAD_STATUS" == "warning" ]]; then OVERALL="degraded" fi if [[ "$JSON_OUTPUT" == "true" ]]; then # Output JSON cat << EOF { "hostname": "$HOSTNAME", "timestamp": "$TIMESTAMP", "overall_status": "$OVERALL", "checks": [ {"metric": "cpu_usage", "value": $CPU_USAGE, "unit": "%", "status": "$CPU_STATUS", "threshold_warning": 70, "threshold_critical": 90}, {"metric": "memory_usage", "value": $MEM_USAGE, "unit": "%", "status": "$MEM_STATUS", "threshold_warning": 75, "threshold_critical": 90}, {"metric": "disk_usage", "value": $DISK_USAGE, "unit": "%", "status": "$DISK_STATUS", "threshold_warning": 75, "threshold_critical": 90}, {"metric": "load_ratio", "value": $LOAD_RATIO, "unit": "x", "status": "$LOAD_STATUS", "threshold_warning": 1.0, "threshold_critical": 2.0} ] } EOF else # Human-readable output echo "=== System Health: $HOSTNAME ===" echo "Timestamp: $TIMESTAMP" echo "Overall: $OVERALL" echo "" echo "CPU Usage: ${CPU_USAGE}% [${CPU_STATUS}]" echo "Memory Usage: ${MEM_USAGE}% [${MEM_STATUS}]" echo "Disk Usage: ${DISK_USAGE}% [${DISK_STATUS}]" echo "Load Ratio: ${LOAD_RATIO}x [${LOAD_STATUS}]" fi # Save to file OUTPUT_FILE="$OUTPUT_DIR/${HOSTNAME}-${TIMESTAMP}.json" mkdir -p "$OUTPUT_DIR" cat << EOF > "$OUTPUT_FILE" { "hostname": "$HOSTNAME", "timestamp": "$TIMESTAMP", "overall_status": "$OVERALL", "checks": [ {"metric": "cpu_usage", "value": $CPU_USAGE, "unit": "%", "status": "$CPU_STATUS", "threshold_warning": 70, "threshold_critical": 90}, {"metric": "memory_usage", "value": $MEM_USAGE, "unit": "%", "status": "$MEM_STATUS", "threshold_warning": 75, "threshold_critical": 90}, {"metric": "disk_usage", "value": $DISK_USAGE, "unit": "%", "status": "$DISK_STATUS", "threshold_warning": 75, "threshold_critical": 90}, {"metric": "load_ratio", "value": $LOAD_RATIO, "unit": "x", "status": "$LOAD_STATUS", "threshold_warning": 1.0, "threshold_critical": 2.0} ] } EOF echo "Report saved to $OUTPUT_FILE" ``` ### Pattern 2: eBPF-Based Latency Investigation ```bash # ❌ BAD: Using strace for performance investigation # strace -p $(pgrep myapp) -e trace=network # Problems: # - High overhead (can slow process by 10-100x) # - Not suitable for production # - Generates massive output # ✅ GOOD: Using eBPF for low-overhead tracing # Install BCC tools sudo apt install -y bpfcc-tools linux-headers-$(uname -r) # Measure TCP connection latency with low overhead sudo bpftrace -e ' tracepoint:tcp:tcp_set_state /args->state == 1/ { @connect_latency[tid] = nsecs; } tracepoint:tcp:tcp_rcv_space_adjust /@connect_latency[tid]/ { $latency = nsecs - @connect_latency[tid]; printf("%d %d\n", tid, $latency); delete(@connect_latency[tid]); } ' # Profile file system latency sudo bpftrace -e ' tracepoint:block:block_rq_issue { @start[args->dev, args->sector] = nsecs; } tracepoint:block:block_rq_complete /@start[args->dev, args->sector]/ { $latency = nsecs - @start[args->dev, args->sector]; @latency_hist[args->dev] = hist($latency); delete(@start[args->dev, args->sector]); } ' # System-wide syscall latency histogram sudo bpftrace -e ' kprobe:do_sys_open { @start[tid] = nsecs; } kretprobe:do_sys_open /@start[tid]/ { $latency = nsecs - @start[tid]; @latency = hist($latency); delete(@start[tid]); } ' ``` ### Pattern 3: Alert Threshold Configuration by Workload **Bash — Alert threshold configuration by workload** ```bash #!/bin/bash # Display alert thresholds for a workload type # Usage: ./alert-thresholds.sh # Workload types: database, web-server set -euo pipefail WORKLOAD="${1:-}" if [[ -z "$WORKLOAD" ]]; then echo "Usage: $0 {database|web-server}" exit 1 fi echo "=== Alert Thresholds: ${WORKLOAD} ===" echo "" case "$WORKLOAD" in database) echo "# Database workload alert thresholds" echo "# File: /etc/prometheus/rules/database-alerts.yml" echo "" cat << 'RULES' groups: - name: database-alerts rules: - alert: HighCPUUsage expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90 for: 5m labels: { severity: critical, workload: database } - alert: HighMemoryUsage expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90 for: 5m labels: { severity: critical, workload: database } - alert: DiskIOSaturation expr: rate(node_disk_io_time_seconds_total[5m]) > 0.95 for: 5m labels: { severity: critical, workload: database } - alert: DBConnectionsHigh expr: postgresql_connections_active > 200 for: 1m labels: { severity: critical, workload: database } - alert: ReplicationLag expr: postgresql_replication_lag_seconds > 300 for: 5m labels: { severity: critical, workload: database } RULES echo "" echo " CPU: warning=70%, critical=90%" echo " Memory: warning=75%, critical=90%" echo " DiskIO: warning=80%, critical=95%" echo " Conns: warning=100, critical=200" echo " RepLag: warning=30s, critical=300s" ;; web-server) echo "# Web server workload alert thresholds" echo "# File: /etc/prometheus/rules/web-server-alerts.yml" echo "" cat << 'RULES' groups: - name: web-server-alerts rules: - alert: HighCPUUsage expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80 for: 5m labels: { severity: critical, workload: web-server } - alert: HighMemoryUsage expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85 for: 5m labels: { severity: critical, workload: web-server } - alert: HighRequestLatency expr: histogram_quantile(0.50, rate(http_request_duration_seconds_bucket[5m])) > 5 for: 5m labels: { severity: critical, workload: web-server } - alert: HighErrorRate expr: rate(http_request_errors_total[5m]) / rate(http_requests_total[5m]) > 0.05 for: 5m labels: { severity: critical, workload: web-server } - alert: HighNginxConnections expr: nginx_active_connections > 5000 for: 1m labels: { severity: critical, workload: web-server } RULES echo "" echo " CPU: warning=60%, critical=80%" echo " Memory: warning=70%, critical=85%" echo " Latency: warning=1.0s, critical=5.0s" echo " ErrorRt: warning=1.0%, critical=5.0%" echo " Conns: warning=1000, critical=5000" ;; *) echo "Unknown workload: $WORKLOAD" echo "Available: database, web-server" exit 1 ;; esac ``` --- ## Constraints ### MUST DO - **MUST** collect metrics at a consistent interval (5 minutes minimum, 1 minute for critical workloads) for accurate trend analysis - **MUST** store metrics with timestamps and labels (hostname, workload, region) for proper filtering and correlation - **MUST** set alert thresholds based on workload baselines, not arbitrary values — collect 2 weeks of baseline data before setting alerts - **MUST** configure log rotation with size limits to prevent disk exhaustion from unbounded log growth - **MUST** use eBPF-based tools (bcc, bpftrace, perf) for production performance investigation — they have minimal overhead compared to strace - **MUST** document alert runbooks for each alert type — responders need clear steps to diagnose and remediate - **MUST** test alerting delivery (email, Slack, PagerDuty) for each alert severity level - **MUST** maintain a 30-day metrics retention window minimum for capacity planning trend analysis ### MUST NOT DO - **MUST NOT** set alert thresholds on the first day of deployment — allow at least 2 weeks of baseline data collection - **MUST NOT** use strace or similar high-overhead tracing tools in production without explicit approval — use eBPF instead - **MUST NOT** store all logs on the local disk indefinitely — forward to centralized log storage with defined retention - **MUST NOT** ignore warning alerts — treat warnings as pre-critical signals that require investigation within SLA timeframes - **MUST NOT** set the same alert thresholds for all workload types — databases, web servers, and batch jobs have fundamentally different profiles - **MUST NOT** rely solely on cloud provider metrics for observability — deploy agent-based metrics collection for application-level visibility - **MUST NOT** set CPU alert thresholds above 90% for production systems — sustained high CPU indicates capacity planning issues --- ## Related Skills | Skill | Purpose | |-------|---------| | `resource-management` | Investigate resource contention issues flagged by monitoring | | `kernel-tuning` | Apply kernel parameter changes based on performance profiling results | | `networking` | Investigate network performance issues flagged by observability | | `linux-security` | Monitor security-relevant metrics (failed logins, audit events) | | `hardware-provisioning` | Plan hardware capacity based on observability-driven growth projections |