Observability

Linux System Observability

Infrastructure engineer implementing comprehensive Linux system observability with metrics collection, log aggregation, performance profiling, and capacity planning for proactive infrastructure management across cloud and on-prem environments.

TL;DR Checklist

• Deploy metrics collection for CPU, memory, disk I/O, and network throughput
• Configure log collection from systemd journal and application logs
• Set up performance profiling with perf and eBPF for latency investigation
• Define alerting thresholds for resource utilization and error rates
• Implement capacity planning with trend analysis and growth projections
• Validate observability stack with synthetic load and failure injection tests
• Document baseline metrics and alerting configuration for each workload type
• Test alerting delivery and runbook execution for each alert type

When to Use

Use this skill when:

• Setting up monitoring from scratch — You're deploying a new system and need to establish observability baselines
• Investigating performance issues — A system is experiencing latency, throughput, or reliability problems and needs profiling
• Capacity planning — You need to project resource needs based on historical growth trends
• Setting up alerting — You need threshold-based alerts for proactive incident detection
• Post-incident review — You need to analyze what happened during an incident using collected metrics and logs
• Cloud vs on-prem observability differences — You need to adapt observability strategy for different deployment environments

When NOT to Use

Avoid this skill for:

• Application-level monitoring — Use application-specific APM tools (OpenTelemetry, Datadog APM) for application metrics
• Network packet analysis — Use tcpdump or Wireshark for packet-level investigation
• Security forensics — Use linux-security skill for forensic investigation of security incidents
• Real-time debugging — Use perf record and eBPF for low-level kernel debugging, not production monitoring

Use resource-management when investigating specific resource contention issues (CPU throttling, memory pressure). Use networking when network performance is the suspected bottleneck.

Core Workflow

1. Deploy Metrics Collection

Set up system metrics collection for CPU, memory, disk, and network.

# Install node_exporter (Prometheus metrics)
sudo useradd --no-create-home --shell /bin/false node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xzf node_exporter-*.tar.gz
sudo cp node_exporter-*/node_exporter /usr/local/bin/
rm -rf node_exporter-*

# Create systemd service
cat > /etc/systemd/system/node_exporter.service << 'NODE'
[Unit]
Description=Node Exporter
After=network.target

[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter \
    --collector.textfile.directory=/var/lib/node_exporter/textfile_collector \
    --web.listen-address=:9100
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
NODE

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

# Verify metrics endpoint
curl -s http://localhost:9100/metrics | head -20

# Collect additional system metrics
# CPU stats
cat /proc/stat | head -5
cat /proc/cpuinfo | grep "model name" | head -1

# Memory stats
free -h
cat /proc/meminfo | head -10

# Disk I/O
cat /proc/diskstats
iostat -x 1 5

# Network stats
ss -s
cat /proc/net/netstat | head -5

Checkpoint: node_exporter is running and exposing metrics on port 9100. All system metrics are collectable.

2. Configure Log Collection

Set up centralized log collection from systemd journal and application logs.

# Configure systemd journal forwarding
cat > /etc/systemd/journald.conf << 'JOURNAL'
[Journal]
Storage=persistent
SystemMaxUse=500M
SystemMaxFileSize=50M
MaxRetentionSec=30day
ForwardToSyslog=yes
Compress=yes
JournalFileFlushInterval=30s
JOURNAL

sudo systemctl restart systemd-journald

# Configure rsyslog for log forwarding (centralized logging)
sudo apt install -y rsyslog  # or dnf install rsyslog

cat > /etc/rsyslog.d/50-forward.conf << 'RSYSLOG'
# Forward all logs to centralized log server
*.* @@log-server.example.com:514

# Forward critical logs with higher priority
mail.*                          /var/log/mail.log
auth,authpriv.*                 /var/log/auth.log
kern.*                          /var/log/kern.log

# Rate limit to prevent log floods
$SystemLogRateLimitInterval 5
$SystemLogRateLimitBurst 1000
RSYSLOG

sudo systemctl restart rsyslog

# Configure logrotate for application logs
cat > /etc/logrotate.d/myapp << 'LOGROTATE'
/var/log/myapp/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 myapp myapp
    sharedscripts
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}
LOGROTATE

Checkpoint: Journal storage is persistent and sized appropriately. Logs are forwarded to central server. Log rotation prevents disk exhaustion.

3. Set Up Performance Profiling

Configure performance profiling with perf and eBPF for latency investigation.

# Install perf and BPF tools
sudo apt install -y linux-tools-generic linux-tools-$(uname -r) bpfcc-tools  # Debian/Ubuntu
sudo dnf install -y kernel-tools bpf-tools  # RHEL/CentOS

# Check eBPF availability
sudo bpftrace -l | head -10  # List available eBPF probes

# Trace system calls for a specific process
sudo perf record -g -p $(pgrep myapp) -F 99 -a -- sleep 30
sudo perf report --stdio

# Profile specific syscalls
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_openat {
    printf("%s %s\n", comm, str(args->filename));
}
'

# Measure disk I/O latency
sudo bpftrace -e '
tracepoint:block:block_rq_issue {
    @start[args->dev] = nsecs;
}
tracepoint:block:block_rq_complete {
    @latency[args->dev] = hist(nsecs - @start[args->dev]);
    delete(@start[args->dev]);
}
'

# Profile kernel functions
sudo perf record -g -e cpu-clock task-clock page-faults context-switches \
    -p $(pgrep myapp) -- sleep 60
sudo perf report --stdio

# Trace network latency with bcc
sudo trace-cmd record -e tcp:tcp_sendmsg -e tcp:tcp_rcv_space_adjust \
    -p $(pgrep myapp) -- sleep 30
sudo trace-cmd report

Checkpoint: Performance profiling tools are installed and functional. Baseline profiles are captured for comparison.

4. Define Alerting Thresholds

Set up threshold-based alerts for resource utilization and error rates.

Bash — Generate Prometheus alerting rules by workload type

#!/bin/bash
# Generate Prometheus alerting rules for a workload type
# Usage: ./generate-alerts.sh <workload-type> [output-dir]
# Workload types: database, web-server, batch-job, monitoring

set -euo pipefail

WORKLOAD="${1:-database}"
OUTPUT_DIR="${2:-/etc/prometheus/rules}"

mkdir -p "$OUTPUT_DIR"

# Define alert thresholds per workload type
case "$WORKLOAD" in
    database)
        CPU_WARN=70; CPU_CRIT=90
        MEM_WARN=75; MEM_CRIT=90
        DISK_WARN=75; DISK_CRIT=85
        NET_ERR_RATE=0.01
        ;;
    web-server)
        CPU_WARN=60; CPU_CRIT=80
        MEM_WARN=70; MEM_CRIT=85
        DISK_WARN=80; DISK_CRIT=90
        NET_ERR_RATE=0.05
        ;;
    batch-job)
        CPU_WARN=80; CPU_CRIT=95
        MEM_WARN=80; MEM_CRIT=95
        DISK_WARN=85; DISK_CRIT=95
        NET_ERR_RATE=0.10
        ;;
    monitoring)
        CPU_WARN=50; CPU_CRIT=70
        MEM_WARN=60; MEM_CRIT=80
        DISK_WARN=70; DISK_CRIT=80
        NET_ERR_RATE=0.01
        ;;
    *)
        echo "Unknown workload: $WORKLOAD"
        echo "Available: database, web-server, batch-job, monitoring"
        exit 1
        ;;
esac

OUTPUT_FILE="${OUTPUT_DIR}/${WORKLOAD}-alerts.yml"

cat > "$OUTPUT_FILE" << EOF
groups:
  - name: ${WORKLOAD}-alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > ${CPU_CRIT}
        for: 5m
        labels:
          severity: critical
          workload: ${WORKLOAD}
        annotations:
          summary: "CPU usage above ${CPU_CRIT}% on \${labels.instance}"
          description: "CPU has been above ${CPU_CRIT}% for more than 5 minutes."

      - alert: HighCPUUsageWarning
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > ${CPU_WARN}
        for: 10m
        labels:
          severity: warning
          workload: ${WORKLOAD}
        annotations:
          summary: "CPU usage above ${CPU_WARN}% on \${labels.instance}"

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > ${MEM_CRIT}
        for: 5m
        labels:
          severity: critical
          workload: ${WORKLOAD}
        annotations:
          summary: "Memory usage above ${MEM_CRIT}% on \${labels.instance}"

      - alert: HighMemoryUsageWarning
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > ${MEM_WARN}
        for: 10m
        labels:
          severity: warning
          workload: ${WORKLOAD}
        annotations:
          summary: "Memory usage above ${MEM_WARN}% on \${labels.instance}"

      - alert: DiskSpaceCritical
        expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100 > ${DISK_CRIT}
        for: 5m
        labels:
          severity: critical
          workload: ${WORKLOAD}
        annotations:
          summary: "Disk usage above ${DISK_CRIT}% on \${labels.instance}"

      - alert: DiskSpaceWarning
        expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100 > ${DISK_WARN}
        for: 10m
        labels:
          severity: warning
          workload: ${WORKLOAD}
        annotations:
          summary: "Disk usage above ${DISK_WARN}% on \${labels.instance}"

      - alert: HighNetworkErrorRate
        expr: rate(node_network_receive_errs_total[5m]) / rate(node_network_receive_bytes_total[5m]) > ${NET_ERR_RATE}
        for: 5m
        labels:
          severity: critical
          workload: ${WORKLOAD}
        annotations:
          summary: "Network error rate above $(echo "$NET_ERR_RATE * 100" | bc)%% on \${labels.instance}"
EOF

echo "Generated Prometheus alerting rules: ${OUTPUT_FILE}"
echo "Thresholds: CPU ${CPU_WARN}/${CPU_CRIT}%, Memory ${MEM_WARN}/${MEM_CRIT}%, Disk ${DISK_WARN}/${DISK_CRIT}%"

Checkpoint: Alert thresholds are defined for each workload type. Rules are deployed to the monitoring stack and tested with synthetic load.

5. Implement Capacity Planning

Set up capacity planning with trend analysis and growth projections.

# Collect historical metrics for capacity analysis
# Create metrics collection script
cat > /usr/local/bin/capacity-collect << 'CAPACITY'
#!/bin/bash
TIMESTAMP=$(date +%s)
HOSTNAME=$(hostname)
OUTPUT="/var/lib/capacity-history/metrics-$(hostname).csv"

mkdir -p /var/lib/capacity-history

# CPU usage (percentage)
CPU_IDLE=$(top -bn1 | grep "Cpu(s)" | awk '{print $8}' | cut -d. -f1)
CPU_USAGE=$((100 - CPU_IDLE))

# Memory usage (percentage)
MEM_TOTAL=$(grep MemTotal /proc/meminfo | awk '{print $2}')
MEM_AVAIL=$(grep MemAvailable /proc/meminfo | awk '{print $2}')
MEM_USAGE=$(echo "scale=1; (1 - $MEM_AVAIL / $MEM_TOTAL) * 100" | bc)

# Disk usage (percentage)
DISK_USAGE=$(df / --output=pcent | tail -1 | tr -d ' %')

# Load average (1min, 5min, 15min)
LOADAVG=$(cat /proc/loadavg | awk '{print $1, $2, $3}')

# Network throughput (bytes received/sent in last interval)
NET_RX=$(cat /proc/net/dev | grep -E "eth[0-9]|ens[0-9]|enp[0-9]" | tail -1 | awk '{print $2}')
NET_TX=$(cat /proc/net/dev | grep -E "eth[0-9]|ens[0-9]|enp[0-9]" | tail -1 | awk '{print $10}')

echo "${TIMESTAMP},${HOSTNAME},${CPU_USAGE},${MEM_USAGE},${DISK_USAGE},${LOADAVG},${NET_RX},${NET_TX}" >> "$OUTPUT"
CAPACITY
sudo chmod +x /usr/local/bin/capacity-collect

# Run every 5 minutes
sudo bash -c 'echo "*/5 * * * * /usr/local/bin/capacity-collect" > /etc/cron.d/capacity-collect'

# Capacity analysis script
cat > /usr/local/bin/capacity-analyze << 'ANALYZE'
#!/bin/bash
# Analyze capacity trends and project when thresholds will be hit

METRICS_FILE="/var/lib/capacity-history/metrics-$(hostname).csv"

if [ ! -f "$METRICS_FILE" ]; then
    echo "No metrics data found. Run capacity-collect first."
    exit 1
fi

# Calculate current averages and trends
echo "=== Capacity Analysis: $(hostname) ==="
echo "Date: $(date)"
echo ""

# Disk trend (simple linear projection)
DISK_CURRENT=$(tail -1 "$METRICS_FILE" | cut -d',' -f5)
DISK_HISTORY=$(awk -F',' '{print $5}' "$METRICS_FILE" | tail -288)  # 24 hours at 5min intervals

echo "Disk Usage:"
echo "  Current: ${DISK_CURRENT}%"
echo "  24h avg: $(echo "$DISK_HISTORY" | awk '{sum+=$1} END {printf "%.1f", sum/NR}')%"
echo "  24h min: $(echo "$DISK_HISTORY" | sort -n | head -1)%"
echo "  24h max: $(echo "$DISK_HISTORY" | sort -n | tail -1)%"

# Project days until disk is full (simple linear extrapolation)
# This is a simplified model; use proper time-series for production
echo ""
echo "Estimated days until disk full: $(echo "$DISK_CURRENT" | awk '{if ($1 >= 100) print "ALREADY FULL"; else printf "%d", (100-$1)/($1-$(echo "$DISK_HISTORY" | sort -n | head -1))}') days"
ANALYZE
sudo chmod +x /usr/local/bin/capacity-analyze

Checkpoint: Historical metrics are being collected. Capacity trends are analyzable. Growth projections provide actionable planning data.

6. Cloud vs On-Prem Observability

┌─────────────────────────────────────────────────────────────────────────┐
│                  Cloud vs On-Prem Observability Strategy                  │
├─────────────────────┬───────────────────────────────────────────────────┤
│ Aspect              │ Cloud                                             │ On-Prem        │
├─────────────────────┼───────────────────────────────────────────────────┤
│ Infrastructure      │ Cloud provider metrics (CloudWatch, Stackdriver,  │
│ Metrics             │ Azure Monitor) — auto-collected, no agents       │           │
│                     │ needed for compute, storage, network                │                   │
├─────────────────────┼───────────────────────────────────────────────────┤
│ Instance Metrics    │ node_exporter + Prometheus for application-      │ node_exporter +  │
│                     │ level metrics; CloudWatch agent for              │ Prometheus       │
│                     │ enhanced monitoring                           │ (self-hosted)    │
├─────────────────────┼───────────────────────────────────────────────────┤
│ Logging             │ CloudTrail + VPC Flow Logs +                  │ rsyslog +        │
│                     │ structured logging to S3/GCS/                  │ journald +       │
│                     │ Blob Storage + CloudWatch Logs                 │ centralized      │
│                     │                                                  │ log server       │
├─────────────────────┼───────────────────────────────────────────────────┤
│ Tracing             │ X-Ray, Cloud Trace, Application               │ Jaeger +         │
│                     │ Insights (auto-instrumentation)               │ OpenTelemetry    │
├─────────────────────┼───────────────────────────────────────────────────┤
│ Cost                │ Pay per metric ingested;                      │ Fixed cost;      │
│                     │ auto-scaling observability is free              │ self-managed     │
├─────────────────────┼───────────────────────────────────────────────────┤
│ Access              │ IAM-based; no physical access needed          │ Full hardware    │
│                     │                                                   │ access; TPM/     │
│                     │                                                   │ IPMI for hardware│
└─────────────────────┴───────────────────────────────────────────────────┘

Implementation Patterns

Pattern 1: System Health Dashboard Script (BAD vs. GOOD)

BAD — Manual health checking without automation

# ❌ BAD: Ad-hoc manual health checks
echo "CPU: $(top -bn1 | grep 'Cpu(s)')"
free -h
df -h
ss -tuln
# Problems:
# - Not repeatable or automatable
# - No historical data for comparison
# - No alerting — must manually review each time
# - Inconsistent format makes comparison impossible
# - No baseline comparison

GOOD — Automated health check with structured output

#!/bin/bash
# Automated system health check with JSON output
# Usage: ./health-check.sh [--json] [--output-dir /var/lib/health-checks/]
set -euo pipefail

JSON_OUTPUT=false
OUTPUT_DIR="/var/lib/health-checks"
HOSTNAME=$(hostname)
TIMESTAMP=$(date -Iseconds)

while [[ $# -gt 0 ]]; do
    case "$1" in
        --json) JSON_OUTPUT=true; shift ;;
        --output-dir) OUTPUT_DIR="$2"; shift 2 ;;
        *) shift ;;
    esac
done

mkdir -p "$OUTPUT_DIR"

# Check CPU usage
get_cpu_usage() {
    local idle
    idle=$(top -bn1 | grep "Cpu(s)" | awk '{print $8}' | cut -d. -f1)
    echo $((100 - ${idle:-0}))
}

# Check memory usage
get_memory_usage() {
    local total avail
    total=$(awk '/^MemTotal:/{print $2}' /proc/meminfo)
    avail=$(awk '/^MemAvailable:/{print $2}' /proc/meminfo)
    if [[ $total -gt 0 ]]; then
        echo "scale=1; (1 - $avail / $total) * 100" | bc | cut -d. -f1
    else
        echo 0
    fi
}

# Check disk usage
get_disk_usage() {
    df / --output=pcent | tail -1 | tr -d ' %'
}

# Check load ratio
get_load_ratio() {
    local load nproc load_ratio
    load=$(awk '{print $1}' /proc/loadavg)
    nproc=$(nproc)
    load_ratio=$(echo "scale=2; $load / $nproc" | bc)
    echo "$load_ratio"
}

# Determine status from threshold
get_status() {
    local value warn crit
    value=$(echo "$1" | cut -d. -f1)
    warn=${2:-70}
    crit=${3:-90}
    if [[ $value -ge $crit ]]; then echo "critical"
    elif [[ $value -ge $warn ]]; then echo "warning"
    else echo "ok"; fi
}

CPU_USAGE=$(get_cpu_usage)
MEM_USAGE=$(get_memory_usage)
DISK_USAGE=$(get_disk_usage)
LOAD_RATIO=$(get_load_ratio)

CPU_STATUS=$(get_status "$CPU_USAGE" 70 90)
MEM_STATUS=$(get_status "$MEM_USAGE" 75 90)
DISK_STATUS=$(get_status "$DISK_USAGE" 75 90)
LOAD_STATUS=$(get_status "${LOAD_RATIO%.*}" 1 2)

# Determine overall status
OVERALL="healthy"
if [[ "$CPU_STATUS" == "critical" || "$MEM_STATUS" == "critical" || "$DISK_STATUS" == "critical" || "$LOAD_STATUS" == "critical" ]]; then
    OVERALL="critical"
elif [[ "$CPU_STATUS" == "warning" || "$MEM_STATUS" == "warning" || "$DISK_STATUS" == "warning" || "$LOAD_STATUS" == "warning" ]]; then
    OVERALL="degraded"
fi

if [[ "$JSON_OUTPUT" == "true" ]]; then
    # Output JSON
    cat << EOF
{
  "hostname": "$HOSTNAME",
  "timestamp": "$TIMESTAMP",
  "overall_status": "$OVERALL",
  "checks": [
    {"metric": "cpu_usage", "value": $CPU_USAGE, "unit": "%", "status": "$CPU_STATUS", "threshold_warning": 70, "threshold_critical": 90},
    {"metric": "memory_usage", "value": $MEM_USAGE, "unit": "%", "status": "$MEM_STATUS", "threshold_warning": 75, "threshold_critical": 90},
    {"metric": "disk_usage", "value": $DISK_USAGE, "unit": "%", "status": "$DISK_STATUS", "threshold_warning": 75, "threshold_critical": 90},
    {"metric": "load_ratio", "value": $LOAD_RATIO, "unit": "x", "status": "$LOAD_STATUS", "threshold_warning": 1.0, "threshold_critical": 2.0}
  ]
}
EOF
else
    # Human-readable output
    echo "=== System Health: $HOSTNAME ==="
    echo "Timestamp: $TIMESTAMP"
    echo "Overall: $OVERALL"
    echo ""
    echo "CPU Usage:    ${CPU_USAGE}% [${CPU_STATUS}]"
    echo "Memory Usage: ${MEM_USAGE}% [${MEM_STATUS}]"
    echo "Disk Usage:   ${DISK_USAGE}% [${DISK_STATUS}]"
    echo "Load Ratio:   ${LOAD_RATIO}x [${LOAD_STATUS}]"
fi

# Save to file
OUTPUT_FILE="$OUTPUT_DIR/${HOSTNAME}-${TIMESTAMP}.json"
mkdir -p "$OUTPUT_DIR"
cat << EOF > "$OUTPUT_FILE"
{
  "hostname": "$HOSTNAME",
  "timestamp": "$TIMESTAMP",
  "overall_status": "$OVERALL",
  "checks": [
    {"metric": "cpu_usage", "value": $CPU_USAGE, "unit": "%", "status": "$CPU_STATUS", "threshold_warning": 70, "threshold_critical": 90},
    {"metric": "memory_usage", "value": $MEM_USAGE, "unit": "%", "status": "$MEM_STATUS", "threshold_warning": 75, "threshold_critical": 90},
    {"metric": "disk_usage", "value": $DISK_USAGE, "unit": "%", "status": "$DISK_STATUS", "threshold_warning": 75, "threshold_critical": 90},
    {"metric": "load_ratio", "value": $LOAD_RATIO, "unit": "x", "status": "$LOAD_STATUS", "threshold_warning": 1.0, "threshold_critical": 2.0}
  ]
}
EOF
echo "Report saved to $OUTPUT_FILE"

Pattern 2: eBPF-Based Latency Investigation

# ❌ BAD: Using strace for performance investigation
# strace -p $(pgrep myapp) -e trace=network
# Problems:
# - High overhead (can slow process by 10-100x)
# - Not suitable for production
# - Generates massive output

# ✅ GOOD: Using eBPF for low-overhead tracing
# Install BCC tools
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

# Measure TCP connection latency with low overhead
sudo bpftrace -e '
tracepoint:tcp:tcp_set_state /args->state == 1/ {
    @connect_latency[tid] = nsecs;
}

tracepoint:tcp:tcp_rcv_space_adjust /@connect_latency[tid]/ {
    $latency = nsecs - @connect_latency[tid];
    printf("%d %d\n", tid, $latency);
    delete(@connect_latency[tid]);
}
'

# Profile file system latency
sudo bpftrace -e '
tracepoint:block:block_rq_issue {
    @start[args->dev, args->sector] = nsecs;
}

tracepoint:block:block_rq_complete /@start[args->dev, args->sector]/ {
    $latency = nsecs - @start[args->dev, args->sector];
    @latency_hist[args->dev] = hist($latency);
    delete(@start[args->dev, args->sector]);
}
'

# System-wide syscall latency histogram
sudo bpftrace -e '
kprobe:do_sys_open {
    @start[tid] = nsecs;
}

kretprobe:do_sys_open /@start[tid]/ {
    $latency = nsecs - @start[tid];
    @latency = hist($latency);
    delete(@start[tid]);
}
'

Pattern 3: Alert Threshold Configuration by Workload

Bash — Alert threshold configuration by workload

#!/bin/bash
# Display alert thresholds for a workload type
# Usage: ./alert-thresholds.sh <workload-type>
# Workload types: database, web-server

set -euo pipefail

WORKLOAD="${1:-}"
if [[ -z "$WORKLOAD" ]]; then
    echo "Usage: $0 {database|web-server}"
    exit 1
fi

echo "=== Alert Thresholds: ${WORKLOAD} ==="
echo ""

case "$WORKLOAD" in
    database)
        echo "# Database workload alert thresholds"
        echo "# File: /etc/prometheus/rules/database-alerts.yml"
        echo ""
        cat << 'RULES'
groups:
  - name: database-alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 5m
        labels: { severity: critical, workload: database }

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 5m
        labels: { severity: critical, workload: database }

      - alert: DiskIOSaturation
        expr: rate(node_disk_io_time_seconds_total[5m]) > 0.95
        for: 5m
        labels: { severity: critical, workload: database }

      - alert: DBConnectionsHigh
        expr: postgresql_connections_active > 200
        for: 1m
        labels: { severity: critical, workload: database }

      - alert: ReplicationLag
        expr: postgresql_replication_lag_seconds > 300
        for: 5m
        labels: { severity: critical, workload: database }
RULES
        echo ""
        echo "  CPU:    warning=70%, critical=90%"
        echo "  Memory: warning=75%, critical=90%"
        echo "  DiskIO: warning=80%, critical=95%"
        echo "  Conns:  warning=100, critical=200"
        echo "  RepLag: warning=30s, critical=300s"
        ;;

    web-server)
        echo "# Web server workload alert thresholds"
        echo "# File: /etc/prometheus/rules/web-server-alerts.yml"
        echo ""
        cat << 'RULES'
groups:
  - name: web-server-alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels: { severity: critical, workload: web-server }

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
        for: 5m
        labels: { severity: critical, workload: web-server }

      - alert: HighRequestLatency
        expr: histogram_quantile(0.50, rate(http_request_duration_seconds_bucket[5m])) > 5
        for: 5m
        labels: { severity: critical, workload: web-server }

      - alert: HighErrorRate
        expr: rate(http_request_errors_total[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 5m
        labels: { severity: critical, workload: web-server }

      - alert: HighNginxConnections
        expr: nginx_active_connections > 5000
        for: 1m
        labels: { severity: critical, workload: web-server }
RULES
        echo ""
        echo "  CPU:      warning=60%, critical=80%"
        echo "  Memory:   warning=70%, critical=85%"
        echo "  Latency:  warning=1.0s, critical=5.0s"
        echo "  ErrorRt:  warning=1.0%, critical=5.0%"
        echo "  Conns:    warning=1000, critical=5000"
        ;;

    *)
        echo "Unknown workload: $WORKLOAD"
        echo "Available: database, web-server"
        exit 1
        ;;
esac

Constraints

MUST DO

• MUST collect metrics at a consistent interval (5 minutes minimum, 1 minute for critical workloads) for accurate trend analysis
• MUST store metrics with timestamps and labels (hostname, workload, region) for proper filtering and correlation
• MUST set alert thresholds based on workload baselines, not arbitrary values — collect 2 weeks of baseline data before setting alerts
• MUST configure log rotation with size limits to prevent disk exhaustion from unbounded log growth
• MUST use eBPF-based tools (bcc, bpftrace, perf) for production performance investigation — they have minimal overhead compared to strace
• MUST document alert runbooks for each alert type — responders need clear steps to diagnose and remediate
• MUST test alerting delivery (email, Slack, PagerDuty) for each alert severity level
• MUST maintain a 30-day metrics retention window minimum for capacity planning trend analysis

MUST NOT DO

• MUST NOT set alert thresholds on the first day of deployment — allow at least 2 weeks of baseline data collection
• MUST NOT use strace or similar high-overhead tracing tools in production without explicit approval — use eBPF instead
• MUST NOT store all logs on the local disk indefinitely — forward to centralized log storage with defined retention
• MUST NOT ignore warning alerts — treat warnings as pre-critical signals that require investigation within SLA timeframes
• MUST NOT set the same alert thresholds for all workload types — databases, web servers, and batch jobs have fundamentally different profiles
• MUST NOT rely solely on cloud provider metrics for observability — deploy agent-based metrics collection for application-level visibility
• MUST NOT set CPU alert thresholds above 90% for production systems — sustained high CPU indicates capacity planning issues

Related Skills