Resource Management

Linux Resource Management with cgroups and systemd

Senior systems engineer configuring and managing Linux resources using cgroups v2, namespaces, and systemd to isolate workloads and guarantee resource allocation across cloud and on-prem environments.

TL;DR Checklist

• Verify cgroups v2 is enabled (cat /sys/fs/cgroup/cgroup.controllers)
• Assign workloads to dedicated cgroup hierarchies by service or tenant
• Set CPU limits using cpu.max (quota/period) and memory limits using memory.max
• Configure I/O bandwidth limits with io.max for block devices
• Apply systemd resource directives in unit files for managed services
• Set OOM scoring and memory pressure handlers for critical services
• Monitor resource utilization with systemd-cgtop and cgroup stat tools
• Validate resource isolation by running stress tests within cgroups

When to Use

Use this skill when:

• Multi-tenant workloads — You're running multiple applications or tenants on the same host and need resource isolation
• Resource guarantee requirements — A service must have a minimum CPU/memory allocation to meet SLA targets
• Preventing resource starvation — A noisy-neighbor workload is consuming all available resources and starving others
• Container host configuration — You're configuring the underlying host for container runtimes that rely on cgroups
• Batch job management — You need to cap resource usage for intermittent batch or compute workloads

When NOT to Use

Avoid this skill for:

• Single workload on dedicated host — If one application owns the entire server, cgroup overhead adds complexity without benefit
• Container-only environments — Let the container runtime (Docker, containerd, podman) manage cgroups; configure the runtime instead
• Soft resource preferences — If a workload should prefer more resources but not be capped, use CPU priority (cpu.weight) instead of hard limits
• Kernel version < 5.7 without cgroups v1 fallback — cgroups v2 is required for unified resource control; verify your kernel supports it

Use linux-services when you need to manage service lifecycle (start/stop/restart) alongside resource constraints. Use kernel-tuning when the issue is kernel parameter behavior rather than resource allocation.

Core Workflow

1. Verify cgroups v2 Support

Check that the unified cgroups hierarchy is active.

# Verify cgroups v2 is mounted and active
stat -fc %T /sys/fs/cgroup/

# Check available controllers
cat /sys/fs/cgroup/cgroup.controllers

# Enable missing controllers
echo "+cpu +memory +io" | sudo tee /sys/fs/cgroup/cgroup.subtree_control

Checkpoint: cgroups v2 is mounted (type cgroup2) and all required controllers (cpu, memory, io) are listed in cgroup.subtree_control.

2. Create Cgroup Hierarchy

Build a hierarchical cgroup structure matching your workload topology.

# Create hierarchy for multi-workload host
sudo mkdir -p /sys/fs/cgroup/workloads/{web,db,batch}

# Assign CPU and memory limits to each workload
echo "50000 100000" | sudo tee /sys/fs/cgroup/workloads/web/cpu.max
# 50ms CPU quota per 100ms period = 50% of one core

echo "2G" | sudo tee /sys/fs/cgroup/workloads/web/memory.max

# Database gets higher limits
echo "100000 100000" | sudo tee /sys/fs/cgroup/workloads/db/cpu.max
echo "8G" | sudo tee /sys/fs/cgroup/workloads/db/memory.max

# Batch job gets lower priority with OOM protection
echo "25000 100000" | sudo tee /sys/fs/cgroup/workloads/batch/cpu.max
echo "4G" | sudo tee /sys/fs/cgroup/workloads/batch/memory.max

Checkpoint: Each workload has its own cgroup directory with CPU and memory limits set. Hierarchy reflects workload criticality.

3. Configure I/O Control

Set I/O bandwidth limits to prevent disk contention between workloads.

# Check which block devices are tracked
cat /sys/fs/cgroup/io.stat

# Get major:minor of device
ls -l /dev/sda

# Set I/O limits: max 100MB/s read, 50MB/s write for batch
echo "8:0 rbps=104857600 wbps=52428800" | sudo tee /sys/fs/cgroup/workloads/batch/io.max

# Database gets higher I/O priority with higher limits
echo "8:0 rbps=524288000 wbps=262144000" | sudo tee /sys/fs/cgroup/workloads/db/io.max

# Set I/O weight (fair-share priority): default=100, range=1-10000
echo "500" | sudo tee /sys/fs/cgroup/workloads/db/io.weight
echo "100" | sudo tee /sys/fs/cgroup/workloads/batch/io.weight

Checkpoint: I/O limits are set per device and workloads have proportional weights reflecting their priority.

4. Integrate with systemd

For services managed by systemd, use resource directives in unit files instead of raw cgroup manipulation.

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application Service
After=network.target

[Service]
Type=simple
ExecStart=/opt/myapp/server
User=myapp

# CPU limits: 2 cores max, 1 core minimum with weight-based sharing
CPUQuota=200%
CPUWeight=500

# Memory limits with OOM protection
MemoryMax=8G
MemoryHigh=7G
OOMPolicy=stop

# I/O weight and bandwidth
IOWeight=500
IOReadBandwidthMax=/dev/sda 500M
IOWriteBandwidthMax=/dev/sda 250M

# Restart policy
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Checkpoint: Unit file is created, reloaded (systemctl daemon-reload), and the service starts with resource constraints applied.

5. Configure OOM Handling

Set OOM scoring and handling for critical workloads.

# OOM score adj: lower = less likely to be killed (range -1000 to 1000)
echo "-500" | sudo tee /sys/fs/cgroup/workloads/db/cgroup.oom压力_handler
# Note: use systemd OOMPolicy for managed services instead

# Set up memory pressure monitoring
cat > /usr/local/bin/cgroup-oom-handler << 'HANDLER'
#!/bin/bash
# Handler for OOM events in cgroup
echo "$(date) OOM event in $1" >> /var/log/cgroup-oom.log
systemctl restart "$1"
HANDLER
chmod +x /usr/local/bin/cgroup-oom-handler

Checkpoint: Critical services have OOM policies that prevent cascading failures. Database uses OOMPolicy=stop, non-critical services use OOMPolicy=kill.

6. Monitor and Validate

Verify resource isolation with monitoring tools.

# Real-time cgroup monitoring
systemd-cgtop

# Read cgroup resource statistics
cat /sys/fs/cgroup/workloads/web/cpu.stat
cat /sys/fs/cgroup/workloads/web/memory.stat

# Check current limits vs usage
systemctl status myapp.service | grep -A 5 "Memory"

# Stress test within cgroup limits
sudo systemd-run --scope -p MemoryMax=1G -p CPUQuota=50% stress-ng --vm 2 --vm-bytes 500M

Checkpoint: Monitoring confirms workloads respect their limits. Stress tests confirm cgroup enforcement and no cross-workload resource leakage.

Implementation Patterns

Pattern 1: Cgroup v2 Resource Limits (BAD vs. GOOD)

BAD — cgroups v1 mixed hierarchy with manual process management

# ❌ BAD: cgroups v1 split hierarchy, manual process tracking
mkdir -p /sys/fs/cgroup/cpu/web
mkdir -p /sys/fs/cgroup/memory/web
echo "50000" > /sys/fs/cgroup/cpu/web/cpu.cfs_quota_us
echo "100000" > /sys/fs/cgroup/cpu/web/cpu.cfs_period_us
echo "2147483648" > /sys/fs/cgroup/memory/web/memory.limit_in_bytes

# Manual process tracking (breaks when process forks/execs)
echo $PID > /sys/fs/cgroup/cpu/web/tasks
echo $PID > /sys/fs/cgroup/memory/web/tasks

# Problems:
# - Split hierarchy: CPU and memory in separate cgroups (no unified control)
# - Memory limit in bytes, not human-readable (2147483648 = 2G)
# - Manual PID management misses child processes
# - No I/O control integration

GOOD — cgroups v2 unified hierarchy with systemd integration

# ✅ GOOD: systemd-managed cgroups v2 with unified resource control
# File: /etc/systemd/system/webapp.service

[Service]
# CPU: 2 cores maximum, fair-share weight of 500 (default 100)
CPUQuota=200%
CPUWeight=500

# Memory: hard limit 4G, soft limit 3G (throttle before hard limit)
MemoryMax=4G
MemoryHigh=3G

# I/O: weight 500, bandwidth cap 200MB/s read, 100MB/s write
IOWeight=500
IOReadBandwidthMax=/dev/sda 200M
IOWriteBandwidthMax=/dev/sda 100M

# OOM handling: stop service instead of killing individual processes
OOMPolicy=stop

Pattern 2: Resource Templates for Common Workloads

Bash — Resource templates for common workload types

#!/bin/bash
# Generate systemd service with resource constraints for common workload types
# Usage: ./generate-resource-service.sh <profile-name> <service-name> [exec-cmd]
# Profiles: web-server, database, batch-job, monitoring

set -euo pipefail

PROFILE="${1:-}"
SERVICE_NAME="${2:-}"
EXEC_CMD="${3:-/opt/${SERVICE_NAME:-webapp}/server}"

if [[ -z "$PROFILE" || -z "$SERVICE_NAME" ]]; then
    echo "Usage: $0 <profile-name> <service-name> [exec-cmd]"
    echo "Profiles: web-server, database, batch-job, monitoring"
    exit 1
fi

SERVICE_FILE="/etc/systemd/system/${SERVICE_NAME}.service"

# Define resource profiles
case "$PROFILE" in
    web-server)
        CPU_QUOTA="100%"
        CPU_WEIGHT="200"
        MEM_MAX="2G"
        MEM_HIGH="1.5G"
        IO_WEIGHT="200"
        OOM_POLICY="restart"
        DESCRIPTION="Web application server"
        ;;
    database)
        CPU_QUOTA="400%"
        CPU_WEIGHT="500"
        MEM_MAX="16G"
        MEM_HIGH="12G"
        IO_WEIGHT="500"
        OOM_POLICY="stop"
        DESCRIPTION="Database server"
        ;;
    batch-job)
        CPU_QUOTA="200%"
        CPU_WEIGHT="50"
        MEM_MAX="4G"
        MEM_HIGH=""
        IO_WEIGHT="50"
        OOM_POLICY="kill"
        DESCRIPTION="Batch processing workload"
        ;;
    monitoring)
        CPU_QUOTA="50%"
        CPU_WEIGHT="50"
        MEM_MAX="512M"
        MEM_HIGH="256M"
        IO_WEIGHT="50"
        OOM_POLICY="restart"
        DESCRIPTION="Monitoring and telemetry agent"
        ;;
    *)
        echo "Unknown profile: $PROFILE"
        echo "Available: web-server, database, batch-job, monitoring"
        exit 1
        ;;
esac

cat > "$SERVICE_FILE" << EOF
[Unit]
Description=${DESCRIPTION}
After=network.target

[Service]
Type=simple
ExecStart=${EXEC_CMD}

# CPU limits
CPUQuota=${CPU_QUOTA}
CPUWeight=${CPU_WEIGHT}

# Memory limits
MemoryMax=${MEM_MAX}
EOF

if [[ -n "$MEM_HIGH" ]]; then
    echo "MemoryHigh=${MEM_HIGH}" >> "$SERVICE_FILE"
fi

cat >> "$SERVICE_FILE" << EOF

# I/O weight
IOWeight=${IO_WEIGHT}

# OOM handling
OOMPolicy=${OOM_POLICY}
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

echo "Generated ${SERVICE_FILE}"
echo "Run: systemctl daemon-reload && systemctl enable --now ${SERVICE_NAME}"

Pattern 3: Cloud Instance Resource Awareness

Bash — Detect cloud instance type and suggest resource limits

#!/bin/bash
# Detect cloud instance type and suggest appropriate resource limits
# Usage: ./detect-instance.sh [--suggest]
set -euo pipefail

SUGGEST=false
if [[ "${1:-}" == "--suggest" ]]; then
    SUGGEST=true
fi

# Detect cloud provider
detect_provider() {
    # AWS
    if curl -s -m 2 http://169.254.169.254/latest/meta-data/instance-type 2>/dev/null | grep -q '^i-'; then
        local inst_type
        inst_type=$(curl -s -m 2 http://169.254.169.254/latest/meta-data/instance-type)
        echo "aws|$inst_type"
        return
    fi
    # GCP
    if curl -s -m 2 -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/machine-type 2>/dev/null | grep -q 'n1-'; then
        local inst_type
        inst_type=$(curl -s -m 2 -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/machine-type)
        inst_type=$(basename "$inst_type")
        echo "gcp|$inst_type"
        return
    fi
    # Azure
    if curl -s -m 2 -H "Metadata-Severity: Interactive" "http://169.254.169.254/metadata/instance?api-version=2021-02-01" 2>/dev/null | grep -q '"compute"'; then
        local inst_type
        inst_type=$(curl -s -m 2 -H "Metadata-Severity: Interactive" "http://169.254.169.254/metadata/instance?api-version=2021-02-01" | jq -r '.compute.vmSize')
        echo "azure|$inst_type"
        return
    fi
    echo "bare-metal|unknown"
}

# Get actual vCPU count
get_vcpus() {
    nproc 2>/dev/null || echo 1
}

# Get memory in GB
get_memory_gb() {
    local kb
    kb=$(awk '/^MemTotal:/{print $2}' /proc/meminfo)
    echo "scale=0; $kb / 1024 / 1024" | bc
}

# Main detection
read -r PROVIDER INST_TYPE <<< "$(detect_provider | tr '|' ' ')"
VCPUS=$(get_vcpus)
MEMORY_GB=$(get_memory_gb)

echo "=== Instance Detection ==="
echo "Provider:   $PROVIDER"
echo "Type:       $INST_TYPE"
echo "vCPUs:      $VCPUS"
echo "Memory:     ${MEMORY_GB}GB"
echo ""

if [[ "$SUGGEST" == "true" ]]; then
    echo "=== Suggested Resource Limits ==="
    # Reserve 25% vCPU for OS/hypervisor
    APP_VCPUS=$((VCPUS * 75 / 100))
    # Reserve 15% memory for OS/hypervisor
    APP_MEM=$((MEMORY_GB * 85 / 100))
    APP_MEM_HIGH=$((APP_MEM * 80 / 100))

    echo "CPUQuota=${APP_VCPUS}00%"
    echo "MemoryMax=${APP_MEM}G"
    echo "MemoryHigh=${APP_MEM_HIGH}G"
    echo ""
    echo "Note: Reserve 25% vCPU and 15% memory for OS/hypervisor overhead"
    echo "On bare metal, adjust reserves to 10% vCPU and 10% memory"
fi

if [[ "$PROVIDER" == "bare-metal" ]]; then
    echo ""
    echo "Bare metal detected — use manual resource configuration"
    echo "Recommended: CPUQuota=400%, MemoryMax=80% of total, MemoryHigh=80% of Max"
fi

Constraints

MUST DO

• MUST verify cgroups v2 is active before configuring — check /sys/fs/cgroup/cgroup.controllers exists and is type cgroup2
• MUST set memory limits on all workloads that could grow unbounded — prevent OOM killer from terminating critical services
• MUST use systemd resource directives for managed services instead of raw cgroup manipulation — systemd manages process tracking and hierarchy
• MUST set MemoryHigh (soft limit) below MemoryMax (hard limit) to enable memory reclaim pressure before hard OOM
• MUST set CPU quota high enough for the workload's peak, not just average — CPU throttling is visible as latency spikes
• MUST test resource isolation with stress tests to confirm limits are enforced and no cross-workload leakage occurs
• MUST account for hypervisor overhead when sizing cloud instances — the host OS and hypervisor consume resources not available to cgroups

MUST NOT DO

• MUST NOT set CPU quota to 0 — this effectively freezes the cgroup and kills all processes inside it
• MUST NOT set memory.max below the application's minimum working set — it will OOM immediately on startup
• MUST NOT manage cgroup tasks manually when using systemd — let systemd track processes via the cgroup hierarchy
• MUST NOT apply the same resource limits to all workloads — database, web, and batch workloads have fundamentally different resource profiles
• MUST NOT forget to set I/O limits on shared-storage hosts — a single noisy-neighbor can degrade all workloads' I/O performance
• MUST NOT use cgroups v1 split hierarchy on new deployments — cgroups v1 is deprecated and lacks unified resource control

Related Skills