Linux System Monitoring

System Monitoring

Distro support

Monitoring commands are identical on both families — top, htop, ss, vmstat, iostat, free, uptime, ps, sar behave the same. The only differences are how the tools are installed and a couple of package names. Body uses Debian/Ubuntu; the RHEL family (Fedora, RHEL, CentOS Stream, Rocky, Alma, Oracle) equivalents are in the matrix.

In sk-* scripts use the common.sh primitives (pkg_install, svc_name, firewall_allow) instead of hardcoding the family. See linux-bash-scripting and docs/multi-distro/plan.md.

Use when

• Checking overall host health across CPU, memory, disk, and network.
• Looking for warning signs before or during an incident.
• Verifying backup health as part of routine operations.

Do not use when

• The task is a specific incident diagnosis with a known symptom; use linux-troubleshooting.
• The task is telemetry system design rather than local host inspection; use linux-observability.

Required Inputs

Capability Contract

Default mode is read-only using process, kernel, socket, filesystem, and backup status commands. Installing tools, changing priorities/limits, killing processes, tuning kernel values, or restarting services requires explicit authority and a specialist handoff.

Degraded Mode

If tools, privileges, historical baselines, or repeated samples are unavailable, report the instantaneous measurements and missing context. Do not classify a transient snapshot as sustained health or pressure.

Decision Rules

Workflow

1. Confirm host role, clock, symptom, impact, and timeframe.
2. Capture a read-only broad snapshot of load, memory, I/O, sockets, filesystems, processes, and backup age.
3. Sample abnormal resources over time and compare with workload baseline.
4. Correlate metrics before selecting a failure domain; stop when a single snapshot cannot support causality.
5. Route to the owning skill with evidence rather than applying tuning.
6. If collection fails, recover by using available core commands and mark omitted checks unassessed.

Evidence Produced

Quality standards

• Start broad, then narrow based on evidence.
• Distinguish transient spikes from sustained pressure.
• End with a clear operational conclusion, not just raw command output.

Anti-patterns

• Tuning before a health snapshot. Fix: collect broad read-only evidence first.
• Treating one metric as the whole story. Fix: correlate load, CPU, memory, I/O, and impact.
• Ignoring backup health. Fix: record last successful backup against policy.
• Reading free as free-memory exhaustion. Fix: interpret available memory and reclaim behaviour.
• Calling one spike sustained pressure. Fix: sample across a defined interval.
• Killing a process during monitoring. Fix: hand remediation to an authorised specialist task.

Outputs

Worked Example

Load average is 12 on an eight-core host, but CPU idle remains 70% and I/O wait is 25%. Repeated vmstat/iostat samples show blocked tasks on one volume, so route to linux-disk-storage instead of changing CPU limits.

References

• references/monitoring-commands.md
• references/warning-signs.md

This skill is self-contained. Every command below is a standard tool on both families (Debian/Ubuntu and the RHEL family); only the install step and a couple of package names differ — see Distro support above. The sk-* scripts in the Optional fast path section are convenience wrappers — never required.

Quick Health Check

echo "=LOAD=" && uptime && \
echo "=MEMORY=" && free -h && \
echo "=DISK=" && df -h && \
echo "=SERVICES=" && \
  for s in nginx mysql php8.3-fpm apache2 fail2ban; do
    printf "%-20s %s\n" $s $(systemctl is-active $s 2>/dev/null)
  done && \
echo "=LAST BACKUP=" && ls -lt ~/backups/ 2>/dev/null | head -3

CPU & Load

uptime                    # load: 1m 5m 15m — concern if sustained > nproc
nproc                     # CPU core count
htop                      # P=CPU sort, M=memory sort, q=quit
top -bn1 | head -20       # non-interactive snapshot
ps aux --sort=-%cpu | head -10

Memory

free -h
# No swap = OOM kill fires when available → 0
ps aux --sort=-%mem | head -10

Disk I/O

iostat -x 1 5             # %util > 80% = bottleneck, await > 50ms = slow disk
sudo iotop -bod 5         # per-process I/O (requires: apt install iotop)

Network

ss -tunapl                # all connections with process
ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn   # count by state
ss -tlnp                  # listening services

Backup Health

crontab -l | grep -i backup                      # backup cron present?
find ~/backups -name "*.gpg" -mtime -3 2>/dev/null | wc -l  # backups in 3 days
rclone about gdrive: 2>/dev/null | head -2       # remote reachable?

Full command reference with output interpretation: references/monitoring-commands.md

Optional fast path (when sk-* scripts are installed)

Running sudo install-skills-bin linux-system-monitoring installs:

These wrap the manual commands above — optional.

Scripts

This skill installs the following scripts to /usr/local/bin/. To install:

sudo install-skills-bin linux-system-monitoring