Linux Troubleshooting

Troubleshooting

Distro support

Two-family skill. The diagnosis trees apply to both families; only service names, log paths, and a few tools differ — plus the RHEL family adds SELinux as a cause of failures that look like permission or connection bugs.

RHEL-family "it should work but doesn't": when unix permissions look correct but you still get 403 / EACCES / "connection refused" from a service (Apache 403, PHP can't reach the DB, a daemon won't bind a port), suspect SELinux before anything else:

sudo ausearch -m AVC -ts recent | audit2why     # what did SELinux block, and why
sudo getenforce                                  # Enforcing?

Fix with the right context/boolean/port — do not setenforce 0. See ../../07-security-and-hardening/linux-server-hardening/references/selinux-reference.md and docs/multi-distro/plan.md. In sk-* scripts resolve unit names via svc_name from common.sh.

Use when

• A server incident needs symptom-driven diagnosis.
• You know the symptom but not yet the owning subsystem.
• You need a structured triage flow before making changes.
• An unexplained outage needs the failure domain narrowed before remediation.
• Server access is unavailable and the operator needs a read-only diagnostic plan rather than an invented result.

Do not use when

• The problem is already clearly scoped to one specialist skill.
• The task is proactive monitoring or audit rather than incident response.

Required Inputs

Capability Contract

Default triage uses read/search access only. Restarts, deletes, rollbacks, package/config changes, database writes, firewall actions, and recovery operations require explicit authority after evidence identifies the failure mode.

Degraded Mode

Without logs, history, privileges, network probes, or reproduction, return the most specific supported failure-domain hypothesis and list unassessed branches. Do not close an incident because the symptom temporarily disappears.

Decision Rules

Workflow

After the immediate user-visible check, create an incident learning record when the symptom required intervention, repeated, or exposed a runbook gap. Preserve the timeline and before state, name the discriminating evidence, record unknowns, test one bounded countermeasure, and verify rollback/recovery before closure. Standardise the learning in the skill, diagnosis tree, script fixture, or monitoring rule only after the same failed path is exercised.

1. Establish impact, onset, host role, recent changes, and authority; start an evidence timeline.
2. Capture read-only system, socket, service, disk, memory, and relevant log state before intervention.
3. Select one symptom branch and test competing hypotheses with the cheapest discriminating checks.
4. Stop when evidence is insufficient or the branch requires specialist/destructive action.
5. Apply only the authorised minimal fix with explicit success and rollback checks.
6. On failure, recover the previous state, verify the user-visible symptom, and preserve residual evidence.

Evidence Produced

Quality standards

• Diagnose from evidence, not intuition.
• Separate triage from final remediation until the failure mode is clear.
• Keep the path short and explicit so incidents stay understandable under pressure.

Anti-patterns

• Restarting or deleting before triage. Fix: preserve a broad read-only snapshot first.
• Mixing symptom branches without reason. Fix: choose discriminating checks and record why the branch changed.
• Closing on a guess. Fix: repeat the user-visible and subsystem verification checks.
• Treating a recent deployment as proof. Fix: compare timestamps and test alternatives.
• Disabling SELinux to test permissions. Fix: inspect AVCs, labels, and expected policy.
• Applying several fixes at once. Fix: make one bounded change and checkpoint its effect.

Outputs

Worked Example

For a 502 after deployment, first capture proxy and upstream status, listeners, disk/memory, and matching logs. If Nginx is healthy but PHP-FPM is absent due to invalid config, validate that config and route the bounded service fix; do not restart the whole host.

References

• references/diagnosis-tree.md
• ../../docs/continuous-improvement/incident-learning-standard.md
• ../../docs/continuous-improvement/safe-reversible-operations-standard.md
• references/packet-capture-and-tracing.md — tcpdump packet capture (BPF filters, pcap, ring buffers) and strace/ltrace/lsof process & file diagnostics
• ../../07-security-and-hardening/linux-server-hardening/references/selinux-reference.md — SELinux as a hidden cause (RHEL family)

This skill is self-contained. Every command below works on a stock Debian/Ubuntu or RHEL-family server with no additional tooling. The sk-* scripts in the Optional fast path section at the bottom are convenience wrappers — they are never required.

Ask: "What's the symptom?" then follow the matching branch in references/diagnosis-tree.md.

Symptom Index

Full diagnosis commands for each: references/diagnosis-tree.md

Quick Triage (Run First For Any Issue)

# System health snapshot
uptime && free -h && df -h

# Failed services
sudo systemctl list-units --type=service --state=failed

# Recent errors across all services
sudo journalctl -p err --since "1 hour ago" --no-pager | head -30

# Nginx error log
sudo tail -20 /var/log/nginx/error.log

Most Common Fixes

# Service crashed → restart it
sudo systemctl restart <service>

# Nginx config broken → find and fix
sudo nginx -t

# Disk full → clear apt cache and vacuum journal
sudo apt clean && sudo journalctl --vacuum-size=500M

# 502 → restart the upstream
sudo systemctl restart php8.3-fpm
sudo systemctl restart apache2

# SSL expired → force renew
sudo certbot renew --force-renewal

Optional fast path (when sk-* scripts are installed)

Running sudo install-skills-bin linux-troubleshooting gives you interactive decision-tree walkers for each symptom:

These scripts wrap the manual commands above in a guided walkthrough. They are optional — the manual commands are always the source of truth.

Scripts

This skill installs the following scripts to /usr/local/bin/. To install:

sudo install-skills-bin linux-troubleshooting