Linux Log Management

Log Management

Distro support

journalctl and the systemd journal are identical on both families. The differences are the legacy /var/log/* text-file names and the package manager. The body uses Debian/Ubuntu paths; the RHEL family (Fedora, RHEL, CentOS Stream, Rocky, Alma, Oracle) equivalents are in the matrix.

RHEL-family note: minimal RHEL/Fedora installs may not ship rsyslog, so the legacy /var/log/* files may be absent — the journal (journalctl) is the primary source. journalctl -u <unit>, -p err, --since all work the same.

In sk-* scripts, prefer journalctl (portable) and resolve unit names via svc_name from common.sh. See linux-bash-scripting and docs/multi-distro/plan.md.

Use when

• Reading service, web, security, or database logs on a Linux server.
• Investigating spikes in errors, attacks, or slow queries.
• Reviewing log rotation or journal storage behavior.

Do not use when

• The task is general incident routing without a clear symptom; use linux-troubleshooting.
• The task is metrics collection or centralized observability setup; use linux-observability.

Required Inputs

Capability Contract

Read/search access to journals and log files is sufficient for analysis. Rotation changes, vacuuming, deletion, permission changes, and service reloads require explicit authority; shared evidence must redact secrets and personal data.

Degraded Mode

Fallback when required access is unavailable: report the exact time/source coverage and mark missing periods unassessed. Never equate absent or rotated logs with absence of an event.

Decision Rules

Workflow

1. Confirm service, host, timezone, time window, retention, and read/search boundary.
2. Preserve a bounded evidence slice before changing rotation or services.
3. Correlate events across the minimum relevant sources and separate facts from inference.
4. Route the cause to the owning service; stop when evidence cannot distinguish candidates.
5. Apply only authorised rotation/retention changes and validate syntax.
6. On failure, recover the prior logrotate configuration and retain collected evidence.

Evidence Produced

Quality standards

• Use time-bounded inspection instead of dumping entire logs.
• Prefer concrete log evidence over speculation.
• Keep rotation and retention changes deliberate.

Anti-patterns

• Grepping random logs. Fix: identify service, host, timezone, and timeframe first.
• Treating volume as proof of cause. Fix: correlate content with service behaviour.
• Changing rotation before understanding growth. Fix: measure source and preserve the incident window.
• Publishing raw secrets or tokens. Fix: redact values and retain only useful context.
• Treating rotated logs as proof nothing happened. Fix: mark the missing period unassessed.
• Mixing local and UTC timestamps silently. Fix: normalise or label timezone offsets.

Outputs

Worked Example

For an Nginx 502 spike from 14:00–14:10 EAT, extract only that access/error interval, correlate upstream errors with PHP-FPM journal entries, and preserve request IDs. If older logs rotated, mark earlier onset unassessed rather than declaring 14:00 the start.

References

• ../../docs/continuous-improvement/incident-learning-standard.md

• references/journalctl-reference.md

• references/log-analysis-patterns.md

• references/log-locations.md

This skill is self-contained. Every command below is a standard tool on both the Debian/Ubuntu and RHEL families (only the legacy /var/log/* paths differ — see Distro support above). The sk-* scripts in the Optional fast path section are convenience wrappers — never required.

journalctl

sudo journalctl -u <service> -n 50 --no-pager       # last 50 lines
sudo journalctl -u <service> -f                      # follow live
sudo journalctl -u <service> --since "1 hour ago"
sudo journalctl -p err --since "today" --no-pager    # errors only
sudo journalctl -k --since "today" | grep -i oom     # kernel OOM events
sudo journalctl --disk-usage                         # journal size

Nginx Logs

sudo tail -f /var/log/nginx/error.log
sudo tail -f /var/log/nginx/access.log

# HTTP status code distribution:
sudo awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

# Top IPs by request count:
sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# Recent 5xx errors:
sudo grep '" 5' /var/log/nginx/access.log | tail -20

Attack Pattern Detection

# Login brute-force attempts:
sudo grep -E "POST.*(login|wp-login|admin|xmlrpc)" /var/log/nginx/access.log | \
    awk '{print $1}' | sort | uniq -c | sort -rn | head

# Scanner activity (high 404 rate per IP):
sudo awk '$9 == 404 {print $1}' /var/log/nginx/access.log | \
    sort | uniq -c | sort -rn | head

# Attempts to access sensitive files:
sudo grep -E "\.(env|git|htaccess|sql|bak)" /var/log/nginx/access.log | tail -20

fail2ban Log

sudo tail -f /var/log/fail2ban.log
sudo grep "Ban" /var/log/fail2ban.log | tail -20
sudo grep "$(date '+%Y-%m-%d')" /var/log/fail2ban.log | grep "Ban" | wc -l

Other Key Logs

# PHP errors:
sudo tail -f /var/log/php8.3-fpm.log

# MySQL slow queries:
sudo tail -20 /var/log/mysql/mysql-slow.log 2>/dev/null
mysql -e "SHOW VARIABLES LIKE 'slow_query_log%';" 2>/dev/null

# Apache (port 8080 backend):
sudo tail -f /var/log/apache2/error.log

# Backup cron:
tail -50 ~/backups/mysql/cron.log

logrotate

ls /etc/logrotate.d/                             # existing configs
sudo logrotate -f /etc/logrotate.d/nginx         # force rotate now
sudo logrotate -f /etc/logrotate.d/apache2

All log file locations: references/log-locations.md

Optional fast path (when sk-* scripts are installed)

Running sudo install-skills-bin linux-log-management installs:

These are optional wrappers around the manual commands above.

Scripts

This skill installs the following scripts to /usr/local/bin/. To install:

sudo install-skills-bin linux-log-management