Security Audit

Linux Breach Audit

Operating Posture

Treat the target as hostile until proven otherwise. Prefer a trusted rescue/live environment over booting the installed OS. If the installed OS is already booted, minimize dwell time and avoid typing new secrets into it. A read-only audit must not stop, restart, disable, or reconfigure production services. Containment requires explicit authorization for the exact service and an impact check.

Do not make destructive changes without explicit user confirmation. Before removing keys, users, services, cron jobs, binaries, or containers, preserve the original file or command output in a timestamped evidence directory such as /root/incident-audit-YYYYmmdd-HHMMSS or an external trusted location. If working from rescue media, keep suspect filesystems mounted read-only for triage; remount read-write only after the user approves a specific remediation.

Quick Workflow

1. Establish access through the least-trusting path available: rescue/live boot, initramfs shell, GRUB single-user, VNC console, then SSH only if necessary.
2. Record host context and evidence location.
3. Audit current effective access first: SSH keys, users, sudoers, sshd rules, shell startup files, and any provider/cloud-init access paths before adding any new access.
4. Audit persistence: cron, systemd, timers, rc scripts, containers, package hooks, and cloud-init.
5. Audit processes, network listeners, and active outbound connections if the installed OS is running.
6. Inventory process managers for every service user and inspect reverse tunnels, provider-managed ingress routes, and shared-user privilege boundaries.
7. Inspect temp and writable execution paths: /tmp, /var/tmp, /dev/shm, user homes, /run, and hidden directories.
8. Report findings, classify each item, and recommend exact remediation steps. Remove unauthorized access only after confirmation.

Current Access Priority

When auditing a likely compromised machine, prioritize "who can access this host now" before reconstructing the original breach path. Treat suspected source systems, peer hosts, VPNs, CI/CD systems, and jump boxes as hypotheses for access review rather than as the main objective unless the user explicitly asks for attribution.

Build a current-access inventory:

• Local login-capable users and UID 0 users.
• Users with sudo, admin, wheel, docker, lxd, libvirt, or other privilege-escalating groups.
• SSH authorized keys per user, including key comments and restrictive options.
• Alternate sshd AuthorizedKeysFile paths and Match rules.
• Password login, root login, PAM, MFA, LDAP/SSSD, cloud-init, or provider-injected keys.
• Cron/systemd/container jobs that can recreate users, keys, listeners, or reverse shells.

Clean up access by whitelisting known-good users/keys and removing or disabling everything else only after preserving originals and getting user confirmation. Do not spend time proving exact attacker breadcrumbs unless that evidence affects immediate access control, containment, or data recovery.

Evidence Setup

Use a timestamped evidence directory on a trusted writable filesystem when available:

ts="$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "/root/incident-audit-$ts"
chmod 700 "/root/incident-audit-$ts"

Create report files with mode 600. Capture command output with tee only when safe. Do not collect full process environments, process-manager dumps, private keys, tokens, or database credentials into a general report. Summarize sensitive files by path, owner, permissions, hash, and suspicious lines.

Access And Mount Safety

For rescue/live boots, identify disks first:

lsblk -f
blkid
findmnt

Mount suspect filesystems read-only for triage:

mkdir -p /mnt/suspect-root
mount -o ro,noload /dev/<root-partition> /mnt/suspect-root

For LVM, mdraid, or ZFS, activate/import read-only where possible and note any limitations. Avoid running binaries from the suspect filesystem. Prefer host rescue tools, static tools, or commands that read files as data.

SSH Keys And Access

Audit these before granting new access:

find /root /home -maxdepth 3 -name authorized_keys -o -name authorized_keys2 2>/dev/null
find /root /home -maxdepth 3 \( -name .ssh -o -name authorized_keys \) -exec stat {} + 2>/dev/null
find /root /home -path '*/.ssh/*' -maxdepth 4 -type f -ls 2>/dev/null

Check every authorized_keys line for unexpected principals, comments, source restrictions, forced commands, and forwarding options. Preserve full original files before editing. When removing bad actor keys, write a clean replacement file atomically, then enforce ownership and permissions:

install -d -m 700 -o <user> -g <group> <home-dir>/.ssh
install -m 600 -o <user> -g <group> /tmp/clean-authorized_keys <home-dir>/.ssh/authorized_keys

Also audit SSH daemon configuration:

grep -RInE '^(Include|AuthorizedKeysFile|PermitRootLogin|PasswordAuthentication|PubkeyAuthentication|Match|ForceCommand|AllowUsers|DenyUsers)' /etc/ssh /usr/local/etc/ssh 2>/dev/null

Red flags include alternate AuthorizedKeysFile paths, permissive root/password login, broad Match blocks, unexpected ForceCommand, and backdoor users in AllowUsers.

Users And Privileges

Check:

cat /etc/passwd
cat /etc/group
getent passwd
getent group sudo wheel admin
grep -RIn . /etc/sudoers /etc/sudoers.d 2>/dev/null
last -n 50
lastlog

Flag unexpected UID 0 accounts, new interactive users, service users with shells, sudoers additions, passwordless sudo, and recently active unknown accounts. Preserve /etc/passwd, /etc/shadow, /etc/group, and sudoers files before any remediation.

Also identify public services sharing a Unix account with validators, signing keys, process managers, or privileged operational files. A public app compromise inherits everything readable, writable, or controllable by that UID even when it cannot become root.

Shell Startup Files

Inspect global and per-user shell startup files:

find /etc/profile /etc/profile.d /etc/bashrc /etc/bash.bashrc /root /home -maxdepth 3 \
  \( -name '.profile' -o -name '.bashrc' -o -name '.bash_profile' -o -name '.bash_aliases' -o -name '.zshrc' \) -ls 2>/dev/null

Red flags include curl|wget piped to shell, nc -e, reverse shells, base64 decode plus exec, PATH hijacks, LD_PRELOAD, history suppression, aliases hiding commands, or outbound callbacks.

Persistence

Audit cron:

grep -RIn . /etc/crontab /etc/cron.d /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly 2>/dev/null
for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u "$u" 2>/dev/null | sed "s/^/[$u] /"; done

Audit systemd:

systemctl list-unit-files --state=enabled
systemctl list-timers --all
find /etc/systemd/system /lib/systemd/system /usr/lib/systemd/system /root/.config/systemd -type f -maxdepth 5 -ls 2>/dev/null
find /home -maxdepth 5 -path '*/.config/systemd/*' -type f -ls 2>/dev/null
grep -RInE '(curl|wget|bash -c|/tmp|/var/tmp|/dev/shm|base64|nc |socat|python.*socket|ExecStart)' /etc/systemd/system /lib/systemd/system /usr/lib/systemd/system 2>/dev/null

Also check /etc/rc.local, /etc/init.d, /etc/ld.so.preload, /etc/ld.so.conf.d, /etc/apt/apt.conf.d, /etc/update-motd.d, /var/spool/cron, /var/spool/at, and cloud-init user-data or scripts.

Processes, Network, And Containers

If the installed OS is running, collect:

ps auxf
ps -eo pid,ppid,user,etime,lstart,cmd --sort=etime
ss -tulpen
ss -tpn
lsof -i -n -P
nft list ruleset 2>/dev/null || iptables -L -n -v

For suspicious PIDs, inspect /proc/<pid>/exe, /proc/<pid>/cwd, /proc/<pid>/cmdline, and /proc/<pid>/environ without pasting secrets into chat.

Inventory process managers under every login or service user, including user-scoped systemd and PM2 homes. Inspect outbound tunnels and reverse proxies, then obtain provider-side route inventory before concluding what they expose. A home NAT boundary does not protect routes published through an outbound tunnel.

Check containers because compromise often persists there:

docker ps -a
docker images
docker inspect <container>
docker compose ls
podman ps -a 2>/dev/null

Flag unexpected listeners, public management ports, outbound connections to unfamiliar IPs, containers with host mounts, privileged containers, and processes executing from temp paths.

For databases, assess four layers separately: database bind/listen settings, authentication rules such as pg_hba.conf, host firewall policy, and upstream router/NAT or tunnel publication. Do not infer Internet exposure or safety from only one layer.

During remediation, remove the narrowest confirmed route or service. Do not terminate a whole shared tunnel without a complete route inventory and explicit acceptance of the impact. Preserve critical production PIDs and verify them before and after each change.

Temp And Writable Execution Paths

Inspect /tmp, /var/tmp, /dev/shm, /run, user writable bins, and hidden home directories:

find /tmp /var/tmp /dev/shm /run -xdev -type f -printf '%TY-%Tm-%Td %TH:%TM %m %u %g %s %p\n' 2>/dev/null | sort
find /tmp /var/tmp /dev/shm -xdev -type f \( -perm -111 -o -name '*.sh' -o -name '*.py' -o -name '*.pl' \) -ls 2>/dev/null
find /root /home -maxdepth 3 -type d -name '.*' -ls 2>/dev/null

Red flags include random-looking names, executable files in temp dirs, hidden directories containing binaries, recently modified scripts, and files owned by unexpected users.

Report Format

Lead with the security-relevant conclusion and current exposure state.

End with:

• Overall verdict: OK, Monitor, Investigate, or Compromised.
• Exact remediation already performed, if any.
• Exact remediation still recommended.
• Any checks blocked by permissions, rescue environment limits, missing tools, or encrypted/locked volumes.