Incident Response
A practical IR runbook for small teams and solo operators. Built around the SANS PICERL phases: Preparation → Identification → Containment → Eradication → Recovery → Lessons Learned.
When to invoke
You are in incident territory if any of these are true:
- A site you operate is defaced, redirecting, or serving content you did not publish
- A webshell or backdoor file has been found
- An admin/root account appeared that nobody on the team created
- A hosting provider, registrar, or upstream sent an abuse notice
- A credential you control turned up in a public dump
- Traffic is being dropped from your IP by major email/DNS providers (sign of being on a blocklist)
If you only suspect something is wrong, treat it as a real incident until you have evidence otherwise. False positives are cheap; missed compromises are expensive.
Phase 1 — Preparation (before anything happens)
Have these ready before the first incident:
- Out-of-band contact channel (Signal/Telegram/phone) — your primary work tool may itself be compromised
- Backup of last-known-good for every production system, verified restorable
- Inventory of who has access to what (hosting, DNS, registrar, repo, secrets store)
- A pre-written holding statement for customers ("we are investigating an issue affecting X, more soon")
- A way to reach your registrar fast — domain hijacks are catastrophic and the registrar is often slowest in real time
Phase 2 — Identification
Goal: confirm a real incident and scope it.
# Webserver: who is hitting unusual paths?
sudo tail -n 10000 /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30
# Auth log: failed + successful root/SSH activity
sudo grep -E 'Failed password|Accepted' /var/log/auth.log | tail -100
# Currently logged-in users + open sessions
who -a
last -n 30
# Listening processes — anything new?
sudo ss -tlnp
# Crontabs at every level
for u in $(cut -d: -f1 /etc/passwd); do sudo crontab -u "$u" -l 2>/dev/null | sed "s|^|[$u] |"; done
sudo ls -la /etc/cron.* /var/spool/cron/
# Recently modified files in sensitive trees
sudo find /var/www /etc /usr/local/bin -mtime -30 -type f 2>/dev/null
Document everything you find as you find it. Timestamps in UTC. Append-only — never edit prior entries.
Phase 3 — Containment
The aim is to stop the bleeding without destroying evidence. Do not just delete malicious files immediately; snapshot first.
Short-term containment
- Snapshot the affected filesystem (
tar, hosting-panel snapshot, VM snapshot) and the database. Store off-host.
- Pull the site off the internet at the cheapest layer that works:
- Cloudflare proxy → site-wide maintenance page (rule)
- Webserver → return 503 from the vhost
- Last resort → stop the service
- Rotate the credentials you know are exposed — keep a list of every credential you rotate, when, and by whom.
- Revoke active sessions on admin panels and force re-login.
- Block known-bad IPs at the edge if you have a clear indicator.
Long-term containment (hours/days)
- Move the service to a clean host before recovery if root cause is unclear
- Add WAF rules / CDN rules for the attack pattern
- If lateral movement is plausible (shared hosting, shared SSH keys, shared CI), assume every adjacent system is at risk and apply the same containment there
Phase 4 — Eradication
You cannot eradicate what you have not understood. Before you wipe and reinstall, answer:
- Entry vector — how did the attacker get in?
- Foothold — what did they install (webshell, cron, systemd unit, modified binary, DB-resident backdoor)?
- Persistence — would a reboot or a file-restore alone evict them?
- Lateral movement — did they reach other systems?
Only then:
- Rebuild from known-good (golden image, fresh OS install, fresh CMS install).
- Restore data from a backup taken before the foothold timestamp. If you cannot determine when the foothold was placed, treat all backups as suspect and restore data selectively (export rows, not full dumps).
- Patch the entry vector specifically. If you do not know the vector, you have not finished investigating.
- Re-issue all secrets that touched the compromised system.
Phase 5 — Recovery
- Bring the service back up in a staged way — internal first, then a small percentage of traffic, then full.
- Watch the logs you set up in Phase 2 for repeat indicators.
- Keep elevated monitoring for at least 30 days. Attackers often return.
- Notify users if their data may have been exposed. Be honest about what you know and do not know. In most jurisdictions personal-data breaches have legal notification deadlines (e.g. 72 hours under GDPR Art. 33) — check your jurisdiction.
Phase 6 — Lessons learned
Write the post-mortem within a week, while the details are fresh. Structure:
# Post-Mortem — <incident name> — <date>
## Summary
One paragraph: what happened, impact, current state.
## Timeline (UTC)
- 2026-MM-DD HH:MM — first indicator observed
- 2026-MM-DD HH:MM — confirmed incident, containment started
- ...
## Impact
- Systems affected
- Data potentially exposed
- Users notified, regulators notified, etc.
## Root cause
The single technical reason this was possible. Be specific.
## Contributing factors
Process / monitoring / staffing gaps that let it reach production or stay undetected.
## What worked
Things to keep doing.
## Action items
Each with an owner and a due date. Track them somewhere durable (issue tracker).
## Indicators of compromise (IOCs)
File hashes, IPs, user-agents, domain names. Keep these — they are useful for next time.
The post-mortem is for learning, not blame. Name what failed; do not name who failed.
A note on legality and ethics
- Only run this against systems you own or have explicit written authorization for.
- Notify affected users where required by law and where it is the right thing to do regardless.
- Do not "hack back" — it is illegal in most jurisdictions and unhelpful in all of them.
- Preserve evidence in a way that would stand up if law enforcement gets involved.
What this skill will not do
- It will not help you cover up a breach or delay required notifications.
- It will not perform offensive actions against any system, including ones attacking you.
- It is not a substitute for a qualified IR firm for serious incidents. Know your limits and call for help.
1---
2name: incident-response
3description: Run a structured response to a suspected web or server compromise. Follows SANS PICERL — Preparation, Identification, Containment, Eradication, Recovery, Lessons Learned — and includes a post-mortem template. Invoke when a site is defaced, when malware or webshells appear, when admin accounts arrive unannounced, or when a provider sends an abuse notice.
4metadata:
5 source: https://skillmd.com/skills/yanacuti1121/incident-response
6---
7
8# Incident Response
9
10A practical IR runbook for small teams and solo operators. Built around the SANS PICERL phases: **Preparation → Identification → Containment → Eradication → Recovery → Lessons Learned**.
11
12## When to invoke
13
14You are in incident territory if any of these are true:
15
16- A site you operate is defaced, redirecting, or serving content you did not publish
17- A webshell or backdoor file has been found
18- An admin/root account appeared that nobody on the team created
19- A hosting provider, registrar, or upstream sent an abuse notice
20- A credential you control turned up in a public dump
21- Traffic is being dropped from your IP by major email/DNS providers (sign of being on a blocklist)
22
23If you only **suspect** something is wrong, treat it as a real incident until you have evidence otherwise. False positives are cheap; missed compromises are expensive.
24
25## Phase 1 — Preparation (before anything happens)
26
27Have these ready before the first incident:
28
29- **Out-of-band contact channel** (Signal/Telegram/phone) — your primary work tool may itself be compromised
30- **Backup of last-known-good** for every production system, verified restorable
31- **Inventory** of who has access to what (hosting, DNS, registrar, repo, secrets store)
32- **A pre-written holding statement** for customers ("we are investigating an issue affecting X, more soon")
33- **A way to reach your registrar fast** — domain hijacks are catastrophic and the registrar is often slowest in real time
34
35## Phase 2 — Identification
36
37Goal: confirm a real incident and scope it.
38
39```bash
40# Webserver: who is hitting unusual paths?
41sudo tail -n 10000 /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30
42
43# Auth log: failed + successful root/SSH activity
44sudo grep -E 'Failed password|Accepted' /var/log/auth.log | tail -100
45
46# Currently logged-in users + open sessions
47who -a
48last -n 30
49
50# Listening processes — anything new?
51sudo ss -tlnp
52
53# Crontabs at every level
54for u in $(cut -d: -f1 /etc/passwd); do sudo crontab -u "$u" -l 2>/dev/null | sed "s|^|[$u] |"; done
55sudo ls -la /etc/cron.* /var/spool/cron/
56
57# Recently modified files in sensitive trees
58sudo find /var/www /etc /usr/local/bin -mtime -30 -type f 2>/dev/null
59```
60
61Document everything you find as you find it. Timestamps in UTC. Append-only — never edit prior entries.
62
63## Phase 3 — Containment
64
65The aim is to stop the bleeding **without destroying evidence**. Do not just delete malicious files immediately; snapshot first.
66
67### Short-term containment
68
691. **Snapshot** the affected filesystem (`tar`, hosting-panel snapshot, VM snapshot) and the database. Store off-host.
702. **Pull the site off the internet** at the cheapest layer that works:
71 - Cloudflare proxy → site-wide maintenance page (rule)
72 - Webserver → return 503 from the vhost
73 - Last resort → stop the service
743. **Rotate the credentials you know are exposed** — keep a list of every credential you rotate, when, and by whom.
754. **Revoke active sessions** on admin panels and force re-login.
765. **Block known-bad IPs** at the edge if you have a clear indicator.
77
78### Long-term containment (hours/days)
79
80- Move the service to a clean host before recovery if root cause is unclear
81- Add WAF rules / CDN rules for the attack pattern
82- If lateral movement is plausible (shared hosting, shared SSH keys, shared CI), assume every adjacent system is at risk and apply the same containment there
83
84## Phase 4 — Eradication
85
86You cannot eradicate what you have not understood. Before you wipe and reinstall, answer:
87
88- **Entry vector** — how did the attacker get in?
89- **Foothold** — what did they install (webshell, cron, systemd unit, modified binary, DB-resident backdoor)?
90- **Persistence** — would a reboot or a file-restore alone evict them?
91- **Lateral movement** — did they reach other systems?
92
93Only then:
94
951. Rebuild from known-good (golden image, fresh OS install, fresh CMS install).
962. Restore data from a backup taken **before** the foothold timestamp. If you cannot determine when the foothold was placed, treat all backups as suspect and restore data selectively (export rows, not full dumps).
973. Patch the entry vector specifically. If you do not know the vector, you have not finished investigating.
984. Re-issue all secrets that touched the compromised system.
99
100## Phase 5 — Recovery
101
1021. Bring the service back up in a staged way — internal first, then a small percentage of traffic, then full.
1032. Watch the logs you set up in Phase 2 for repeat indicators.
1043. Keep elevated monitoring for at least 30 days. Attackers often return.
1054. Notify users if their data may have been exposed. Be honest about what you know and do not know. In most jurisdictions personal-data breaches have legal notification deadlines (e.g. 72 hours under GDPR Art. 33) — check your jurisdiction.
106
107## Phase 6 — Lessons learned
108
109Write the post-mortem within a week, while the details are fresh. Structure:
110
111```
112# Post-Mortem — <incident name> — <date>
113
114## Summary
115One paragraph: what happened, impact, current state.
116
117## Timeline (UTC)
118- 2026-MM-DD HH:MM — first indicator observed
119- 2026-MM-DD HH:MM — confirmed incident, containment started
120- ...
121
122## Impact
123- Systems affected
124- Data potentially exposed
125- Users notified, regulators notified, etc.
126
127## Root cause
128The single technical reason this was possible. Be specific.
129
130## Contributing factors
131Process / monitoring / staffing gaps that let it reach production or stay undetected.
132
133## What worked
134Things to keep doing.
135
136## Action items
137Each with an owner and a due date. Track them somewhere durable (issue tracker).
138
139## Indicators of compromise (IOCs)
140File hashes, IPs, user-agents, domain names. Keep these — they are useful for next time.
141```
142
143The post-mortem is for learning, not blame. Name what failed; do not name who failed.
144
145## A note on legality and ethics
146
147- Only run this against systems you own or have explicit written authorization for.
148- Notify affected users where required by law and where it is the right thing to do regardless.
149- Do not "hack back" — it is illegal in most jurisdictions and unhelpful in all of them.
150- Preserve evidence in a way that would stand up if law enforcement gets involved.
151
152## What this skill will not do
153
154- It will not help you cover up a breach or delay required notifications.
155- It will not perform offensive actions against any system, including ones attacking you.
156- It is not a substitute for a qualified IR firm for serious incidents. Know your limits and call for help.