Log Triage
The loudest message in a log is usually routine. The incident is the rare,
severe message that hit many hosts at once, often in a few minutes of a
multi-day file. Counting by volume finds the noise; this procedure finds the
incident.
Procedure
- Inventory every log.
list_files(path) for the folder. Every file the
user gave you is in scope, and every file is analysed over its full time
range, not the first page.
- Learn the line format from a sample.
read_file(file_path, start_line=1, end_line=40), plus a slice from the middle of the file.
Never read a whole large log into the conversation: you will see the start
and report it as the whole.
- Parse, template and rank with
run_python (snippet below). Fit the
LINE pattern to the sample. If more than a few percent of lines do not
parse, fix the pattern before going on. Indented continuation lines (stack
traces) are the only acceptable leftovers.
- Rank templates by severity, then hosts hit, then rarity, never by count.
Read the table this way:
- Baseline: present across most of the range (
span_pct near 100) at a
steady rate, whatever its level. Routine, even when it says "error" or
"fatal": a platform that logs thousands of FATAL lines every day is
telling you its normal, not an incident. Never count baseline lines as
alerts or add them to an anomaly total; report them once as background.
- Anomaly: a high-severity template, a template that appears in only a
short window (
span_pct near 0), a high peak against its own normal
rate, or a template that hit many distinct hosts at once.
- Find the bursts (second snippet): time buckets where errors jump far
above their median. For each burst, record the exact first and last
timestamp, how many lines, how many distinct hosts, and which templates
occur only there (
share_of_all near 1.0).
- Read around the first burst line. With the burst's
file and line,
read_file(file_path, start_line=max(1, line - 30), end_line=line + 10)
shows what happened just before. The cause comes before its symptoms. A flood of
timeouts after a hardware fault is the effect, not the root cause.
- Check the numbers add up. Template counts sum to the parsed line count
(the snippet asserts this). Per-level counts sum to the same total. Hosts
in a burst are no more than total hosts. Every timestamp you quote is
copied from the output, not rounded.
- Derive everything from the logs you were given. Do not fill in results
from a public dataset you recognise, from memory, or from a labels or
expected-results file found beside the logs, unless the user says it is an
input. If you find such a file, mention it and leave it unused.
Parse, template, rank
import os, re, tempfile
import pandas as pd
PATHS = [r"C:\Users\me\logs\app-01.log", r"C:\Users\me\logs\app-02.log"]
# Fit this to the sample you read. Example line:
# 2024-03-04 00:00:05,760 node-31 WARN cache miss for key 0x63ca828d
LINE = re.compile(r"^(?P<ts>\d{4}-\d\d-\d\d[ T]\d\d:\d\d:\d\d)\S*\s+(?P<host>\S+)\s+(?P<level>[A-Z]+)\s+(?P<msg>.*)$")
MASKS = [
(r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b", "<ID>"),
(r"\b\d{1,3}(?:\.\d{1,3}){3}(?::\d+)?\b", "<IP>"),
(r"\b0x[0-9a-fA-F]+\b", "<HEX>"),
(r"(?<!\w)(?:/[\w.-]+){2,}", "<PATH>"),
(r"\d+", "<N>"),
]
SEVERITY = {"TRACE": 0, "DEBUG": 0, "INFO": 1, "NOTICE": 1, "WARN": 2, "WARNING": 2,
"ERROR": 3, "ERR": 3, "SEVERE": 3, "CRITICAL": 4, "CRIT": 4, "FATAL": 4,
"ALERT": 4, "EMERG": 4, "PANIC": 4}
def template(msg):
for pattern, token in MASKS:
msg = re.sub(pattern, token, msg)
return msg
rows, unparsed = [], {}
for path in PATHS:
with open(path, encoding="utf-8", errors="replace") as f:
for n, line in enumerate(f, 1):
m = LINE.match(line.rstrip("\n"))
if m:
rows.append({**m.groupdict(), "file": os.path.basename(path), "line": n})
elif line.strip():
unparsed.setdefault(path, []).append(n)
df = pd.DataFrame(rows)
df["ts"] = pd.to_datetime(df["ts"].str.replace("T", " "), errors="coerce")
df["sev"] = df["level"].str.upper().map(SEVERITY).fillna(1).astype(int)
df["tpl"] = df["msg"].map(template)
print("parsed", len(df), "| bad timestamps", int(df["ts"].isna().sum()), "| hosts", df["host"].nunique())
for path, lines in unparsed.items():
print("unparsed", len(lines), "in", path, "e.g. lines", lines[:5])
print(df.groupby("file")["ts"].agg(["min", "max", "size"]).to_string()) # every file, full range
print(df["level"].value_counts().to_string())
span = df["ts"].max() - df["ts"].min()
bucket = next((b for b in ["1min", "5min", "15min", "1h", "6h", "1D"] if span / pd.Timedelta(b) <= 500), "7D")
df["bucket"] = df["ts"].dt.floor(bucket)
g = df.groupby("tpl").agg(count=("tpl", "size"), sev=("sev", "max"), hosts=("host", "nunique"),
first=("ts", "min"), last=("ts", "max"))
g["span_pct"] = ((g["last"] - g["first"]) / span * 100).round(0)
g["peak_per_" + bucket] = df.groupby(["tpl", "bucket"]).size().groupby(level=0).max()
assert g["count"].sum() == len(df), "template counts do not add up to parsed lines"
print(len(g), "templates; ranked by severity, then hosts hit, then rarity:")
pd.set_option("display.width", 250, "display.max_colwidth", 70)
print(g.sort_values(["sev", "hosts", "count"], ascending=[False, False, True]).head(15).to_string())
df.to_pickle(os.path.join(tempfile.gettempdir(), "log_triage.pkl")) # the burst step reads this
Adapting it:
- No level field? Map keywords instead: lines matching
fatal|panic|corrupt|crash|abort|out of memory get 4, error|fail|exception|timeout|refused get 3.
- No host field? Use the component, process, or device ID as
host, so
"hosts hit" still measures spread.
- No year in the timestamp (
Mar 5 14:22:01)? Add the year from the file
name or context and say you did.
- Very large files (millions of lines): raise
run_python's timeout.
Bursts
import os, tempfile
import pandas as pd
df = pd.read_pickle(os.path.join(tempfile.gettempdir(), "log_triage.pkl"))
span = df["ts"].max() - df["ts"].min()
bucket = next((b for b in ["1min", "5min", "15min", "1h", "6h", "1D"] if span / pd.Timedelta(b) <= 500), "7D")
step = pd.Timedelta(bucket)
bad = df[df["sev"] >= 3].set_index("ts").sort_index() # errors and worse; >= 2 adds warnings
counts = bad.resample(bucket).size()
threshold = max(10, 5 * counts.median())
hot = list(counts[counts >= threshold].index)
print(f"bucket {bucket}, median {counts.median():.0f}/bucket, threshold {threshold:.0f}, hot buckets {len(hot)}")
bursts = [] # adjacent hot buckets merge into one burst
for t in hot:
if bursts and t - bursts[-1][1] <= step:
bursts[-1][1] = t
else:
bursts.append([t, t])
overall = df["tpl"].value_counts()
for start, end in bursts:
win = bad[(bad.index >= start) & (bad.index < end + step)].reset_index()
print(f"\nBURST {start} -> {end + step}: {len(win)} lines, {win['host'].nunique()} hosts")
t = win.groupby("tpl").agg(count=("tpl", "size"), sev=("sev", "max"), hosts=("host", "nunique"),
first=("ts", "min"), last=("ts", "max"))
t["share_of_all"] = (t["count"] / overall.reindex(t.index)).round(2) # 1.0 = happens only here
t = t.join(win.sort_values("ts").groupby("tpl")[["file", "line"]].first()) # where it starts
print(t.sort_values(["sev", "share_of_all"], ascending=False).head(5).to_string())
No hot buckets does not mean no incident: re-run with sev >= 2, a smaller
bucket, or a single template's rows.
Lists of anomalous entities
When the user asks for a list of bad items (blocks, hosts, users, requests,
sessions), it has to be precise as well as complete:
- An entity goes on the list only if at least one of its events belongs to an
anomaly template from step 4, never because it appears next to a baseline
warning or error. Build the list from the anomaly templates, not from all
non-INFO lines.
- Check the share you flagged. If the list holds most of the entities in the
log, your rule matches the baseline: drop the templates that are present for
most entities and rebuild. Real anomalies are usually a small minority.
- Report how many entities exist, how many you flagged, and the template(s)
that put each group on the list.
ENTITY = r"\b(req-[0-9a-f]{8}|host-\w+)\b" # pin to your log's entity id: request, host, block, user...
df["entity"] = df["msg"].str.extract(ENTITY, expand=False)
ents = df.dropna(subset=["entity"])
share = ents.groupby("tpl")["entity"].nunique() / ents["entity"].nunique()
print(share.sort_values(ascending=False).head(10)) # near 1.0 = baseline, not anomaly
anomaly_tpls = set(ANOMALY_TEMPLATES) # the ones you judged anomalous above
flagged = sorted(ents.loc[ents["tpl"].isin(anomaly_tpls), "entity"].unique())
print(len(flagged), "of", ents["entity"].nunique(), "entities flagged")
Report shape
- Coverage: each file with its first and last timestamp and line count,
plus the unparsed count.
- Incident(s): what happened, in one line each. Exact first and last
timestamp, severity, line count, hosts hit (count plus a few names).
- Timeline: the few lines before the first incident line, then the
sequence.
- Likely cause: the evidence for it, worded as likely unless the log
states it outright. Name symptoms as symptoms.
- Baseline: the top routine templates and why they are routine (steady
across the whole range).
- Limits: unparsed lines, gaps in time, hosts or files with no data.
Example (invented): "Incident: disk controller faults on 31 of 40 nodes,
2024-03-06 13:13:00 to 13:16:58, 120 FATAL lines, a template seen nowhere else
in the 3.5-day log. The 3,002 request timeouts are baseline: steady across the
whole range, about 9 per 15 minutes."
Fork this
Pin LINE, SEVERITY and the host field to your platform's log format, and
add the templates your team already knows are harmless to a baseline list, so
the ranking starts from the unfamiliar.
1---2name: log-triage3description: Find what actually went wrong in log files. Use when the user asks for a root cause, an incident summary, anomalies, errors, failures or "what happened" in system, application, server, device or audit logs. Covers each file's full time range, ranks message templates by severity, rarity and hosts hit rather than volume, and pins burst timestamps.4license: MIT5version: 1.0.06metadata:7 gaia:8 security_tier: community9 tools_required:10 - list_files11 - read_file12 - run_python13 provenance:14 source: https://skillmd.com/skills/amd/log-triage15 upstream: starter-pack16---1718# Log Triage1920The loudest message in a log is usually routine. The incident is the rare,21severe message that hit many hosts at once, often in a few minutes of a22multi-day file. Counting by volume finds the noise; this procedure finds the23incident.2425## Procedure26271. **Inventory every log.** `list_files(path)` for the folder. Every file the28 user gave you is in scope, and every file is analysed over its full time29 range, not the first page.302. **Learn the line format from a sample.** `read_file(file_path,31 start_line=1, end_line=40)`, plus a slice from the middle of the file.32 Never read a whole large log into the conversation: you will see the start33 and report it as the whole.343. **Parse, template and rank with `run_python`** (snippet below). Fit the35 `LINE` pattern to the sample. If more than a few percent of lines do not36 parse, fix the pattern before going on. Indented continuation lines (stack37 traces) are the only acceptable leftovers.384. **Rank templates by severity, then hosts hit, then rarity**, never by count.39 Read the table this way:40 - **Baseline**: present across most of the range (`span_pct` near 100) at a41 steady rate, whatever its level. Routine, even when it says "error" or42 "fatal": a platform that logs thousands of FATAL lines every day is43 telling you its normal, not an incident. Never count baseline lines as44 alerts or add them to an anomaly total; report them once as background.45 - **Anomaly**: a high-severity template, a template that appears in only a46 short window (`span_pct` near 0), a high `peak` against its own normal47 rate, or a template that hit many distinct hosts at once.485. **Find the bursts** (second snippet): time buckets where errors jump far49 above their median. For each burst, record the exact first and last50 timestamp, how many lines, how many distinct hosts, and which templates51 occur only there (`share_of_all` near 1.0).526. **Read around the first burst line.** With the burst's `file` and `line`,53 `read_file(file_path, start_line=max(1, line - 30), end_line=line + 10)`54 shows what happened just before. The cause comes before its symptoms. A flood of55 timeouts after a hardware fault is the effect, not the root cause.567. **Check the numbers add up.** Template counts sum to the parsed line count57 (the snippet asserts this). Per-level counts sum to the same total. Hosts58 in a burst are no more than total hosts. Every timestamp you quote is59 copied from the output, not rounded.608. **Derive everything from the logs you were given.** Do not fill in results61 from a public dataset you recognise, from memory, or from a labels or62 expected-results file found beside the logs, unless the user says it is an63 input. If you find such a file, mention it and leave it unused.6465## Parse, template, rank6667```python68import os, re, tempfile69import pandas as pd7071PATHS = [r"C:\Users\me\logs\app-01.log", r"C:\Users\me\logs\app-02.log"]72# Fit this to the sample you read. Example line:73# 2024-03-04 00:00:05,760 node-31 WARN cache miss for key 0x63ca828d74LINE = re.compile(r"^(?P<ts>\d{4}-\d\d-\d\d[ T]\d\d:\d\d:\d\d)\S*\s+(?P<host>\S+)\s+(?P<level>[A-Z]+)\s+(?P<msg>.*)$")75MASKS = [76 (r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b", "<ID>"),77 (r"\b\d{1,3}(?:\.\d{1,3}){3}(?::\d+)?\b", "<IP>"),78 (r"\b0x[0-9a-fA-F]+\b", "<HEX>"),79 (r"(?<!\w)(?:/[\w.-]+){2,}", "<PATH>"),80 (r"\d+", "<N>"),81]82SEVERITY = {"TRACE": 0, "DEBUG": 0, "INFO": 1, "NOTICE": 1, "WARN": 2, "WARNING": 2,83 "ERROR": 3, "ERR": 3, "SEVERE": 3, "CRITICAL": 4, "CRIT": 4, "FATAL": 4,84 "ALERT": 4, "EMERG": 4, "PANIC": 4}8586def template(msg):87 for pattern, token in MASKS:88 msg = re.sub(pattern, token, msg)89 return msg9091rows, unparsed = [], {}92for path in PATHS:93 with open(path, encoding="utf-8", errors="replace") as f:94 for n, line in enumerate(f, 1):95 m = LINE.match(line.rstrip("\n"))96 if m:97 rows.append({**m.groupdict(), "file": os.path.basename(path), "line": n})98 elif line.strip():99 unparsed.setdefault(path, []).append(n)100df = pd.DataFrame(rows)101df["ts"] = pd.to_datetime(df["ts"].str.replace("T", " "), errors="coerce")102df["sev"] = df["level"].str.upper().map(SEVERITY).fillna(1).astype(int)103df["tpl"] = df["msg"].map(template)104print("parsed", len(df), "| bad timestamps", int(df["ts"].isna().sum()), "| hosts", df["host"].nunique())105for path, lines in unparsed.items():106 print("unparsed", len(lines), "in", path, "e.g. lines", lines[:5])107print(df.groupby("file")["ts"].agg(["min", "max", "size"]).to_string()) # every file, full range108print(df["level"].value_counts().to_string())109110span = df["ts"].max() - df["ts"].min()111bucket = next((b for b in ["1min", "5min", "15min", "1h", "6h", "1D"] if span / pd.Timedelta(b) <= 500), "7D")112df["bucket"] = df["ts"].dt.floor(bucket)113g = df.groupby("tpl").agg(count=("tpl", "size"), sev=("sev", "max"), hosts=("host", "nunique"),114 first=("ts", "min"), last=("ts", "max"))115g["span_pct"] = ((g["last"] - g["first"]) / span * 100).round(0)116g["peak_per_" + bucket] = df.groupby(["tpl", "bucket"]).size().groupby(level=0).max()117assert g["count"].sum() == len(df), "template counts do not add up to parsed lines"118print(len(g), "templates; ranked by severity, then hosts hit, then rarity:")119pd.set_option("display.width", 250, "display.max_colwidth", 70)120print(g.sort_values(["sev", "hosts", "count"], ascending=[False, False, True]).head(15).to_string())121df.to_pickle(os.path.join(tempfile.gettempdir(), "log_triage.pkl")) # the burst step reads this122```123124Adapting it:125126- **No level field?** Map keywords instead: lines matching127 `fatal|panic|corrupt|crash|abort|out of memory` get 4, `error|fail|exception|timeout|refused` get 3.128- **No host field?** Use the component, process, or device ID as `host`, so129 "hosts hit" still measures spread.130- **No year in the timestamp** (`Mar 5 14:22:01`)? Add the year from the file131 name or context and say you did.132- **Very large files** (millions of lines): raise `run_python`'s `timeout`.133134## Bursts135136```python137import os, tempfile138import pandas as pd139140df = pd.read_pickle(os.path.join(tempfile.gettempdir(), "log_triage.pkl"))141span = df["ts"].max() - df["ts"].min()142bucket = next((b for b in ["1min", "5min", "15min", "1h", "6h", "1D"] if span / pd.Timedelta(b) <= 500), "7D")143step = pd.Timedelta(bucket)144bad = df[df["sev"] >= 3].set_index("ts").sort_index() # errors and worse; >= 2 adds warnings145counts = bad.resample(bucket).size()146threshold = max(10, 5 * counts.median())147hot = list(counts[counts >= threshold].index)148print(f"bucket {bucket}, median {counts.median():.0f}/bucket, threshold {threshold:.0f}, hot buckets {len(hot)}")149150bursts = [] # adjacent hot buckets merge into one burst151for t in hot:152 if bursts and t - bursts[-1][1] <= step:153 bursts[-1][1] = t154 else:155 bursts.append([t, t])156overall = df["tpl"].value_counts()157for start, end in bursts:158 win = bad[(bad.index >= start) & (bad.index < end + step)].reset_index()159 print(f"\nBURST {start} -> {end + step}: {len(win)} lines, {win['host'].nunique()} hosts")160 t = win.groupby("tpl").agg(count=("tpl", "size"), sev=("sev", "max"), hosts=("host", "nunique"),161 first=("ts", "min"), last=("ts", "max"))162 t["share_of_all"] = (t["count"] / overall.reindex(t.index)).round(2) # 1.0 = happens only here163 t = t.join(win.sort_values("ts").groupby("tpl")[["file", "line"]].first()) # where it starts164 print(t.sort_values(["sev", "share_of_all"], ascending=False).head(5).to_string())165```166167No hot buckets does not mean no incident: re-run with `sev >= 2`, a smaller168bucket, or a single template's rows.169170## Lists of anomalous entities171172When the user asks for a list of bad items (blocks, hosts, users, requests,173sessions), it has to be precise as well as complete:174175- An entity goes on the list only if at least one of its events belongs to an176 anomaly template from step 4, never because it appears next to a baseline177 warning or error. Build the list from the anomaly templates, not from all178 non-INFO lines.179- Check the share you flagged. If the list holds most of the entities in the180 log, your rule matches the baseline: drop the templates that are present for181 most entities and rebuild. Real anomalies are usually a small minority.182- Report how many entities exist, how many you flagged, and the template(s)183 that put each group on the list.184185```python186ENTITY = r"\b(req-[0-9a-f]{8}|host-\w+)\b" # pin to your log's entity id: request, host, block, user...187df["entity"] = df["msg"].str.extract(ENTITY, expand=False)188ents = df.dropna(subset=["entity"])189share = ents.groupby("tpl")["entity"].nunique() / ents["entity"].nunique()190print(share.sort_values(ascending=False).head(10)) # near 1.0 = baseline, not anomaly191anomaly_tpls = set(ANOMALY_TEMPLATES) # the ones you judged anomalous above192flagged = sorted(ents.loc[ents["tpl"].isin(anomaly_tpls), "entity"].unique())193print(len(flagged), "of", ents["entity"].nunique(), "entities flagged")194```195196## Report shape197198- **Coverage**: each file with its first and last timestamp and line count,199 plus the unparsed count.200- **Incident(s)**: what happened, in one line each. Exact first and last201 timestamp, severity, line count, hosts hit (count plus a few names).202- **Timeline**: the few lines before the first incident line, then the203 sequence.204- **Likely cause**: the evidence for it, worded as likely unless the log205 states it outright. Name symptoms as symptoms.206- **Baseline**: the top routine templates and why they are routine (steady207 across the whole range).208- **Limits**: unparsed lines, gaps in time, hosts or files with no data.209210Example (invented): *"Incident: disk controller faults on 31 of 40 nodes,2112024-03-06 13:13:00 to 13:16:58, 120 FATAL lines, a template seen nowhere else212in the 3.5-day log. The 3,002 request timeouts are baseline: steady across the213whole range, about 9 per 15 minutes."*214215## Fork this216217Pin `LINE`, `SEVERITY` and the host field to your platform's log format, and218add the templates your team already knows are harmless to a baseline list, so219the ranking starts from the unfamiliar.