# Large Log Analyzer

> Analyze large ROS2 and C++ text logs from tens to hundreds of MB using bounded, streaming searches. Use for fault diagnosis, severity statistics, duplicate-message clustering, anomaly context, and timestamped root-cause reports; do not use for small logs that can be safely inspected directly.

- Skill: `louischen1905/large-log-analyzer` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add louischen1905/large-log-analyzer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/louischen1905/large-log-analyzer/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: LouisChen1905 (https://skillmd.com/u/louischen1905)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/louischen1905/large-log-analyzer

---


# Large Log Analyzer

Analyze large logs without loading the whole file into model context. Treat command output as a scarce resource: scan broadly with streaming tools, then inspect only small, relevant slices.

## Hard constraints

- Never use `cat`, an editor, Python `read()`/`readlines()`, or an unbounded tool output on the full log.
- Prefer `rg`, `grep`, `awk`, `head`, `tail`, `sed -n`, `wc`, `sort`, and `uniq`. A tool may stream through the full file, but its displayed output must be bounded.
- Quote paths and use `--` before user-provided paths where the command supports it.
- Do not infer events in a missing time range. Verify that requested timestamps exist and report gaps explicitly.
- Keep producer, transport, and consumer evidence separate for ROS2 rate or message-loss investigations.

## Workflow

1. Confirm the target with `ls -lh`, `file`, and `wc -l`; use `head` and `tail` only for a small format sample. If several files are supplied, inventory them first and preserve file identity in every finding.
2. Run `scripts/summarize_log.sh LOG [OUTPUT_DIR]`. Read `summary.txt`, `clusters.tsv`, and `timeline.tsv`; do not open the generated `severity-events.tsv` wholesale.
3. When the report includes a field observation in the form `timestamp + description` (for example, a wrong turn direction or a Replan loop), use `scripts/nav_log_filter.py` on an already bounded log slice. It selects navigation-specific evidence by time window, topic, and entities such as scanline/row/col; it also collapses bursts and can enforce `--budget-tokens`.
4. Do **not** run `nav_log_filter.py` directly on a raw tens-to-hundreds-MB log: it retains parsed records in memory. First narrow the input using the streaming summary and bounded context/time extraction. Use `--observe`, `--observations`, or `--stdin`; for example: `nav_log_filter.py slice.log --observe '09:38 旋转方向应该是顺时针，实际为逆时针 scanline105' --budget-tokens 6000`.
5. Use the summary or observation bundle to form hypotheses. Search narrowly with bounded commands such as `rg -n -m 50 -C 3 'pattern' -- LOG`. Search for lifecycle transitions, exceptions, timeouts, resets, node/topic names, request IDs, and the last normal event before the failure—not only severity words.
6. Inspect `contexts.txt`, which contains up to five representative FATAL/ERROR/WARN anchors with 100 lines before and after. For a better anchor, derive its line number with bounded `rg`, then inspect exactly `max(1, line-100)` through `line+100` using `sed -n`.
7. Build a chronological timeline from explicit timestamps. Include startup/configuration, last known-good behavior, first anomaly, retries or propagation, terminal state, and recovery. When clocks or formats differ, keep streams separate until correlation is justified.
8. Test competing explanations against the evidence. Repeated messages are usually symptoms; use the first occurrence and preceding state transition to identify the initiating fault. Distinguish confirmed cause, contributing factors, and unverified hypotheses.

## Navigation observation filter

Use `nav_log_filter.py --list-topics` to inspect its topic catalogue. Its current topics cover scanline selection and coverage, turn direction/radius, replan loops, stop/stall, obstacles, boundaries, speed, state machines, localization, and maps. Prefer its compact report over raw output. Preserve the source file, original line number, time, and level when citing its results.

## Duplicate handling

`clusters.tsv` groups severity lines after removing common timestamp and volatile-ID fields. Use its count, first line, last line, and representative message. Do not equate similar normalized text with identical causality when node, component, or operation context differs; refine with targeted `rg` searches.

## Final response

Lead with the conclusion and state confidence. Include:

- **Root cause:** the causal chain, or the narrowest supported conclusion if evidence is insufficient.
- **Evidence:** exact file, timestamp, line number, severity, and short excerpts; identify log gaps.
- **Timeline:** ordered, concise events from normal operation through failure/recovery.
- **Severity summary:** ERROR/WARN/FATAL totals and the dominant deduplicated patterns.
- **Recommendations:** immediate mitigation, code/config fix, and a concrete validation or logging improvement.

Keep excerpts short and avoid pasting large context windows. Separate observed facts from inference.