Large Log Analyzer

Large Log Analyzer

Analyze large logs without loading the whole file into model context. Treat command output as a scarce resource: scan broadly with streaming tools, then inspect only small, relevant slices.

Hard constraints

• Never use cat, an editor, Python read()/readlines(), or an unbounded tool output on the full log.
• Prefer rg, grep, awk, head, tail, sed -n, wc, sort, and uniq. A tool may stream through the full file, but its displayed output must be bounded.
• Quote paths and use -- before user-provided paths where the command supports it.
• Do not infer events in a missing time range. Verify that requested timestamps exist and report gaps explicitly.
• Keep producer, transport, and consumer evidence separate for ROS2 rate or message-loss investigations.

Workflow

1. Confirm the target with ls -lh, file, and wc -l; use head and tail only for a small format sample. If several files are supplied, inventory them first and preserve file identity in every finding.
2. Run scripts/summarize_log.sh LOG [OUTPUT_DIR]. Read summary.txt, clusters.tsv, and timeline.tsv; do not open the generated severity-events.tsv wholesale.
3. When the report includes a field observation in the form timestamp + description (for example, a wrong turn direction or a Replan loop), use scripts/nav_log_filter.py on an already bounded log slice. It selects navigation-specific evidence by time window, topic, and entities such as scanline/row/col; it also collapses bursts and can enforce --budget-tokens.
4. Do not run nav_log_filter.py directly on a raw tens-to-hundreds-MB log: it retains parsed records in memory. First narrow the input using the streaming summary and bounded context/time extraction. Use --observe, --observations, or --stdin; for example: nav_log_filter.py slice.log --observe '09:38 旋转方向应该是顺时针，实际为逆时针 scanline105' --budget-tokens 6000.
5. Use the summary or observation bundle to form hypotheses. Search narrowly with bounded commands such as rg -n -m 50 -C 3 'pattern' -- LOG. Search for lifecycle transitions, exceptions, timeouts, resets, node/topic names, request IDs, and the last normal event before the failure—not only severity words.
6. Inspect contexts.txt, which contains up to five representative FATAL/ERROR/WARN anchors with 100 lines before and after. For a better anchor, derive its line number with bounded rg, then inspect exactly max(1, line-100) through line+100 using sed -n.
7. Build a chronological timeline from explicit timestamps. Include startup/configuration, last known-good behavior, first anomaly, retries or propagation, terminal state, and recovery. When clocks or formats differ, keep streams separate until correlation is justified.
8. Test competing explanations against the evidence. Repeated messages are usually symptoms; use the first occurrence and preceding state transition to identify the initiating fault. Distinguish confirmed cause, contributing factors, and unverified hypotheses.

Navigation observation filter

Use nav_log_filter.py --list-topics to inspect its topic catalogue. Its current topics cover scanline selection and coverage, turn direction/radius, replan loops, stop/stall, obstacles, boundaries, speed, state machines, localization, and maps. Prefer its compact report over raw output. Preserve the source file, original line number, time, and level when citing its results.

Duplicate handling

clusters.tsv groups severity lines after removing common timestamp and volatile-ID fields. Use its count, first line, last line, and representative message. Do not equate similar normalized text with identical causality when node, component, or operation context differs; refine with targeted rg searches.

Final response

Lead with the conclusion and state confidence. Include:

• Root cause: the causal chain, or the narrowest supported conclusion if evidence is insufficient.
• Evidence: exact file, timestamp, line number, severity, and short excerpts; identify log gaps.
• Timeline: ordered, concise events from normal operation through failure/recovery.
• Severity summary: ERROR/WARN/FATAL totals and the dominant deduplicated patterns.
• Recommendations: immediate mitigation, code/config fix, and a concrete validation or logging improvement.

Keep excerpts short and avoid pasting large context windows. Separate observed facts from inference.