majiayu000 Log Error Digest
Automated log file analysis that produces error clustering, frequency statistics, and time distribution reports.
Features
- Error Clustering: Groups similar error messages by normalizing dynamic parts (IPs, UUIDs, numbers, etc.) to identify root causes
- Frequency Statistics: Counts occurrences by error type, sorted by severity
- Time Distribution: Shows error distribution by hour and by date, helping pinpoint peak error periods
Supported Log Formats
| Format | Description | Auto-detection |
|---|---|---|
| JSON | One JSON object per line with timestamp/level/message fields |
Starts with { |
| syslog | RFC 3164 format, e.g. Jan 1 12:00:00 host proc[pid]: msg |
Starts with month name |
| Nginx | Access log or error log format | Starts with IP or date/path pattern |
Usage
python scripts/analyze_logs.py <log_file_path> [options]
Parameters
| Parameter | Description | Default |
|---|---|---|
log_file |
Path to the log file (required) | - |
--format |
Log format: auto/json/syslog/nginx |
auto |
--top |
Show Top N error clusters | 20 |
--output |
Export results to a JSON file | Terminal output only |
--level |
Filter by log level (e.g. ERROR, WARN) |
All levels |
--since |
Only analyze logs after this time (ISO format) | No limit |
--until |
Only analyze logs before this time (ISO format) | No limit |
Examples
# Auto-detect format and analyze the entire log file
python scripts/analyze_logs.py /var/log/app.log
# Specify Nginx format, show only Top 10 errors
python scripts/analyze_logs.py /var/log/nginx/error.log --format nginx --top 10
# Filter ERROR level only, export JSON report
python scripts/analyze_logs.py app.log --level ERROR --output report.json
# Analyze logs within a specific time range
python scripts/analyze_logs.py app.log --since 2024-01-01T00:00:00 --until 2024-01-02T00:00:00
Output
Terminal Output
=======================================================
Log Analysis Report
=======================================================
📊 Overview
Detected format: json
Total lines: 15,234
Parsed: 15,100 (parse failures: 134)
Matched entries: 12,800
Errors: 2,341
Time range: 2024-01-01 00:03:12 ~ 2024-01-01 23:58:45
🔴 Top Error Clusters (47 total)
#1 [×523 ] Connection refused to database at 10.0.1.5:5432
First seen: 2024-01-01T00:15:30 Last seen: 2024-01-01T23:45:12
#2 [×312 ] Timeout waiting for response from user-service after 30000ms
First seen: 2024-01-01T02:10:00 Last seen: 2024-01-01T22:30:45
#3 [×198 ] File not found: /data/uploads/img_99421.png
First seen: 2024-01-01T08:00:00 Last seen: 2024-01-01T20:15:33
...
⏰ Time Distribution (by hour)
00:00 █████░░░░░░░░░░░░░░░ 42
01:00 ██░░░░░░░░░░░░░░░░░░ 18
...
14:00 ████████████████████ 523
...
📅 Time Distribution (by date)
2024-01-01 ████████████████████ 2,341
JSON Output
Use the --output parameter to export a structured JSON report for further processing or integration with monitoring systems.